Working with LLMs

Embabel supports any LLM supported by Spring AI. In practice, this is just about any LLM.

Choosing an LLM

Embabel encourages you to think about LLM choice for every LLM invocation. The PromptRunner interface makes this easy. Because Embabel enables you to break agentic flows up into multiple action steps, each step can use a smaller, focused prompt with fewer tools. This means it may be able to use a smaller LLM.

Considerations:

• Consider the complexity of the return type you expect from the LLM. This is typically a good proxy for determining required LLM quality. A small LLM is likely to struggle with a deeply nested return structure.
• Consider the nature of the task. LLMs have different strengths; review any available documentation. You don’t necessarily need a huge, expensive model that is good at nearly everything, at the cost of your wallet and the environment.
• Consider the sophistication of tool calling required. Simple tool calls are fine, but complex orchestration is another indicator you’ll need a strong LLM. (It may also be an indication that you should create a more sophisticated flow using Embabel GOAP.)
• Consider trying a local LLM running under Ollama or Docker.

Embabel makes it easy to switch LLMs; try the cheapest thing that could work and switch if it doesn’t.

Tuning for Smaller and Local Models

A core goal of Embabel is to make agentic flows work well across the full range of LLMs, so you can choose the cheapest, smallest, or most private model that does the job — rather than always reaching for a frontier model. Smaller chat models behave differently from frontier models in ways that the framework can compensate for:

• Silent failures after tool calls. Weaker open-weights models (e.g. gpt-oss-20b, some Qwen variants) sometimes return blank text with no further tool calls when they don’t know how to proceed. Without intervention the tool loop exits with empty content. Activate embabel.agent.platform.toolloop.empty-response.max-retries: 1 to feed a synthetic nudge back to the model and give it one more chance — see .
• Tool-name confusion. Smaller models more frequently call tools by approximate names. The default AutoCorrectionPolicy handles this by feeding back a "did you mean X?" suggestion; tune embabel.agent.platform.toolloop.tool-not-found.max-retries if your model needs more attempts.
• Iteration headroom. Recovery costs LLM calls. If you enable retry policies, raise embabel.agent.platform.toolloop.max-iterations so a turn that needs an extra round trip doesn’t run out of budget.

These settings are off-by-default so existing deployments using strong models behave exactly as before. Turn them on per-deployment when the model you’ve picked benefits from them.

Advanced: Custom LLM Integration

user-supplied key at runtime, see . That is the recommended path for BYOK scenarios. This section covers implementing LlmMessageSender from scratch for providers not otherwise supported by Embabel.

Embabel’s tool loop is framework-agnostic, allowing you to integrate any LLM provider by implementing the LlmMessageSender interface. This is useful when:

• You want to use an LLM provider not supported by Spring AI
• You need custom request/response handling
• You’re integrating with a proprietary or internal LLM service

The LlmMessageSender Interface

The core abstraction is the LlmMessageSender functional interface:

@FunctionalInterface
public interface LlmMessageSender {
    LlmMessageResponse call(
        List<Message> messages,
        List<Tool> tools
    );
}

The implementation makes a single LLM inference call and returns the response. Importantly, it does not execute tools--it only returns any tool call requests from the LLM. Tool execution is handled by Embabel’s DefaultToolLoop.

embabel.agent.platform.toolloop.type=parallel

For full list of tool loop configuration parameters please refer to ToolLoopConfiguration.

Tool-Not-Found Recovery Policy

When the LLM calls a tool by a name that doesn’t exist in the available set, the behavior is controlled by ToolNotFoundPolicy.

Two built-in policies are provided:

• AutoCorrectionPolicy (default) — feeds the error back to the LLM so it can self-correct. Uses case-insensitive fuzzy matching to suggest corrections for hallucinated tool names (e.g., ragbot_vectorSearch → suggests vectorSearch). When multiple candidates match, all are listed. Throws ToolNotFoundException after 3 consecutive failures.
• ImmediateThrowPolicy — throws ToolNotFoundException immediately.

The system-wide default is AutoCorrectionPolicy, provided as a Spring bean with @ConditionalOnMissingBean. To override it globally, define your own ToolNotFoundPolicy bean.

For per-interaction control, use withToolNotFoundPolicy() on PromptRunner:

promptRunner
    .withToolNotFoundPolicy(new AutoCorrectionPolicy(5))
    .creating(MyOutput.class)
    .create(messages);

Custom policies can be implemented by implementing the ToolNotFoundPolicy interface:

class MyEditDistancePolicy : ToolNotFoundPolicy {
    override fun handle(requestedName: String, availableTools: List<Tool>): ToolNotFoundAction {
        // Custom recovery logic, e.g. edit-distance matching
        ...
    }
}

Response Types

The LlmMessageResponse contains:

• message: The LLM’s response as an Embabel Message
• textContent: Text content from the response
• usage: Optional token usage information

For responses that include tool calls, return an AssistantMessageWithToolCalls:

public record ToolCall(
    String id,         // Unique identifier for the tool call
    String name,       // Name of the tool to invoke
    String arguments   // JSON arguments for the tool
) {}

Example: Custom LLM Provider

Here’s an example of implementing LlmMessageSender for a hypothetical HTTP-based LLM API:

public class MyCustomLlmMessageSender implements LlmMessageSender {

    private final HttpClient httpClient;
    private final String apiKey;
    private final String model;

    public MyCustomLlmMessageSender(HttpClient httpClient, String apiKey, String model) {
        this.httpClient = httpClient;
        this.apiKey = apiKey;
        this.model = model;
    }

    @Override
    public LlmMessageResponse call(List<Message> messages, List<Tool> tools) {
        // Convert Embabel messages to your API's format
        List<Map<String, Object>> apiMessages = messages.stream()
            .map(message -> Map.<String, Object>of(
                "role", message.getRole().name().toLowerCase(),
                "content", message.getTextContent()
            ))
            .toList();

        // Convert tool definitions to your API's format
        List<Map<String, Object>> apiTools = tools.stream()
            .map(tool -> Map.<String, Object>of(
                "name", tool.getDefinition().getName(),
                "description", tool.getDefinition().getDescription(),
                "parameters", tool.getDefinition().getInputSchema().jsonSchema()
            ))
            .toList();

        // Make API request (using your preferred HTTP client)
        MyApiResponse responseBody = httpClient.post("https://api.my-llm.com/chat")
            .header("Authorization", "Bearer " + apiKey)
            .body(Map.of(
                "model", model,
                "messages", apiMessages,
                "tools", apiTools.isEmpty() ? null : apiTools
            ))
            .execute(MyApiResponse.class);

        // Check if LLM requested tool calls
        List<ToolCall> toolCalls = null;
        if (responseBody.getToolCalls() != null) {
            toolCalls = responseBody.getToolCalls().stream()
                .map(call -> new ToolCall(
                    call.getId(),
                    call.getFunction().getName(),
                    call.getFunction().getArguments()
                ))
                .toList();
        }

        Message embabelMessage;
        if (toolCalls == null || toolCalls.isEmpty()) {
            embabelMessage = new AssistantMessage(
                responseBody.getContent() != null ? responseBody.getContent() : ""
            );
        } else {
            embabelMessage = new AssistantMessageWithToolCalls(
                responseBody.getContent() != null ? responseBody.getContent() : "",
                toolCalls
            );
        }

        Usage usage = null;
        if (responseBody.getUsage() != null) {
            usage = new Usage(
                responseBody.getUsage().getPromptTokens(),
                responseBody.getUsage().getCompletionTokens()
            );
        }

        return new LlmMessageResponse(embabelMessage, responseBody.getContent(), usage);
    }
}

Creating an LlmService

To make your custom LLM available through Embabel’s ModelProvider, implement the LlmService interface:

public class MyCustomLlmService implements LlmService<MyCustomLlmService> {

    private final String name;
    private final String provider;
    private final HttpClient httpClient;
    private final String apiKey;
    private final LocalDate knowledgeCutoffDate;
    private final List<PromptContributor> promptContributors;
    private final PricingModel pricingModel;

    public MyCustomLlmService(
            String name,
            String provider,
            HttpClient httpClient,
            String apiKey,
            LocalDate knowledgeCutoffDate,
            PricingModel pricingModel) {
        this.name = name;
        this.provider = provider;
        this.httpClient = httpClient;
        this.apiKey = apiKey;
        this.knowledgeCutoffDate = knowledgeCutoffDate;
        this.promptContributors = knowledgeCutoffDate != null
            ? List.of(new KnowledgeCutoffDate(knowledgeCutoffDate))
            : List.of();
        this.pricingModel = pricingModel;
    }

    @Override
    public String getName() { return name; }

    @Override
    public String getProvider() { return provider; }

    @Override
    public LocalDate getKnowledgeCutoffDate() { return knowledgeCutoffDate; }

    @Override
    public List<PromptContributor> getPromptContributors() { return promptContributors; }

    @Override
    public PricingModel getPricingModel() { return pricingModel; }

    @Override
    public LlmMessageSender createMessageSender(LlmOptions options) {
        return new MyCustomLlmMessageSender(
            httpClient,
            apiKey,
            options.getModel() != null ? options.getModel() : name
        );
    }

    @Override
    public MyCustomLlmService withKnowledgeCutoffDate(LocalDate date) {
        return new MyCustomLlmService(name, provider, httpClient, apiKey, date, pricingModel);
    }

    @Override
    public MyCustomLlmService withPromptContributor(PromptContributor promptContributor) {
        var newContributors = new ArrayList<>(promptContributors);
        newContributors.add(promptContributor);
        return new MyCustomLlmService(
            name, provider, httpClient, apiKey, knowledgeCutoffDate,
            newContributors, pricingModel
        );
    }
}

Then register it as a Spring bean:

@Configuration
public class MyLlmConfiguration {

    @Bean
    public LlmService<?> myCustomLlm(
            HttpClient httpClient,
            @Value("$\{my-llm.api-key}") String apiKey) {
        return new MyCustomLlmService(
            "my-custom-model",
            "MyProvider",
            httpClient,
            apiKey,
            LocalDate.of(2024, 12, 1),
            null
        );
    }
}

The bean will be automatically discovered and made available through the ModelProvider.

How Model Discovery and Selection Works

When your application starts, ConfigurableModelProvider collects all LlmService beans from the Spring application context. Your custom LLM is matched by the name property you set on your LlmService implementation.

By name: Use the name from your LlmService directly. This works with @LlmCall, ai.withLlm(), and AgenticTool.withLlm():

// In a declarative action
@LlmCall(llm = "my-custom-model")
String myAction();

// In an imperative action
ai.withLlm("my-custom-model")
    .create("Tell me a joke", String.class);

By role: Map a role name to your model name in configuration, then reference it with the # prefix:

embabel:
  models:
    default-llm: my-custom-model  # ①
    llms:
      best: my-custom-model       # ②
      cheapest: my-small-model    # ③

1. Sets the default LLM used when no explicit model is specified
2. Maps the best role to your custom model
3. Maps the cheapest role to a different model

Then reference roles with #:

// By role
@LlmCall(llm = "#best")
String myAction();

// Or programmatically
ai.withLlmByRole("best")
    .create("Tell me a joke", String.class);

Roles across providers

The flat llms map above names one model per role, and a model belongs to one provider. That is the right shape when a deployment will only ever be keyed for one provider. When it will not - a bring-your-own-key application, or one that fails over between providers - give the role a provider dimension with roles:

embabel:
  models:
    default-llm: gpt-4.1-mini
    roles:
      cheapest:
        openai:
          model: gpt-4.1-nano       # ①
        anthropic:
          model: claude-haiku-4-5
          temperature: 0.3          # ②
      best:
        openai:
          model: gpt-4.1
        anthropic:
          model: claude-sonnet-4-6

1. The model this role means when OpenAI is the active provider
2. Tuning configured against the role, applied whenever this role resolves to this provider

Call sites do not change: ai.withLlmByRole("cheapest") resolves against whichever provider is active for the call. With no user key in play, that is the provider supplying the default-llm, so a single-provider deployment needs nothing further.

Anything the caller sets explicitly still wins over what the role configures, so ai.withLlmByRole("cheapest").withTemperature(0.9) keeps 0.9.

A role that cannot be satisfied - because the active provider has no entry for it, or the model it names is not registered - throws NoSuitableModelException at the point of use. It does not quietly resolve to something else: falling back to the default-llm would let a role like cheapest become the most capable, and most expensive, model in the deployment without anyone noticing.

Startup is a separate question, and the answer depends on whether the deployment holds a key. A deployment that has one is expected to name models it has: an unresolvable name there is a typo, and letting it boot would move the failure to whichever unrelated call first asks for that role. A deployment still waiting for a key has no models registered yet by definition, so the same name is expected and only worth reporting.

Whichever way startup goes, asking for a role that cannot be satisfied still throws at the point of use, and the message names the role, the active provider, and the models that were available.

The deployment default as a role

default-llm may name a role rather than a model:

embabel:
  models:
    default-llm: cheapest       # ①
    roles:
      cheapest:
        openai:
          model: gpt-4.1-nano
        anthropic:
          model: claude-haiku-4-5

1. A role from roles or from the flat llms map, not a model name

A model name is resolved once, against the services registered at startup. A role is resolved per call, through the same resolver chain as any other role - so the default follows whichever provider key is active, and a deployment whose first key arrives after startup gets a working default without a restart. That is the case this exists for: under BYOK nothing is registered before the first key, so a default fixed at construction can only be the placeholder until the process restarts.

A model name that nothing registers falls back to the role chain as well, so default-llm keeps working whichever way it was meant. Nothing existing changes: a name that does resolve against a registered service is still resolved once and never sent through the resolvers.

A default that nothing can satisfy does not throw the way an unsatisfied role does. It falls back to whatever default-llm resolved to at startup - the placeholder, in a deployment awaiting a key - so the caller sees the same actionable "no LLM configured" error. Startup is stricter: with no placeholder to fall back to, a default-llm that names nothing this deployment has registered - whether a typo or a role whose every model is missing - is still fatal, and the message says which of the two it is.

Resolving roles in code

To decide what a role means per user rather than per deployment, register a RoleResolver bean. Resolvers are consulted in Ordered order, ahead of the configuration above, and returning null delegates to the next one.

@Component
class ByokRoleResolver(
    private val userKeyStore: UserKeyStore,
) : RoleResolver {

    override fun resolve(role: String, context: ModelSelectionContext): RoleResolution? =
        context.userId
            ?.let { userKeyStore.activeKeyFor(it) }
            ?.let { RoleResolution.Credential(ProviderCredential(it.provider, it.apiKey)) }
}

RoleResolution.Credential hands back a key and lets the platform do the rest: it looks up what the role means under that provider, builds the service for it, and caches it per (provider, key, model) so it is not rebuilt on every call. Return RoleResolution.Options to name a model and its tuning directly, or RoleResolution.Service to supply an LlmService you built yourself.

embabel-agent-starter-byok already knows where to send a key for every provider BYOK supports — Anthropic, OpenAI, DeepSeek, Mistral, Gemini and Atlas Cloud. The resolver above is therefore all the code a deployment using any of them needs.

The ModelSelectionContext a resolver receives comes from ModelSelectionContextHolder, which the application sets at its request boundary:

ModelSelectionContextHolder.with(ModelSelectionContext(userId = currentUserId)) {
    agentPlatform.runAgent(...)
}

The platform carries the context across the threads it starts itself — a background agent run, parallel actions, OperationContext.parallelMap — so setting it once at the request boundary is enough. It does not reach threads your own code spawns: capture it there with ModelSelectionContextHolder.get() and re-establish it inside the new thread.

Adding a provider of your own

For a provider the starter does not cover — a gateway, a proxy, a self-hosted endpoint — register a CredentialEndpointResolver. It says where a key goes and what the service built from it should call itself; the platform builds the client:

@Bean
fun ourGatewayEndpoint() = CredentialEndpointResolver { credential, _ ->
    if (!credential.provider.equals("OurGateway", ignoreCase = true)) null
    else CredentialEndpoint.OpenAiCompatible(provider = "OurGateway", baseUrl = GATEWAY_URL)
}

The case names the wire protocol, which is what decides the client: OpenAiCompatible for the OpenAI protocol — OpenAI itself, DeepSeek, Mistral, Gemini, Atlas Cloud and most gateways — or Anthropic for Anthropic’s. Both carry pricingModel, which defaults to PricingModel.ALL_YOU_CAN_EAT because a BYOK call is billed to the user’s key rather than to the deployment, and knowledgeCutoffDate, which is null unless you know it for the model the role named.

Your resolvers are asked first, in Ordered order, and the endpoints the starter already knows are the fallback — so answering for a provider the framework handles overrides it, which is how you route OpenAI through your own proxy. No @Order is needed for that; it decides only which of your own resolvers is asked first. Return null for a provider you do not handle: a resolver that answers for everything would point someone else’s key at your endpoint. If nothing handles the provider, the role fails with NoSuitableModelException and a log line naming it.

A resolver returns a value, so it needs no Spring context to test and nothing from com.embabel.agent.spi, which application code should not depend on.

For a wire protocol this framework has no client for, register a CredentialLlmServiceFactory instead and build the service yourself. That one does name LlmService, an SPI type, which is why it is the second choice rather than the first. The shipped implementations — one per protocol — stand aside for a bean of the same name, but replacing one that way also replaces the code that builds what your CredentialEndpointResolver beans resolve for that protocol.

Reading what a role is configured to mean

Resolution answers "what will this call use", which depends on the active key and on any RoleResolver beans. A settings screen or diagnostic endpoint wants a different question — "what does this deployment declare this role to be" — and answering it by reading embabel.models directly means reimplementing the precedence between the flat and nested shapes, which is how a settings screen ends up disagreeing with what actually runs.

ModelProvider.configuredOptionsForRole answers that question against the deployment’s own provider, ignoring user keys and application resolvers:

val declared: String? = modelProvider.configuredOptionsForRole("cheapest")?.modelName

Use resolveLlmOptions instead when you need to know what a specific call will really do.

Using Your Custom Implementation (Alternative)

If you need more control over the LLM operations layer itself, you can extend ToolLoopLlmOperations:

public class MyCustomLlmOperations extends ToolLoopLlmOperations {

    private final HttpClient httpClient;
    private final String apiKey;

    public MyCustomLlmOperations(
            HttpClient httpClient,
            String apiKey,
            ModelProvider modelProvider,
            ToolDecorator toolDecorator,
            Validator validator) {
        super(modelProvider, toolDecorator, validator);
        this.httpClient = httpClient;
        this.apiKey = apiKey;
    }

    @Override
    protected LlmMessageSender createMessageSender(LlmService<?> llm, LlmOptions options) {
        return new MyCustomLlmMessageSender(
            httpClient,
            apiKey,
            options.getModel() != null ? options.getModel() : "default-model"
        );
    }
}

The ToolLoopLlmOperations base class provides several extension points:

• createMessageSender(): Create the LLM communication layer
• createOutputConverter(): Parse LLM responses into typed objects
• sanitizeStringOutput(): Clean up raw text responses
• emitCallEvent(): Emit observability events

Key Implementation Notes

1. Tool calls are not executed by your sender. Just return the tool call requests--Embabel’s tool loop handles execution and continuation.
2. Handle both tool and non-tool responses. Return AssistantMessage for plain text, AssistantMessageWithToolCalls when the LLM wants to invoke tools.
3. Include usage information when available. This enables cost tracking and observability.
4. Message types matter. The tool loop expects specific message types:
   • UserMessage: User input
   • SystemMessage: System prompts
   • AssistantMessage: LLM text response
   • AssistantMessageWithToolCalls: LLM response with tool requests
   • ToolResultMessage: Result returned to LLM after tool execution

Advanced: Custom Embedding Service

Just as you can integrate a custom LLM, you can implement a custom embedding service that doesn’t depend on Spring AI. This is useful when:

• You want to use an embedding provider not supported by Spring AI
• You need custom pre/post-processing of embeddings
• You’re integrating with a proprietary or internal embedding API

The EmbeddingService Interface

The EmbeddingService interface is framework-agnostic. Unlike SpringAiEmbeddingService, a custom implementation does not need to wrap a Spring AI EmbeddingModel:

public interface EmbeddingService {
    float[] embed(String text);
    List<float[]> embed(List<String> texts);
    int getDimensions();
    String getName();
    String getProvider();
}

Example: Custom Embedding Provider

Here’s an example of implementing EmbeddingService for an HTTP-based embedding API:

public class MyCustomEmbeddingService implements EmbeddingService {

    private final String name;
    private final String provider;
    private final int dimensions;
    private final HttpClient httpClient;
    private final String apiKey;

    public MyCustomEmbeddingService(
            String name,
            String provider,
            int dimensions,
            HttpClient httpClient,
            String apiKey) {
        this.name = name;
        this.provider = provider;
        this.dimensions = dimensions;
        this.httpClient = httpClient;
        this.apiKey = apiKey;
    }

    @Override
    public String getName() { return name; }

    @Override
    public String getProvider() { return provider; }

    @Override
    public int getDimensions() { return dimensions; }

    @Override
    public float[] embed(String text) {
        return embed(List.of(text)).get(0);
    }

    @Override
    public List<float[]> embed(List<String> texts) {
        MyEmbeddingResponse response = httpClient
            .post("https://api.my-embeddings.com/embed")
            .header("Authorization", "Bearer " + apiKey)
            .body(Map.of("texts", texts, "model", name))
            .execute(MyEmbeddingResponse.class);
        return response.getEmbeddings();
    }
}

Registering as a Spring Bean

Register your custom embedding service as a Spring bean and it will be automatically discovered:

@Configuration
public class MyEmbeddingConfiguration {

    @Bean
    public EmbeddingService myCustomEmbeddings(
            HttpClient httpClient,
            @Value("$\{my-embeddings.api-key}") String apiKey) {
        return new MyCustomEmbeddingService(
            "my-custom-embeddings",
            "MyProvider",
            384,
            httpClient,
            apiKey
        );
    }
}

Discovery and Selection

Custom embedding services follow the same discovery and selection pattern as LLMs (see [](/How Model Discovery and Selection Works)).

By name: Use ai.withEmbeddingService() with the name from your implementation:

ai.withEmbeddingService("my-custom-embeddings")
    .embed("Hello world");

By role: Map a role name to your embedding service in configuration:

embabel:
  models:
    default-embedding-model: my-custom-embeddings  # ①
    embedding-services:
      cheapest: my-custom-embeddings               # ②

1. Sets the default embedding service
2. Maps the cheapest role to your custom embedding service

Embedding roles under bring-your-own-key

default-embedding-model may name a role rather than a model. A model name is resolved once, against the services registered at startup; a role is resolved on every call, through the same resolver chain roles use for chat.

That difference matters only for a deployment whose provider key arrives after boot. Such a deployment registers no embedding service at startup, so a default fixed at construction can only be the placeholder — permanently, until the process restarts. Naming a role instead means the key that satisfies every other role satisfies the embedding default too, with no restart.

embabel:
  models:
    default-embedding-model: documents   # ①
    embedding-services:
      documents: text-embedding-3-small  # ②

1. A role, because documents is a key in embedding-services. Resolved per call.
2. What the role means for a single-provider deployment.

Where the provider is not known until a key arrives, embedding-roles gives a role a model per provider — the embedding counterpart of embabel.models.roles:

embabel:
  models:
    default-embedding-model: documents
    embedding-roles:
      documents:
        openai: text-embedding-3-small
        mistral: mistral-embed

It is flatter than roles, which maps to a whole LlmOptions. An embedding model takes text and returns a vector: there is no temperature to set, and the one parameter that does vary — the width — is the model’s to state and the index’s to follow.

Entries naming a model this deployment has not registered are reported at startup and tolerated, because that is what the shape is for: the model is built from a key supplied at runtime.

Which column is read comes from the active key. Where there is no active key, it comes from the registered embedding services — but only where they agree on one provider. A deployment with two registered providers and no key reads embedding-services instead of picking a column, because the alternative would be a choice made by bean registration order.

To resolve an embedding role for a user’s own key, register an EmbeddingRoleResolver returning EmbeddingRoleResolution.Credential, and the platform builds the service through a CredentialEmbeddingServiceFactory and caches it per provider, key and model. embabel-agent-starter-byok ships one covering every OpenAI-compatible provider, so this usually needs no application code.

Locally served models that appear after startup

A local runner — Docker Model Runner, LM Studio, Ollama — is enumerated at startup, and each model it was serving is registered as a bean. A model pulled a minute later is invisible to that, however reachable it is, because the only thing that would have seen it ran once while the platform was being built.

So a role may also be satisfied by asking the runner what it is serving now. This is the local counterpart of the credential path, seam for seam: configuration names the model per provider, the resolver is asked per call, and the service is built on the spot. docker model pull followed by a request is enough; the model needs no restart to become usable, and nothing distinguishes it from one that was already there.

Name the model under the provider column, not in the flat llms / embedding-services map:

embabel:
  models:
    default-embedding-model: documents
    embedding-roles:
      documents:
        docker: ai/embeddinggemma       # ①
    roles:
      cheapest:
        docker:
          model: ai/qwen3               # ②
          temperature: 0.0              # ③

1. The embedding model this role means on a Docker runner. Need not be pulled yet.
2. The same for a chat role. The provider key is docker, lmstudio or ollama.
3. Options configured beside the model travel with it, exactly as for a registered model.

The flat maps are validated at startup and a name nothing has registered is fatal there — correctly, since in a deployment that holds its models a missing name is a typo. A provider column is not, because its entries name models this process may have no way to serve yet: embedding-roles is never checked, and a roles entry under a runner’s column is reported and tolerated rather than fatal, so an appliance whose default LLM is local still starts before anything has been pulled. A runner declares that through LateArrivingModels, implemented by its LocalModelCatalog; without it — or with discovery disabled, since then nothing will ever ask the runner — the same entry under the deployment’s own provider is still a typo and still fatal.

The runner is asked at most once per refresh interval, so this is not an HTTP round trip per embedding:

embabel:
  agent:
    platform:
      models:
        local-discovery:
          enabled: true                 # ①
          refresh-interval: 10s         # ②
          miss-refresh-interval: 1s     # ③

1. Off restores the startup-only behaviour: a model pulled after boot stays invisible until the process comes back.
2. How long a list of served models is reused. Also the floor on how long a model that has just been removed keeps resolving.
3. How often a model absent from the cached list may force a fresh look, so a model pulled two seconds ago is usable on the next request rather than the one after the refresh interval.

A role is declined, and left to the rest of the chain, in three cases: configuration says nothing about this provider; the runner is not serving the named model, so an absent runner simply answers nothing; or the call carries a user key for a different provider, since a user who brought a key asked for their provider and a model on this machine would answer with the wrong one.

A locally served model is also listed and usable by name, not only reachable through a role. listModelNames and listModels report what the runners are serving now alongside what was registered at startup, so a model pulled a moment ago appears in the UI meant to select it; asking for one of those names — getLlm, getEmbeddingService, default-llm, default-embedding-model — resolves it. A model registered at startup wins over a runner serving the same name, and appears once.

A by-name lookup has a name to miss on, so it re-asks the runner and finds a just-pulled model within miss-refresh-interval; a listing has nothing to miss on and is served from the snapshot, so it can trail a by-name lookup by up to refresh-interval. Closing that gap would mean an HTTP round trip per listing, which is the cost the snapshot exists to avoid.

default-llm and default-embedding-model are asked of the runners only when they name a model. When either names a role it resolves through the chain, which already contains the local resolvers — so asking again by that name could only miss, and a miss is a request.

Which of the two lists a model appears in comes from the runner: LM Studio reports a type per model, while Docker and Ollama read the same configuration their startup registration reads, so a model cannot change category by virtue of having arrived late. That classification decides what a caller naming a model gets back — a chat model will not answer an embedding request by name — but it does not gate role resolution, where the role that asked is the better signal.

Late models are described by name and provider only. The rest of a model’s metadata is read off a built service, and building one per entry would turn a listing into a round of work against the runner.

To do the same for a runner the framework does not ship, implement LocalModelSource — what it is serving, with what each model is for, and how to build a service for one — and publish the three beans LocalModelBeans builds over it: the catalog, and the two resolvers sharing it.

Advanced Caching with Anthropic

While many providers have implicit caching managed internally, Anthropic exposes public APIs for explicit prompt caching control. This allows you to optimize costs and latency for applications with long prompts, many tools, or extended conversations.

Motivation

Anthropic’s prompt caching feature provides significant benefits:

• Cost savings: Cache reads cost 90% less than regular input tokens
• Latency improvements: Cached content is processed faster
• Ideal for: Long system prompts, extensive tool definitions, multi-turn conversations

Without caching, every API call processes the entire prompt from scratch. With caching, repeated content (system prompts, tools, conversation history) can be cached and reused across requests.

How It Works

Anthropic caches the trailing portion of your prompt context. The cache is identified by an exact match of the content hashcode. Any change to the cached portion invalidates the cache.

Key concepts:

• Cache creation: First time content is seen, it’s written to cache with a 25% premium over regular input tokens (for 5-minute TTL)
• Cache reads: Subsequent requests with matching content read from cache at 10% of regular input token cost
• Cache TTL: 5 minutes (default) or 1 hour (premium, higher creation cost)
• Minimum size: 1024 tokens for older models, 4096 tokens for Claude Sonnet 4.5 and newer.

Cache Strategies

Embabel provides several caching strategies through AnthropicCachingConfig:

System Prompt Caching

Cache the system prompt for reuse across multiple requests:

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setSystemPrompt(true);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

Tools Caching

Cache tool definitions when using many tools or tools with large schemas:

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setTools(true);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

System + Tools Caching

Combine both strategies:

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setSystemPrompt(true);
cachingConfig.setTools(true);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

Conversation History Caching

Cache conversation history for long multi-turn conversations:

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setConversationHistory(true);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

Advanced Configuration

Message Type Minimum Content Length

Control which messages are eligible for caching based on their content length:

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setSystemPrompt(true);
cachingConfig.messageTypeMinContentLength(MessageRole.SYSTEM, 1024);
cachingConfig.messageTypeMinContentLength(MessageRole.USER, 512);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

Message Type TTL

Set cache TTL per message type (default is 5 minutes):

AnthropicCachingConfig cachingConfig = new AnthropicCachingConfig();
cachingConfig.setSystemPrompt(true);
cachingConfig.messageTypeTtl(MessageRole.SYSTEM, AnthropicCacheTtl.ONE_HOUR);

LlmOptions options = LlmOptions.withDefaultLlm();
options = withAnthropicCaching(options, cachingConfig);

Accessing Cache Metrics

Embabel provides extension methods to access Anthropic-specific cache metrics from the Usage object:

import static com.embabel.agent.config.models.anthropic.AnthropicUsage.*;

AssistantMessage response = promptRunner.respond(messages);
Usage usage = response.getUsage();

// Check if cache was created or read
boolean cacheCreated = hasAnthropicCacheCreation(usage);
boolean cacheRead = hasAnthropicCacheRead(usage);

// Get token counts
Integer creationTokens = anthropicCacheCreationTokens(usage);
Integer readTokens = anthropicCacheReadTokens(usage);

// Get summary string for logging
String summary = anthropicCacheSummary(usage);
// Example output: "cache[creation=1061, read=0]"

Best Practices

1. Cache long, stable content: System prompts and tool definitions that don’t change frequently are ideal candidates
2. Mind the minimum size: Content must meet the minimum token requirement (1024 or 4096 depending on model)
3. Monitor cache metrics: Use the cache extension methods to track cache hit rates and validate savings
4. Consider TTL vs cost: 1-hour TTL has higher creation cost but better for longer sessions
5. Test before deploying: Cache behavior can vary based on prompt structure and usage patterns

Reference

For complete details on Anthropic’s prompt caching, see: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

Advanced Feature: Native Structured Output

Native structured output enables the model provider to enforce a JSON Schema directly, rather than relying only on prompt instructions and Embabel’s object construction. Embabel still owns schema generation and object binding; the native provider path is an optimization and compatibility feature used only when it is explicitly supported and safe for the call.

Payload Format

For OpenAI-compatible providers, Spring AI wraps Embabel’s schema in a response_format JSON schema payload. For example, a MonthItem object can produce a request fragment like:

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "MonthItem",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "temperature": { "type": "integer" }
        },
        "required": [ "name", "temperature" ],
        "additionalProperties": false
      }
    }
  },
}

OpenAI validates this schema before generation. If the schema is not valid for the provider-native path, the request fails before the model produces output.

Participants Roles

• Embabel generates the JSON Schema, decides whether the native path is enabled for a call, and parses the final response into the requested object.
• Spring AI wraps Embabel’s schema into provider-specific format options, such as OpenAI’s response_format, and serializes the HTTP request.
• The provider such as OpenAI validates the native structured-output payload and returns output that matches the accepted schema.

Configuration and API

Native structured output is advertised in model metadata:

native_support_defaults:
  structured_output:
    supported: true
    strategy: response_format
    strict: false
    prompt_instructions: include

For a specific call, enable it through LlmOptions:

PromptRunner runner = ai.withLlm(
        NativeStructuredOutputMode.ENABLED.applyTo(LlmOptions.withDefaultLlm())
);

MonthItem result = runner.createObject(prompt, MonthItem.class);

In Java, use static imports to keep additional LlmOptions configuration in the same chain:

import static com.embabel.common.ai.model.NativeStructuredOutputMode.ENABLED;
import static com.embabel.common.ai.model.NativeStructuredOutputModeKt.withNativeStructuredOutput;

var options = LlmOptions.withDefaultLlm();
options = withNativeStructuredOutput(options, ENABLED)
        .withTemperature(0.8);

PromptRunner runner = ai.withLlm(options);

• NativeStructuredOutputMode.ENABLED: Use the native path unconditionally when the provider supports it, bypassing the schema compatibility check.
• NativeStructuredOutputMode.DEFAULT: Check schema compatibility first; fall back to prompt-based extraction with a DEBUG log if the schema is not compatible. Use ENABLED to force the native path regardless, or DISABLED to silence the log.
• NativeStructuredOutputMode.DISABLED: Always use prompt-based extraction; never use the native path.

For Java objects, mark required reference fields explicitly:

static class MonthItem {
    @JsonProperty(required = true)
    private String name;

    @JsonProperty(required = true)
    private Integer temperature;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public Integer getTemperature() {
        return temperature;
    }

    public void setTemperature(Integer temperature) {
        this.temperature = temperature;
    }
}

Limitations

• Required fields: OpenAI native structured output requires required to include every property. Optional Java reference fields are not treated as required unless annotated, for example with @JsonProperty(required = true) or @NotNull.
• Optional values: Provider-native optionality usually needs nullable required properties, for example type: ["string", "null"], rather than omitting the property from required.
• Arrays of objects: Some providers, including OpenAI according to Spring AI documentation, do not support arrays of objects natively. Embabel falls back to normal object construction for unsupported shapes.
• Streaming: Native structured output is not supported for streaming by Spring AI currently
• ifPossible APIs: createObjectIfPossible and related paths use MaybeReturn semantics and stay on Embabel’s fallback object construction path.
• Provider coverage: OpenAI-compatible response_format is the primary supported path. Anthropic native structured output is currently disabled by default until its semantics are verified.