Customizing Embabel

Adding LLMs

You can add custom LLMs as Spring beans by implementing the LlmService interface. Embabel provides SpringAiLlmService for wrapping Spring AI ChatModel instances.

Using SpringAiLlmService

SpringAiLlmService implements the LlmService interface and provides framework-agnostic LLM operations including support for the Embabel tool loop and message sender abstraction.

@Configuration
public class LlmsConfig {
    @Bean
    public LlmService<?> myLlm() {
        org.springframework.ai.chat.model.ChatModel chatModel = ...
        return new SpringAiLlmService(
                "myChatModel",              // ①
                "myChatModelProvider",      // ②
                chatModel)                  // ③
            .withOptionsConverter(new MyLlmOptionsConverter()) // ④
            .withKnowledgeCutoffDate(LocalDate.of(2025, 4, 1)); // ⑤
    }
}

1. The name of the LLM (used for model selection).
2. The provider name, such as "OpenAI" or "Anthropic".
3. The Spring AI ChatModel instance.
4. Customize with an OptionsConverter implementation to convert Embabel LlmOptions to Spring AI ChatOptions.
5. Set the knowledge cutoff date if available.

LLM Configuration Options

SpringAiLlmService supports the following configuration:

• name (required)
• provider, such as "Mistral" (required)
• OptionsConverter to convert Embabel LlmOptions to Spring AI ChatOptions
• knowledge cutoff date (if available)
• any additional PromptContributor objects to be used in all LLM calls. If knowledge cutoff date is provided, add the KnowledgeCutoffDate prompt contributor.
• pricing model (if available)

A common requirement is to add an OpenAI-compatible LLM. This can be done by extending the OpenAiCompatibleModelFactory class as follows:

@Configuration
public class CustomOpenAiCompatibleModels extends OpenAiCompatibleModelFactory {

    public CustomOpenAiCompatibleModels(
            @Value("$\{MY_BASE_URL:#\{null}}") String baseUrl,
            @Value("$\{MY_API_KEY}") String apiKey,
            ObservationRegistry observationRegistry) {
        super(baseUrl, apiKey, observationRegistry);
    }

    @Bean
    public LlmService<?> myGreatModel() {
        // Call superclass method
        return openAiCompatibleLlm(
            "my-great-model",
            "me",
            LocalDate.of(2025, 1, 1),
            new PerTokenPricingModel(0.40, 1.6)
        );
    }
}

Adding embedding models

Embedding models can also be added as beans of the Embabel type EmbeddingService. Use the SpringAiEmbeddingService class to wrap a Spring AI EmbeddingModel.

Typically, this is done in an @Configuration class like this:

@Configuration
public class EmbeddingModelsConfig {
    @Bean
    public EmbeddingService myEmbeddingModel() {
        org.springframework.ai.embedding.EmbeddingModel embeddingModel = ...
        return new SpringAiEmbeddingService(
                "myEmbeddingModel",
                "myEmbeddingModelProvider",
                embeddingModel);
    }
}

Bring Your Own Key (BYOK)

By default, Embabel resolves LLMs through autoconfiguration: you set one or more API keys as an environment variable or property (e.g. ANTHROPIC_API_KEY), and the relevant autoconfigure module registers a pool of LlmService beans at startup. This is the right approach for a platform-level key shared across all users.

BYOK is for cases where the key is not known at startup, or where you want to resolve an LlmService on the fly:

• User-supplied keys — each user provides their own API key; the application must validate it and wire it into the prompt runner for that session.
• End-to-end testing — spin up a real LlmService with a dedicated test key outside a full Spring context.
• Multi-tenant or cost-controlled apps — select a provider dynamically based on per-tenant configuration or available quota.

Embabel provides factory classes that validate a key and return a ready LlmService, plus a detectProvider() utility that concurrently probes multiple providers and returns the first that accepts the key.

Embabel does not store, log, or otherwise manage the key — the validated LlmService is returned to the caller, who is responsible for secure key handling: transmission over HTTPS only, no plaintext logging, limiting key lifetime, and revoking cached services on logout or key rotation.

Building a validated service (known provider)

Use this when the provider is already known — for example, a per-provider field in a settings UI.

// Anthropic
val service: LlmService<*> = AnthropicModelFactory(apiKey = userKey).buildValidated()

// OpenAI
val service: LlmService<*> = OpenAiCompatibleModelFactory.openAi(userKey).buildValidated()

// DeepSeek (OpenAI-compatible endpoint)
val service: LlmService<*> = OpenAiCompatibleModelFactory.deepSeek(userKey).buildValidated()

// Mistral (OpenAI-compatible endpoint)
val service: LlmService<*> = OpenAiCompatibleModelFactory.mistral(userKey).buildValidated()

// Gemini (OpenAI-compatible endpoint)
val service: LlmService<*> = OpenAiCompatibleModelFactory.gemini(userKey).buildValidated()

// Atlas Cloud (OpenAI-compatible endpoint)
val service: LlmService<*> = OpenAiCompatibleModelFactory.atlasCloud(userKey).buildValidated()

buildValidated() makes a single probe call with no retries. On success it returns a production LlmService; on failure it throws InvalidApiKeyException.

A blank key is treated as absent and rejected before any network call, also with InvalidApiKeyException. This matters more than it sounds: Docker Compose passes ANTHROPIC_API_KEY=$\{ANTHROPIC_API_KEY:-}, so in a container the variable is routinely set-but-empty, and a caller reading it with a null default — @Value("$\{ANTHROPIC_API_KEY:#\{null}}") or System.getenv(...) — gets "" rather than null. That empty string passes a null check and would otherwise reach the provider, coming back as an opaque authentication failure far from its cause. Callers therefore do not need to pre-check for blank keys themselves.

required (see ) and still fails at startup when one is missing — putting a provider starter on the classpath is a statement that the deployment has that key.

Auto-detecting the provider

Use this when a user pastes a key without specifying a provider — for example, a sign-up flow that accepts keys from any supported provider.

detectProvider() races the candidates concurrently using virtual threads and returns the first LlmService that validates successfully. The detected provider is available as service.provider on the result.

val service: LlmService<*> = detectProvider(
    AnthropicModelFactory(apiKey = userKey),
    OpenAiCompatibleModelFactory.openAi(userKey),
    OpenAiCompatibleModelFactory.deepSeek(userKey),
    OpenAiCompatibleModelFactory.mistral(userKey),
    OpenAiCompatibleModelFactory.gemini(userKey),
    OpenAiCompatibleModelFactory.atlasCloud(userKey),
)
val detectedProvider: String = service.provider

A single-argument call is valid — it validates against one provider without concurrency, which is the right path for a settings flow where the provider is known but you still want `detectProvider’s consistent error handling.

val service: LlmService<*> = detectProvider(AnthropicModelFactory(apiKey = userKey))

If all candidates throw InvalidApiKeyException, detectProvider also throws InvalidApiKeyException.

Overriding the validation model

Each factory validates the key using a default model (e.g. gpt-4.1-mini for OpenAI, claude-haiku-4-5 for Anthropic). Override this if the key only grants access to a specific set of models:

// OpenAI — use a different model tier for the probe
OpenAiCompatibleModelFactory.openAi(userKey)
    .validating(OpenAiModels.GPT_41_NANO, OpenAiModels.PROVIDER)

// Anthropic — set validation model at construction time
AnthropicModelFactory(apiKey = userKey, validationModel = AnthropicModels.CLAUDE_SONNET_4_5)

Adding support for another provider

Any provider that exposes an OpenAI-compatible HTTP API can be added as a one-liner extension function on OpenAiCompatibleModelFactory.Companion:

fun OpenAiCompatibleModelFactory.Companion.acme(apiKey: String) =
    OpenAiCompatibleModelFactory.byok(
        baseUrl = "https://api.acme.example.com/v1",
        apiKey = apiKey,
        validationModel = "acme-small",    // ①
        validationProvider = "Acme",       // ②
    )

1. The cheapest model available on the provider, used for the key-validation probe.
2. The provider name; returned as service.provider after detection.

The extension function integrates with detectProvider like any built-in factory:

val service = detectProvider(
    AnthropicModelFactory(apiKey = userKey),
    OpenAiCompatibleModelFactory.openAi(userKey),
    OpenAiCompatibleModelFactory.acme(userKey),
)

Embedding services

Embedding models are resolved through the same BYOK mechanism, so a key that arrives after startup can activate an embedding model without a restart.

// OpenAI
val embeddings: EmbeddingService =
    OpenAiCompatibleModelFactory.openAiEmbedding(userKey, "text-embedding-3-small")
        .buildValidated()

// Any OpenAI-compatible provider
val acmeEmbeddings: EmbeddingService = OpenAiCompatibleModelFactory.byokEmbedding(
    baseUrl = "https://api.acme.example.com/v1",
    apiKey = userKey,
    model = "acme-embed-small",
    provider = "Acme",
).buildValidated()

buildValidated() embeds a short probe text. On failure it throws InvalidApiKeyException. A blank key is treated as absent and rejected before any network call — in a container OPENAI_API_KEY is routinely set-but-empty, and that empty string would otherwise reach the provider and come back as an opaque authentication error. The message names the model: an embedding model name is caller-supplied and mandatory here, so a typo in it reaches the provider the same way a bad key does. On success the returned EmbeddingService already knows its dimension — it was observed during the probe — so no live dimensions() call is made later.

Two things work differently from LLM BYOK, because an embedding model is a schema commitment rather than a stateless per-call service:

• No provider detection. The vector index is built at a fixed dimension and every stored vector must come from the same model at that dimension. Whichever provider happens to accept a pasted key first must not be allowed to decide that, so there is no embedding equivalent of detectProvider() and the model is always named explicitly.
• Installation or world scope, not per-user. Per-user LLM keys are coherent; per-user embedding keys are not, because vectors from different models would land in one shared index. The dimension belongs to the store, not to the user.

The dimension is never configured on the BYOK path — it is whatever the model actually returned during validation. When writing into an index that already exists, compare the two yourself before storing anything, since vectors of different widths cannot share an index:

val embeddings = OpenAiCompatibleModelFactory
    .openAiEmbedding(userKey, "text-embedding-3-small")
    .buildValidated()

check(embeddings.dimensions == index.dimensions) {
    "Index is $\{index.dimensions}-dimensional; $\{embeddings.name} returns $\{embeddings.dimensions}"
}

only builds and validates the service.

Using the validated service

Once you have an LlmService, pass it directly to PromptRunner or Ai via withLlmService():

LlmService<?> userLlm = ... // from buildValidated() or detectProvider()
promptRunner
    .withLlmService(userLlm)
    .creating(MyOutput.class)
    .create(messages);

Internally this flows through the same model selection path as all other LLM resolution via PreResolvedModelSelectionCriteria — no separate resolution path is needed.

Pure BYOK deployments

Everything above assumes the application already has an LlmService to fall back on. A pure BYOK deployment has none: it ships with no provider key at all and obtains one at runtime, per user or per tenant. Two things get in its way, and embabel-agent-starter-byok addresses both.

Depend on the BYOK starter, not a provider starter. The provider starters bundle their autoconfiguration, which requires an API key at startup — exactly what this deployment does not have. The BYOK starter carries the model factories and none of that autoconfiguration:

<dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-byok</artifactId>
</dependency>

Point default-llm at the placeholder. With no models registered, the platform has nothing to resolve embabel.models.default-llm to. The starter contributes a placeholder LlmService named setup-required for precisely this case:

embabel:
  models:
    default-llm: setup-required

The name is available as the constant SetupRequiredLlm.NAME. Opting in through configuration is deliberate — the placeholder is registered unconditionally but is only ever selected if you name it, so adding the starter alongside a real provider changes nothing.

The placeholder completes nothing. Any call that reaches it throws NoLlmConfiguredException, so a missing key surfaces as an actionable error rather than a silently empty answer. Catch it to render your own "add an API key" experience:

try {
    promptRunner.withLlmService(userLlm).creating(MyOutput::class.java).create(messages)
} catch (e: NoLlmConfiguredException) {
    // no key for this user yet — send them to your settings page
}

In normal operation the placeholder is never reached: resolve the user’s key, build a service with one of the factories above, and pass it to withLlmService().

Point default-embedding-model at the embedding placeholder, if the application uses RAG or memory. Anything that resolves the default embedding service while its own beans are being created — which is what RAG and memory components do — would otherwise fail the context refresh, for the same reason the LLM side did before its placeholder existed:

embabel:
  models:
    default-llm: setup-required
    default-embedding-model: setup-required-embedding

The name is available as SetupRequiredEmbedding.NAME. Calls that reach it throw NoEmbeddingServiceConfiguredException, the embedding counterpart of NoLlmConfiguredException.

other call rather than returning a number, and this is deliberate. Any value it returned would be used to create a vector index; writes to that index would succeed; and a real model configured later would silently disagree with everything already stored. That failure surfaces as bad search results rather than as an error, which is far worse than not starting.

So code that provisions a vector index must ask whether there is a model yet rather than call dimensions and handle the failure — a thrown exception cannot be told apart from a provider that is merely unreachable right now. Test for the marker and skip:

if (embeddingService.awaitingProviderKey) {
    // No model yet. Skip provisioning — do NOT invent a dimension.
    logger.info("Awaiting an embedding key; vector index not provisioned yet")
} else {
    provisioner.ensure(VectorIndexSpec(label, "embedding", embeddingService.dimensions))
}

Three things about that check are easy to get wrong, and each costs an afternoon.

Ask awaitingProviderKey, never is PlaceholderEmbeddingService. Wrapping is common — event tracking decorates the configured service, and applications add layers for hot-swapping the model, metering, or per-tenant routing. A wrapper around a placeholder is not itself a placeholder, so a type test answers "there is a model" and the caller commits an index at a dimension it obtained by catching an exception. awaitingProviderKey is a property of the interface, so Kotlin’s by delegation forwards it through any depth of wrapping at no cost.

must forward awaitingProviderKey explicitly. It defaults to false, so a wrapper that forgets reports "there is a model" on behalf of one that cannot embed.

Re-resolve the reference; do not hold it. The marker lives on the object, so a held reference that is the placeholder stays the placeholder for the life of the process — re-checking it can never notice the key arriving, and recovery needs a restart. A component that wants to start working without one should take a supplier and ask the platform each time:

class MyProvisioner(private val embeddingService: () -> EmbeddingService) {
    fun ensure() {
        val embeddings = embeddingService()          // asks again; the answer changes
        if (embeddings.isAwaitingKey()) return
        // ... provision at embeddings.dimensions
    }
}

The marker answers "is there a model yet", not "is it still the same one". Once a real model is registered the placeholder is out of the picture, and a later change to a model of a different width is a different problem: the existing index keeps its original dimension, and vectors written under the new model disagree with everything already stored. That is drift, and it needs the corpus re-embedded — see the note under [](/Embedding services) above.

application’s responsibility, as described under above.

Error handling

try {
    val service = detectProvider(
        AnthropicModelFactory(apiKey = userKey),
        OpenAiCompatibleModelFactory.openAi(userKey),
    )
    // store or use service
} catch (e: InvalidApiKeyException) {
    // return 401 / surface error to user
    // no Spring AI types to unwrap
}

InvalidApiKeyException is in com.embabel.common.byok and carries no provider-specific implementation details.

Security considerations

The BYOK factories validate keys and return a ready LlmService — key lifecycle management is entirely the caller’s responsibility.

As a reference implementation, Guide holds keys in server-side memory only (UserKeyStore). When a key is validated, the client receives an AES-256-GCM encrypted blob — keyed by a secret known only to the server — for local-storage caching. A stolen blob is useless without the server’s decryption key. On page reload the client sends the blob back; the server decrypts it and restores the in-memory key. Keys are never written to disk or a database.

see .

Configuration via application.properties or application.yml

You can specify Spring configuration, your own configuration and Embabel configuration in the regular Spring configuration files. Profile usage will work as expected.

Customizing threading configuration

Embabel uses a dedicated executor for agent operations. You can customize the threading model using properties in agent-platform.properties:

# Flip inherited threading model (virtual-to-platform, platform-to-virtual)
embabel.agent.platform.threading.override=false

# Share app's executor when both use the same threading model
embabel.agent.platform.threading.shared=false

See for details about the threading model and configuration options.

Customizing logging

You can customize logging as in any Spring application.

For example, in application.properties you can set properties like:

logging.level.com.embabel.agent.a2a=DEBUG

You can also configure logging via a logback-spring.xml file if you have more sophisticated requirements.

See the Spring Boot Logging reference.

By default, many Embabel examples use personality-based logging experiences such as Star Wars. You can disable this by updating application.properties accordingly.

embabel.agent.logging.personality=severance

Remove the embabel.agent.logging.personality key to disable personality-based logging.

As all logging results from listening to events via an AgenticEventListener, you can also easily create your own customized logging.