Building Chatbots

Chatbots are an important application of Gen AI, although far from the only use, especially in enterprise.

Unlike many other frameworks, Embabel does not maintain a conversation thread to do its core work. This is a good thing as it means that context compression is not required for most tasks.

If you want to build a chatbot you should use the Conversation interface explicitly, and expose a Chatbot bean, typically backed by action methods that handle UserMessage events.

Core Concepts

Long-Lived AgentProcess

An Embabel chatbot is backed by a long-lived AgentProcess that pauses between user messages. This design has important implications:

• The same AgentProcess can respond to events besides user input
• The blackboard maintains state across the entire session
• Actions can be triggered by user messages, system events, or other objects added to the blackboard
• It’s a working context rather than just a chat session

When a user sends a message, it’s added to the blackboard as a UserMessage. The AgentProcess then runs, selects an appropriate action to handle it, and pauses again waiting for the next event.

Utility AI for Chatbots

Utility AI is often the best approach for chatbots. Instead of defining a fixed flow, you define multiple actions with costs, and the planner selects the highest-value action to respond to each message.

This allows:

• Multiple response strategies (e.g., RAG search, direct answer, clarification request)
• Dynamic behavior based on context
• Easy extensibility by adding new action methods

Goals in Chatbots

Typically, chatbot agents do not need a goal. The agent process simply waits for user messages and responds to them indefinitely.

However, you can define a goal if you want to ensure the conversation terminates and the AgentProcess completes rather than waiting forever. This is useful for:

• Transactional conversations (e.g., completing a booking)
• Wizard-style flows with a defined endpoint
• Conversations that should end after collecting specific information

Key Interfaces

Chatbot

The Chatbot interface manages multiple chat sessions:

public interface Chatbot {
    ChatSession createSession(
        User user,
        OutputChannel outputChannel,
        String contextId,
        String conversationId
    );

    ChatSession findSession(String conversationId);
}

Context IDs and Session State

The contextId parameter allows you to pre-populate the session’s blackboard with objects from a named context. This is useful when:

• Users have multiple contexts - A user might have different projects, accounts, or workspaces. Each context can maintain its own state that persists across sessions.
• Resuming prior state - When a user returns, you can restore their previous session state (e.g., user preferences, in-progress work, conversation history from a previous session).
• Pre-loading domain objects - You can populate the blackboard with objects that should always be present, such as the current user’s profile, active subscription, or relevant configuration.

// Create a session with a specific context
ChatSession session = chatbot.createSession(
    user,
    outputChannel,
    "project-alpha",  // Context ID - loads saved state for this project
    null
);

// Or create an anonymous session without context
ChatSession anonymousSession = chatbot.createSession(
    null,
    outputChannel,
    null,
    null
);

The context mechanism works with `AgentPlatform’s context storage:

1. When createSession is called with a contextId, the platform looks up any saved objects for that context
2. Those objects are added to the new session’s blackboard
3. As the session runs, changes to the blackboard can be persisted back to the context
4. The next time a session is created with that contextId, the updated state is restored

This enables stateful conversations across sessions without requiring the chatbot to manually track and restore state.

ChatSession

Each session represents an ongoing conversation:

public interface ChatSession {
    OutputChannel getOutputChannel();
    User getUser();
    Conversation getConversation();
    String getProcessId();

    void onUserMessage(UserMessage userMessage);
    boolean isFinished();
}

Conversation

The Conversation interface holds the message history and tracks assets:

public interface Conversation extends StableIdentified, AssetView {
    List<Message> getMessages();
    AssetTracker getAssetTracker();
    List<Asset> getAssets();  // Combined view of all assets
    Message addMessage(Message message);
    UserMessage lastMessageIfBeFromUser();
}

Message types include:

• UserMessage - messages from the user (supports multimodal content)
• AssistantMessage - responses from the chatbot (can include assets)
• SystemMessage - system-level instructions

Asset Tracking

Chatbots can track assets—structured outputs like generated documents, search results, or user-created content—at two levels:

Conversation-Level Assets

The Conversation has an AssetTracker for explicitly tracking assets throughout the session:

// Add an asset to the conversation tracker
conversation.getAssetTracker().addAsset(myAsset);

// Get all tracked assets
List<Asset> trackedAssets = conversation.getAssetTracker().getAssets();

Use conversation-level tracking when:

• Assets are created by tools or external processes
• Assets should persist across multiple messages
• You want explicit control over what’s tracked

Message-Level Assets

AssistantMessage implements AssetView and can include assets directly:

AssistantMessage message = new AssistantMessage(
    "Here's the report you requested",
    null,  // name
    null,  // awaitable
    List.of(reportAsset, summaryAsset)  // assets
);
conversation.addMessage(message);

Use message-level assets when:

• Assets are directly tied to a specific response
• You want assets to appear alongside the message in the UI
• The asset represents output from that specific interaction

Combined Asset View

The Conversation.assets property provides a merged view of all assets:

// Gets assets from BOTH the tracker AND all messages
List<Asset> allAssets = conversation.getAssets();

The merge follows these rules:

1. Tracker assets appear first (explicit tracking takes priority)
2. Message assets follow in chronological order
3. Duplicates are removed by ID (tracker version wins)

This allows flexible asset management:

@Action(canRerun = true, trigger = UserMessage.class)
void respond(Conversation conversation, ActionContext context) {
    // Create an asset from the response
    Asset resultAsset = createResultAsset(result);

    // Option 1: Add to message (appears with this response)
    var message = new AssistantMessage(
        "Here's your analysis",
        null, null,
        List.of(resultAsset)
    );
    conversation.addMessage(message);

    // Option 2: Add to tracker (explicitly tracked)
    conversation.getAssetTracker().addAsset(resultAsset);

    // Either way, it's visible via conversation.getAssets()
}

Using Assets as Tools

Assets can be exposed to the LLM as tools via their LlmReference:

// Get references from recent assets
List<LlmReference> refs = conversation.mostRecent(5).references();

// Use in a prompt
var response = context.ai()
    .withReferences(refs)  // Assets become available as tools
    .respond(conversation.getMessages());

This enables scenarios like:

• Editing previously generated content
• Combining multiple assets
• Querying structured data from earlier in the conversation

Building a Chatbot

Step 1: Create Action Methods

Define action methods in an @EmbabelComponent that respond to user messages using the trigger parameter:

@EmbabelComponent
public class ChatActions {

    private final ToolishRag toolishRag;
    private final RagbotProperties properties;

    public ChatActions(
            SearchOperations searchOperations,
            RagbotProperties properties) {
        this.toolishRag = new ToolishRag(
                "sources",
                "Sources for answering user questions",
                searchOperations
        );
        this.properties = properties;
    }

    @Action(canRerun = true, trigger = UserMessage.class) // ① ②
    void respond(
            Conversation conversation, // ③
            ActionContext context) {
        var assistantMessage = context.ai()
                .withLlm(properties.chatLlm())
                .withReference(toolishRag)
                .rendering("ragbot")
                .respondWithSystemPrompt(conversation, Map.of(
                        "properties", properties
                ));
        context.sendMessage(conversation.addMessage(assistantMessage)); // ④
    }
}

1. trigger = UserMessage.class - action is invoked when a UserMessage is the last object added to the blackboard
2. canRerun = true - action can be executed multiple times (for each user message)
3. Conversation parameter is automatically injected from the blackboard
4. context.sendMessage() sends the response to the output channel

Step 2: Configure the Chatbot Bean

Use AgentProcessChatbot.utilityFromPlatform() to create a utility-based chatbot that discovers all available actions:

@Configuration
class ChatConfiguration {

    @Bean
    Chatbot chatbot(AgentPlatform agentPlatform) {
        return AgentProcessChatbot.utilityFromPlatform(agentPlatform); // ① ②
    }
}

1. Creates a chatbot using Utility AI planning to select the best action
2. Discovers all @Action methods from @EmbabelComponent classes on the platform

For debugging, you can pass a custom Verbosity configuration:

@Bean
Chatbot chatbot(AgentPlatform agentPlatform) {
    return AgentProcessChatbot.utilityFromPlatform(
            agentPlatform,
            new InMemoryConversationFactory(), // ①
            new Verbosity().showPrompts()      // ②
    );
}

1. Conversation factory (required when specifying verbosity)
2. Verbosity configuration for debugging prompts

Otherwise the actions needed to respond to chat may not be available in the session.

Conversation Storage

By default, chatbots use in-memory conversations that are lost when the session ends. For production applications, you typically want to persist conversations to a backing store.

Storage Types

Embabel supports two conversation storage types via ConversationStoreType:

Configuring Persistent Storage

To use persistent conversations, inject ConversationFactoryProvider and pass the appropriate factory when creating the chatbot:

@Configuration
class ChatConfiguration {

    @Bean
    Chatbot chatbot(
            AgentPlatform agentPlatform,
            ConversationFactoryProvider conversationFactoryProvider) { // ①

        ConversationFactory factory = conversationFactoryProvider
                .getFactory(ConversationStoreType.STORED); // ②

        return new AgentProcessChatbot(
                agentPlatform,
                user -> createAgent(agentPlatform),
                factory,  // ③
                // ... other configuration
        );
    }
}

1. Inject the ConversationFactoryProvider via Spring DI
2. Get the factory for the desired storage type
3. Pass the factory to the chatbot - storage is configured once at creation time

This ensures consistent behavior across all sessions.

Adding embabel-chat-store

To enable persistent storage, add the embabel-chat-store dependency:

<dependency>
    <groupId>com.embabel.chat</groupId>
    <artifactId>embabel-chat-store</artifactId>
</dependency>

This provides:

• StoredConversationFactory - creates conversations that persist to Neo4j
• StoredConversation - conversation implementation with async persistence
• Message lifecycle events (MessageEvent) for UI updates
• Title generation for conversation sessions

Restoring Conversations

To restore a conversation, pass the conversationId when creating a session:

// Restore existing conversation or create new one
ChatSession session = chatbot.createSession(
    user,
    outputChannel,
    null,            // contextId
    conversationId   // ①
);

// Messages are already loaded if conversation existed
List<Message> history = session.getConversation().getMessages();

1. If the conversation exists in storage, it will be loaded automatically. If not, a new conversation is created with this ID.

This allows applications to:

• Resume conversations across server restarts
• Display conversation history to returning users
• Continue multi-turn interactions from where they left off

Step 3: Use the Chatbot

Interact with the chatbot through its session interface:

// New session (fresh state, generated conversation ID)
ChatSession session = chatbot.createSession(user, outputChannel, null, null); // ①

// Session with context (restores blackboard state)
ChatSession withContext = chatbot.createSession(user, outputChannel, "user-workspace-123", null); // ②

// Restore existing conversation by ID
ChatSession restored = chatbot.createSession(user, outputChannel, null, savedConversationId); // ③

// Both context and conversation restoration
ChatSession full = chatbot.createSession(user, outputChannel, "user-workspace-123", savedConversationId); // ④

session.onUserMessage(new UserMessage("What does this document say about taxes?")); // ⑤
// Response is automatically sent to the outputChannel

1. Create a new session with fresh blackboard and auto-generated conversation ID
2. Load prior blackboard state from the "user-workspace-123" context
3. Restore an existing conversation with its message history
4. Both: load context state AND restore conversation history
5. Send a user message - triggers the agent to select and run an action

How Message Triggering Works

When you specify trigger = UserMessage.class on an action:

1. The chatbot adds the UserMessage to both the Conversation and the AgentProcess blackboard
2. The planner evaluates all actions whose trigger conditions are satisfied
3. For utility planning, the action with the highest value (lowest cost) is selected
4. The action method receives the Conversation (with the new message) via parameter injection

This trigger-based approach means:

• You can have multiple actions that respond to user messages with different costs
• The planner picks the most appropriate response strategy
• Actions can also be triggered by other event types (not just UserMessage)

Dynamic Cost Methods

For more sophisticated action selection, use @Cost methods:

@Cost // ①
double dynamic(Blackboard bb) { // ②
    return bb.getObjects().size() > 5 ? 100 : 10; // ③
}

@Action(canRerun = true,
        trigger = UserMessage.class,
        costMethod = "dynamic") // ④
void respond(Conversation conversation, ActionContext context) {
    // ...
}

1. @Cost marks this as a cost calculation method
2. Receives the Blackboard to inspect current state
3. Returns cost value - lower costs mean higher priority
4. costMethod links the action to the cost calculation method

Prompt Templates

Chatbots typically use Jinja prompt templates rather than inline string prompts. This isn’t strictly necessary—simple chatbots can use regular string prompts built in code:

var assistantMessage = context.ai()
        .withLlm(properties.chatLlm())
        .withSystemPrompt("You are a helpful assistant. Answer questions concisely.") // ①
        .respond(conversation.getMessages());

1. Simple inline prompt - fine for basic chatbots

However, production chatbots often need longer, more complex prompts for:

• Personality and tone (personas)
• Guardrails and safety instructions
• Domain-specific objectives
• Dynamic behavior based on configuration

For these cases, Jinja templates are the better choice:

var assistantMessage = context.ai()
        .withLlm(properties.chatLlm())
        .withReference(toolishRag)
        .rendering("ragbot") // ①
        .respondWithSystemPrompt(conversation, Map.of( // ②
                "properties", properties,
                "persona", properties.persona(),
                "objective", properties.objective()
        ));

1. Loads prompts/ragbot.jinja from resources
2. Template bindings - accessible in Jinja as properties.persona() etc.

Templates allow:

• Separation of prompt engineering from code
• Dynamic persona and objective selection via configuration
• Reusable prompt elements (guardrails, personalization)
• Prompt iteration without code changes

Resilient Responses with respond

In a chatbot, it’s critical never to leave the user without a reply. The respond method on Rendering wraps respondWithSystemPrompt with error handling, so that an LLM or infrastructure failure still returns an AssistantMessage to the user rather than propagating an exception:

var assistantMessage = context.ai()
        .rendering("ragbot")
        .respond(conversation, model, error -> {
            logger.error("Failed to generate response", error);
            return new AssistantMessage("Sorry, something went wrong. Please try again.");
        });

Template Structure Example

A typical chatbot template structure from the rag-demo project:

prompts/
├── ragbot.jinja                    # Main entry point
├── elements/
│   ├── guardrails.jinja            # Safety restrictions
│   └── personalization.jinja       # Dynamic persona/objective loader
├── personas/
│   ├── clause.jinja                # Legal expert persona
│   └── ...
└── objectives/
    └── legal.jinja                 # Legal document analysis objective

The main template (ragbot.jinja) composes from reusable elements:

{% include "elements/guardrails.jinja" %} // ①

{% include "elements/personalization.jinja" %} // ②

1. Include safety guardrails first
2. Then include persona and objective (which are dynamically selected)

Guardrails define safety boundaries (elements/guardrails.jinja):

{# Safety and content guardrails for the ragbot. #}

DO NOT DISCUSS POLITICS OR CONTROVERSIAL TOPICS.

Personalization dynamically loads persona and objective (elements/personalization.jinja):

{% set persona_template = "personas/" ~ properties.persona() ~ ".jinja" %} // ①
{% include persona_template %}

{% set objective_template = "objectives/" ~ properties.objective() ~ ".jinja" %} // ②
{% include objective_template %}

1. Build template path from properties.persona() (e.g., "clause" → "personas/clause.jinja")
2. Build template path from properties.objective() (e.g., "legal" → "objectives/legal.jinja")

A persona template (personas/clause.jinja):

Your name is Clause.
You are a brilliant legal chatbot who excels at interpreting
legislation and legal documents.

An objective template (objectives/legal.jinja):

You are an authoritative interpreter of legislation and legal documents.
You are renowned for thoroughness and for never missing anything.

You answer questions definitively, in a clear and concise manner.
You cite relevant sections to back up your answers.
If you don't know, say you don't know.
NEVER FABRICATE ANSWERS.

You ground your answers in literal citations from the provided sources.
Always use the available tools. // ①

1. Instructs the LLM to use RAG tools provided via withReference()

This modular approach lets you:

• Switch personas via application.yml without code changes
• Share guardrails across multiple chatbot configurations
• Test different objectives independently

Advanced: State Management with @State

For complex chatbots that need to track state across messages, use @State classes. State classes are automatically managed by the agent framework:

• State objects are persisted in the blackboard
• Actions can depend on specific state being present
• State transitions drive the conversation flow

Cross-reference the @State annotation documentation for details on:

• Defining state classes
• State-dependent actions
• Nested state machines

Complete Example

See the rag-demo project for a complete chatbot implementation including:

• ChatActions.java - Action methods responding to user messages
• ChatConfiguration.java - Chatbot bean configuration
• RagbotShell.java - Spring Shell integration for interactive testing
• Jinja templates for persona-driven prompts
• RAG integration for document-grounded responses

To run the example:

./scripts/shell.sh

# In the shell:
ingest ./data/document.md
chat
> What does the document say about...