Architecting Custom AI Chatbots: State, RAG, and Tool-Calling Workflows

Moving Beyond Stateless Prompt Wrappers

Building an AI chatbot in 2026 is no longer about sending a user prompt to an LLM endpoint and dumping the raw text into a chat bubble. Real-world users expect chatbots to remember prior context, answer questions against private documentation with high precision, call backend APIs to perform real actions, and stream low-latency responses without crashing under network jitter.

If you only wrap a standard completion API, your bot will suffer from hallucinations, runaway token costs, and context truncation bugs. Here is how modern software engineers architect production-grade conversational AI systems.

1. Managing Conversational State and Token Budgets

Large language models are inherently stateless. Every request must include the conversation history so the model knows what was previously discussed. However, as conversations grow, naively sending every past message will quickly exhaust your context window and dramatically increase API costs.

Production chatbots use three state management strategies:

• Sliding Context Windows: Keep only the system prompt plus the last N messages (typically the last 6 to 10 conversational turns). Older messages are dropped from the immediate prompt.
• Periodic Summary Compression: When message history exceeds a token threshold (for example, 4,000 tokens), trigger a background task that condenses the conversation history into a dense summary paragraph: "User is debugging a PostgreSQL timeout on AWS RDS and has already verified security group ports." Inject this summary into the system prompt.
• Entity and Preference Extraction: Store durable facts (such as user ID, preferred programming language, or account tier) in a relational database or Redis key-value store rather than relying on LLM memory.

2. Retrieval-Augmented Generation (RAG) Architecture

When users ask domain-specific questions about your company docs, proprietary APIs, or internal codebases, the base LLM does not have this information. RAG bridges this gap without requiring expensive model fine-tuning.

1. Ingestion and Chunking: Parse your documentation into clean markdown chunks (typically 400 to 600 tokens per chunk) with slight overlaps to preserve semantic continuity across headings.
2. Vector Embedding: Generate dense vector embeddings for each chunk using an embedding model and store them in a vector database such as pgvector, Pinecone, or Qdrant.
3. Hybrid Retrieval: When a user sends a prompt, combine vector semantic similarity with full-text keyword search (BM25). This ensures you catch both conceptual matches and exact product names or error codes.
4. Augmented Generation: Inject the top retrieved chunks into the prompt context under a strict instruction: "Answer the question using only the provided context. If the context does not contain the answer, say you do not know."

3. Tool Calling and Function Execution

A passive chatbot only talks; an agentic chatbot takes action. Modern models support structured function calling, allowing the LLM to output a JSON payload specifying which backend tool to execute.

// Defining a structured tool schema for an AI assistant
export const supportTools = [
  {
    name: "lookup_order_status",
    description: "Retrieves shipping status and tracking code for an order ID",
    parameters: {
      type: "object",
      properties: {
        orderId: {
          type: "string",
          description: "The customer order ID (e.g. ORD-9821)"
        }
      },
      required: ["orderId"]
    }
  }
];

When the user asks "Where is my order ORD-9821?", the model does not guess a response. It returns a tool call request:

{
  "name": "lookup_order_status",
  "arguments": { "orderId": "ORD-9821" }
}

Your backend executes the database query, fetches the tracking record, and sends the tool output back to the model. The model then formats a helpful, human-friendly response containing the real tracking link.

4. Production Streaming Server Implementation

Users hate staring at blank loading spinners for four seconds while an LLM generates a complete paragraph. Server-Sent Events (SSE) stream tokens to the frontend as soon as they are computed. Here is a clean Node.js implementation using standard Web Streams:

import { Request, Response } from "express";

export async function chatStreamHandler(req: Request, res: Response) {
  const { messages } = req.body;

  // Set SSE response headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  try {
    const stream = await llmClient.chat.completions.create({
      model: "gpt-4o",
      messages: [
        { role: "system", content: "You are a technical assistant for Dropout Developer." },
        ...messages
      ],
      stream: true
    });

    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content || "";
      if (content) {
        res.write(`data: ${JSON.stringify({ content })}\n\n`);
      }
    }

    res.write("data: [DONE]\n\n");
    res.end();
  } catch (error) {
    res.write(`data: ${JSON.stringify({ error: "Stream interrupted" })}\n\n`);
    res.end();
  }
}

5. Guardrails and Cost Protection

Deploying a public chatbot without rate limiting and input validation will quickly lead to abuse or inflated bills. Implement these safeguards before launching to users:

• Strict IP and User Rate Limits: Cap free users at 20 queries per hour using Redis token buckets.
• Prompt Injection Defense: Never allow raw user inputs to override your core system prompt. Delimit user inputs clearly using XML or markdown code blocks.
• Max Token Ceilings: Always pass an explicit max_tokens parameter to prevent malicious prompts from generating endless loops of text.

Designing modern chatbots is full-stack systems engineering. When you combine state compression, accurate hybrid RAG, structured tool calling, and token streaming, you create conversational interfaces that users genuinely rely on.