Production LLM Optimization: Prompt Caching, Semantic Caching, and Latency Reduction

The $300 Wake-Up Call

When I was building an early AI feature for a side project, everything seemed effortless during local testing. I plugged in an OpenAI API key, tested a few prompts in my browser, and watched the responses stream back. It felt like magic. But at the end of our first public month, I opened my billing dashboard and stared in disbelief: our API bill was over $300 (roughly ₹25,000).

For a developer in India bootstrapping on personal savings, that was a massive blow to the chest. When I inspected our request logs, the problem was obvious: over 60% of incoming questions were almost identical (like "How to fix CORS error in Express" or "How to parse JSON in Python"). Because our backend made a fresh, un-cached API call on every single request, we were burning thousands of expensive tokens repeating work the model had already done five minutes earlier.

On top of the bill, users were complaining that requests took four to five seconds to respond. We spent a weekend re-architecting our pipeline: adding provider prompt caching, deploying an in-memory Redis semantic cache, and setting up tiered model routing. Our monthly bill dropped from $300 to $42, and our p95 response time fell from 4,500ms down to 35ms. Here is exactly how you can do it.

The Three-Tier Production Architecture

Never send every raw user query directly to a massive frontier model. Filter and cache your requests through three defensive layers:

User Query
    |
    v
[Layer 1: Semantic Vector Cache (Redis)]   --> Cache Hit? Return answer in 15ms (Cost: ₹0)
    |
    v (Cache Miss)
[Layer 2: Complexity Router]              --> Simple task? Route to small fast model ($0.15/1M)
    |
    v (Complex reasoning needed)
[Layer 3: Cached Frontier Model Call]     --> Static system prompts hit provider KV cache (90% discount)

1. Provider-Level Prompt Caching

Major providers like Anthropic and OpenAI offer prompt caching. When large portions of your prompt (such as a 10-page system specification, schema definitions, or documentation examples) stay the same across requests, the provider stores the token states in fast GPU memory.

Cached input tokens cost up to 90% less than standard tokens, and they reduce time-to-first-token by up to 80%. Here is how to configure prompt caching with Anthropic Claude using TypeScript:

// src/lib/cached-client.ts
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

export async function runCachedReview(userCode: string, sharedDocumentation: string) {
  const response = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    system: [
      {
        type: 'text',
        text: 'You are an experienced senior engineer reviewing code against strict reliability standards.',
      },
      {
        type: 'text',
        text: sharedDocumentation,
        // Mark this heavy static documentation as a cached breakpoint
        cache_control: { type: 'ephemeral' },
      },
    ],
    messages: [
      {
        role: 'user',
        content: userCode,
      },
    ],
  });

  // Log your token savings directly in your console
  const usage = response.usage;
  console.log(`Input Tokens:      ${usage.input_tokens}`);
  console.log(`Cache Read Tokens: ${(usage as any).cache_read_input_tokens || 0}`);
  console.log(`Cache Write Tokens: ${(usage as any).cache_creation_input_tokens || 0}`);

  return response.content[0];
}

The golden rule for prompt caching: place all static text at the very beginning of your prompt, and place dynamic user input at the very end. Any change near the start of the prompt invalidates the cache for everything that follows.

2. Semantic Caching with Redis

Traditional HTTP caching uses exact string matching. If User A asks "How to parse JSON in Node.js?" and User B asks "How do I parse JSON strings with Node?", an exact string match fails. Both queries hit the LLM.

Semantic caching solves this problem. You convert the incoming query into a vector embedding, query your Redis cache, and calculate cosine similarity. If the similarity is above 0.94, you return the cached response instantly:

// src/lib/semantic-cache.ts
import { createClient } from 'redis';

interface CacheHit {
  query: string;
  response: string;
  similarity: number;
}

export class SemanticCache {
  private redis = createClient({ url: process.env.REDIS_URL });

  async connect() {
    if (!this.redis.isOpen) {
      await this.redis.connect();
    }
  }

  private calculateCosineSimilarity(a: number[], b: number[]): number {
    let dot = 0;
    for (let i = 0; i < a.length; i++) {
      dot += a[i] * b[i];
    }
    return dot;
  }

  async findMatch(queryVector: number[], threshold = 0.94): Promise<CacheHit | null> {
    const keys = await this.redis.keys('cache:faq:*');
    
    for (const key of keys) {
      const record = await this.redis.hGetAll(key);
      if (!record.embedding || !record.response) continue;

      const storedVector: number[] = JSON.parse(record.embedding);
      const similarity = this.calculateCosineSimilarity(queryVector, storedVector);

      if (similarity >= threshold) {
        return {
          query: record.query,
          response: record.response,
          similarity: Math.round(similarity * 100) / 100,
        };
      }
    }
    return null;
  }

  async save(query: string, embedding: number[], response: string, ttlSeconds = 86400) {
    const key = `cache:faq:${Date.now()}_${Math.random().toString(36).substring(7)}`;
    await this.redis.hSet(key, {
      query,
      response,
      embedding: JSON.stringify(embedding),
    });
    await this.redis.expire(key, ttlSeconds);
  }
}

3. Smart Model Routing (Stop Using GPT-4 for Everything)

Not every feature requires a flagship reasoning model. If all you need is to extract a user name from a sentence or classify an email into three categories, using a $3.00/1M model is throwing money away.

Use a two-tier routing strategy:

• Tier 1 (Small & Fast): Use models like gpt-4o-mini or claude-3-5-haiku for classification, summarization, and simple extraction. These cost $0.15 per million tokens and respond in under 300 milliseconds.
• Tier 2 (Deep Reasoning): Reserve top-tier models exclusively for multi-step coding, complex architecture plans, and heavy logic.

If Tier 2 hits an unexpected rate limit (HTTP 429), your router can quickly retry against a fallback provider to keep the application running smoothly.

4. Monitor Your Token Spend with Exact Metrics

You cannot optimize what you do not measure. Track the following numbers in your backend dashboards:

• Time to First Token (TTFT): Keep this under 400ms by streaming responses via Server-Sent Events (SSE).
• Cache Hit Ratio: Aim for at least a 35% to 50% hit rate on frequent customer questions using Redis.
• Cost per Active User: Track how much each active user costs your business per month in API tokens.

You can calculate prompt tokens and plan your usage with our free Token Counter. Check out our Free Developer Tools and our guide on Free Cloud Resources to host your Redis and PostgreSQL infrastructure for zero rupees.

Building AI apps does not require a venture capital budget. Put static prompts in cache, store frequent answers in Redis, route simple queries to smaller models, and keep your infrastructure lean. Start optimizing tonight.