Prompt Caching Architecture for AI Coding Agents

Imagine you're having an intense debugging session with an AI coding assistant like Claude Code, Cursor, or Copilot.

You’ve been at it for 25 turns. The AI knows your repository structure, your custom test runner, the messy bug in auth_controller.ts, and the three failed attempts you made ten minutes ago to fix it. When you type:

"Cool, now write a unit test for that edge case,"

...it immediately spits out a flawless test using your project’s exact mocking conventions.

It feels like you’re pairing with a sharp senior engineer who has been sitting beside you all morning, soaking in context.

Except that’s not what’s happening at all.

Behind that slick terminal or editor UI, the AI didn't "remember" a single thing. Large Language Models (LLMs) have no persistent memory. They don't have a background process quietly holding state. They are as stateless as a plain HTTP GET request.

To give you the illusion of memory, the agent harness on your machine packed up the entire conversation from the beginning—every prompt, every assistant reply, every 1,500-line file you inspected, and every terminal error trace—and fired that gargantuan payload across the internet to the model.

Welcome to The Stateless Snowball.

Without a breakthrough optimization, this snowball would make AI coding agents impossibly slow and outrageously expensive. That optimization is Prompt Caching—the unsung hero of the modern AI stack.

Grab a fresh cup of coffee. Let’s treat the LLM like a backend service, pop open the hood, and see how prompt caching actually works.

1. The "Stateless Snowball": The Problem Nobody Talks About

If you’re a backend engineer, you already know the golden rule of REST: the server maintains no client state between requests. Every HTTP request must contain all the context the server needs to fulfill it.

LLMs work the exact same way. When you call an LLM API (like Anthropic’s /v1/messages or OpenAI’s /v1/chat/completions), you aren't sending a quick message saying "Hey, remember what I said earlier?" You are sending an array of messages representing the entire universe of that conversation:

POST /v1/messages
{
  "system": "You are an expert TypeScript coding assistant...",
  "tools": [ { "name": "read_file", ... }, { "name": "run_terminal", ... } ],
  "messages": [
    { "role": "user", "content": "Turn 1: Find the bug in auth..." },
    { "role": "assistant", "content": "Turn 2: Let me inspect auth_controller.ts" },
    { "role": "user", "content": "Turn 3: [File contents of auth_controller.ts (1,500 lines)]" },
    { "role": "assistant", "content": "Turn 4: I see the issue on line 42..." },
    ...
    { "role": "user", "content": "Turn 30: Now write a unit test for that." }
  ]
}

Now, trace what happens to the size of that payload as a coding session progresses.

The Anatomy of Context Bloat

Let's walk through a realistic coding agent session:

• Turn 1: You ask: "Why is login failing on staging?" Payload size: ~2,500 tokens (~10 KB, including system prompts and tool schemas). Fast and cheap.
• Turn 2: The agent invokes a tool: read_file("src/controllers/auth_controller.ts").
• Turn 3: The harness returns the file. It’s a 1,500-line enterprise controller (~15,000 tokens).
• Turn 4: The agent spots a database query and runs read_file("src/models/user.ts") (~5,000 tokens).
• Turn 5: The agent edits a file and runs the test suite: run_terminal("npm test"). The test fails with a massive 200-line stack trace (~3,000 tokens).

By Turn 20, a single prompt like "Looks good, commit it" isn’t sending a 5-word sentence. It is dragging a 150,000-token payload (~600 KB of raw code and text) along with it.

2. The Financial and Latency Disaster

If you run an AI coding agent without prompt caching, two catastrophic bottlenecks occur:

1. The Latency Wall (Time-To-First-Token)

When a GPU receives an input prompt, it has to "read" and process all the input tokens before it can generate the very first word of its response. This is known as Time-To-First-Token (TTFT).

Processing tokens through deep transformer layers requires massive matrix multiplications. For a 150,000-token payload, an uncached model can spend 8 to 15 seconds just crunching input tokens before it outputs a single character.

Waiting 15 seconds after every message completely shatters the interactive pairing flow.

2. The Quadratic Cost Curve

Let's look at the numbers using standard enterprise pricing (~$3.00 per 1M input tokens for flagship frontier models like Claude 3.5 Sonnet):

Notice how the cost escalates? You aren't paying for what you just typed; you are paying rent on everything you’ve typed and read since the session started.

A single 25-turn refactoring task would cost nearly $7.00. If an engineer runs 15 such tasks a day, the team’s monthly bill exceeds $3,000 per developer.

Something had to give.

3. Enter Prompt Caching: The Redis of the AI World

In traditional web architectures, what do you do when an expensive SQL query is executed repeatedly with identical parameters?

You put Redis or Memcached in front of it. You compute the expensive result once, stash it in high-speed RAM, and serve subsequent requests in 2 milliseconds.

Prompt Caching is Redis for LLMs.

What Actually Happens During a Cache Hit?

When an LLM processes your text, it converts raw words into numeric vectors and computes intermediate key-value states (the KV cache) across all its attention layers. This initial computation—called the prefill phase—is where 90% of the GPU compute and time is burned.

Normally, once the API finishes generating its reply, the GPU wipes those intermediate states from memory to free up space for the next customer's request.

With Prompt Caching, cloud inference engines do something brilliant: they don't throw those computed states away. They store the precomputed states directly in ultra-fast GPU memory (HBM - High Bandwidth Memory).

When your next turn arrives, the inference router inspects the incoming token sequence:

1. "Have I already processed tokens 1 through 148,000 in this exact order?"
2. Yes! (Cache Hit 🎉)
3. The GPU loads the precomputed KV tensors instantly from memory.
4. The model only computes the prefill for the 2,000 new tokens you just added!

The Impact on Cost and Speed

• Price: Providers (Anthropic, OpenAI, DeepSeek) offer a 90% discount on cached input tokens. Instead of $3.00 per million tokens, cached tokens cost just $0.30 per million.
• Latency: TTFT drops from 10+ seconds down to under 1 second.

That 25-turn refactoring session that used to cost $6.80 now costs roughly $0.85, and the AI responds almost instantly on turn 25.

4. The "Prefix Match" Rule: The Barista Analogy

Here is where software engineering principles come in. You can't just toss random data at an LLM and expect the cache to work. Prompt caches are strictly prefix-based.

To understand why, think of your local barista:

• Day 1: You order: "I want an oat milk latte, extra shot, 1 pump vanilla, cinnamon on top." The barista writes out the ticket, pulls the shot, steams the oat milk, and prepares your drink.
• Day 2: You walk up and say: "Give me the exact same latte as yesterday, plus a blueberry muffin." The barista immediately pulls yesterday's saved recipe ticket. They don't need you to explain what oat milk is or how many pumps of vanilla to add. They only have to grab the muffin. That’s a prefix hit.
• Day 3: You walk up and say: "Actually, today make it an almond milk latte, extra shot, 1 pump vanilla, cinnamon on top, plus a blueberry muffin."

Because you changed the very first ingredient (almond milk instead of oat milk), the entire saved recipe ticket is invalid! The barista has to recalculate and measure everything from scratch.

Token Index:    [ 0 ] [ 1 ] [ 2 ] [ 3 ] ... [ 99,999 ] [ 100,000 ]
Cached Stream:  [ S ] [ Y ] [ S ] [ T ] ... [  R  ]   [   U   ]
Incoming Stream:[ S ] [ Y ] [ S ] [ T ] ... [  R  ]   [   X   ]  <-- Diverges!
                 |===== CACHE HIT (Free & Fast) =====| |== Compute ==|

5. Why Code Structure Matters (The Million-Dollar Architecture Secret)

Once you understand the prefix rule, you realize something profound:

How you order the content inside your prompt determines whether your AI application is economically viable or financially suicidal.

🌟 The Golden Rule: Static Stuff at the TOP, Dynamic Stuff at the BOTTOM

Take a look at how a production-grade agent payload is constructed:

┌────────────────────────────────────────────────────────┐
│ 1. Core System Prompt (Persona, safety, formatting)    │ <-- STABLE (100% Cache Hit)
├────────────────────────────────────────────────────────┤
│ 2. Tool Definitions (JSON schemas for bash, read, etc) │ <-- STABLE (100% Cache Hit)
├────────────────────────────────────────────────────────┤
│ 3. Project Guidelines (CLAUDE.md / .cursorrules)       │ <-- STABLE (100% Cache Hit)
├────────────────────────────────────────────────────────┤
│ 4. Conversation History (Turns 1 through N-1)          │ <-- APPEND-ONLY (Hits cache!)
├────────────────────────────────────────────────────────┤
│ 5. Fresh Turn & Latest Tool Output                     │ <-- DYNAMIC (The only compute!)
└────────────────────────────────────────────────────────┘

Because the top sections never change, tokens 0 through 50,000 remain an identical prefix on every single request. And because conversation history is append-only, Turn 1 through Turn 15 also form a stable prefix for Turn 16!

The Rookie Mistake: The Cache Invalidation Trap

Here is a real bug that cost one startup thousands of dollars in LLM API bills in a single week.

A junior engineer decided to be helpful and added the current timestamp to the system prompt so the AI would know what time it was:

// ❌ THE CACHE DESTROYER
const systemPrompt = `You are a helpful coding assistant.
The current server time is: ${new Date().toISOString()}
Here are your rules...`;

new Date().toISOString() changes on every turn:

• Turn 1: 2026-10-08T04:24:18.102Z
• Turn 2: 2026-10-08T04:24:32.481Z

Because the timestamp was placed on line 2 (around token #15), every single turn alters token #15. The prefix match fails after 14 tokens! The entire 150,000-token conversation history following it missed the cache every single turn.

The Cache-Friendly Architecture:

If the model needs dynamic runtime context like timestamps, inject it at the leaf (the latest user message or tool response):

// ✅ CACHE-FRIENDLY ARCHITECTURE
const systemPrompt = `You are a helpful coding assistant.
Here are your rules...`; // Never changes! (100% Cache Hit)

const userTurn = `[Context: timestamp=2026-10-08T04:24:32Z]
Please refactor this function.`; // Put dynamic data at the very bottom!

6. The 5-Minute Cache Cliff (Cache Eviction)

In backend engineering, Redis keys have a TTL (Time To Live) so memory doesn't leak.

In the AI world, GPU High Bandwidth Memory (HBM3e) is the most expensive, supply-constrained hardware on earth. Cloud providers cannot let your 150,000 tokens sit on an NVIDIA H100 GPU indefinitely while you step away.

The 5-Minute Window

Most major providers maintain prompt caches with an ephemeral idle timeout of roughly 5 minutes:

• If you stay in the flow: You ask a question, test the code, reply 45 seconds later. Every request lands well within the 5-minute window. You enjoy 100% cache hits, sub-second latency, and 90% cost savings.
• If you get distracted: You go grab coffee, join a 15-minute standup, or read docs for 6 minutes. The cache expires.
• The next turn: Your next message suffers a Cache Miss. The GPU must re-read all 150,000 tokens from line 1. It takes 10 seconds and costs full price. Once re-computed, the 5-minute timer starts ticking again.

How Modern Agent Systems Hack the Cliff

1. Background Heartbeats / Warm Pings: Advanced tools send tiny background keep-alive probes right around the 4-minute mark if a long tool execution is running, ensuring the cache stays warm.
2. Context Compaction on Eviction: If a tool knows the cache is cold anyway after a long idle period, that is the ideal moment to summarize and prune old conversation history before re-filling the cache.

Related reading

• Opening the Black Box: How CLAUDE.md Really Works — why rules belong in user context, not system prompt
• How Cursor Router picks the right model — complexity scoring, model taxonomy, and LLM routing
• Cheap intelligence, expensive trust — when to use DeepSeek/GLM/Kimi vs Claude/OpenAI
• Coding is dying. What comes next? — wealth playbook and career shift for software engineers
• AI Systems Consulting — production agent sprints and systems advisory