Context, Caching, and Prompt Reuse

Every request you send Claude has a fixed budget: the context window. System prompt, tool definitions, conversation history, retrieved documents, and the user’s turn all compete for the same finite space, and every token in that space is billed on every call. An architect who ignores this ships a system that works in the demo. Then, in production, it grows slow and expensive one turn at a time: the window fills, and the same large prefix is re-sent and re-paid on request after request. This chapter is about spending that budget well. Keep the window lean, and — the larger lever — reuse the stable parts of a prompt so you pay full price for them once instead of every time.

What the context budget is, and why it fills

The context window is the total amount of text the model can consider at once, measured in tokens. It is not just the user’s question; it is the whole assembled request. In a support platform that means the system prompt and its rules, the tool definitions, the retrieved policy and catalogue chunks from the RAG layer, the running conversation, and finally the customer’s latest message — all of it, every call.

Two forces fill it. Conversation growth: a multi-turn support session accumulates history. Because the API is stateless you re-send the entire transcript each turn, so a long chat’s prompt is mostly its own past. Retrieval: a RAG step stuffs documents into the prompt, and a generous retriever can dominate the budget with material the answer barely uses. As the window fills, two things happen. Cost climbs, because you pay for every resent token. And quality can erode, because a model’s focus degrades when the relevant fact is buried in a wall of marginally-relevant context. Managing the budget is therefore both a cost and a quality concern.

Token optimization: keep the window lean

The first discipline is to put less in. A few moves cover most of the gain:

• Trim history. A long conversation does not need every turn resent verbatim. Summarize older turns into a compact running note, or drop turns no longer relevant, so the transcript stays bounded rather than growing without limit.
• Retrieve tightly. Return the few chunks that actually answer the question, not the top fifty. Better retrieval is cheaper and more accurate, because it both shrinks the prompt and raises the signal in it.
• Structure tool output. This is the one architects most often miss. When a tool returns data to the model, return the fields the model needs, not a raw API dump. A lookup_order tool that hands back the whole order JSON — internal IDs, timestamps, warehouse codes, nested metadata — spends hundreds of tokens on noise the model must read past. Have the tool return a small, purpose-shaped object: order state, items, amount, delivery date. You control the tool’s output; make it context-efficient by design.

Trimming is necessary but bounded. It reduces what you send; it cannot make a genuinely large, genuinely necessary prefix cheap. For that you need reuse.

The big lever: prompt caching

Most of a support platform’s prompt is the same on every call: the role, the rules, the PII policy, the escalation policy, often a large block of governing text. You are re-sending and re-paying for those thousands of tokens every request. Prompt caching lets Claude store that stable prefix and serve it back cheaply. You mark a cache breakpoint with cache_control; everything from the start of the request to that breakpoint is cached. The first call writes the prefix to cache; later calls with a byte-identical prefix read it at a steep discount. Cache reads bill at roughly a tenth of the input rate, writes at about 1.25× for the default short TTL.

Run against claude-haiku-4-5, two back-to-back calls with a large stable system prefix:

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"

# A stable policy prefix, built well past Haiku's cache minimum.
SECTION = """\
BOOKSHOP SUPPORT POLICY — SECTION {n}
Refunds are allowed within thirty days of delivery for orders in 'completed',
'shipped', or 'returned' state, never exceeding the amount charged; any refund
over fifty dollars needs second-level approval logged against the agent.
PII (names, emails, addresses, order history) is handled under least privilege:
retrieve only what a task needs, never echo a full record, never log it.
Ground every recommendation in the retrieved catalogue; escalate chargebacks,
legal threats, and safety concerns to a human.
"""
LARGE_POLICY = "\n\n".join(SECTION.format(n=i) for i in range(1, 13))
SHORT_POLICY = "You are the bookshop support assistant. Be concise and accurate."


def call(system_text, user_text):
    resp = client.messages.create(
        model=MODEL, max_tokens=32,
        system=[{"type": "text", "text": system_text,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": user_text}],
    )
    u = resp.usage
    return u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens

print(call(LARGE_POLICY, "Can I refund an order that is still 'placed'?"))  # write
print(call(LARGE_POLICY, "What is the ceiling on a refund amount?"))        # read
print(call(SHORT_POLICY, "Hello"))                                          # skipped

The observed output:

call 1 (write):  input=19  cache_creation=4777  cache_read=0
call 2 (read):   input=15  cache_creation=0     cache_read=4777
short system:    input=23  cache_creation=0     cache_read=0

The 4,777-token policy prefix was written to cache on call 1 and read from cache on call 2, where only the 15 live tokens of the new user turn billed at the normal input rate. That prefix cost about a tenth the second time. Across the thousands of turns a support platform serves against one policy, that is the gap between an affordable feature and an unaffordable one.

The silent-skip trap

The third line is the one the exam likes. The same code with a short system prompt produced cache_creation=0 — caching was silently skipped. Each model has a minimum prefix size before caching engages, and it is larger than people expect. On Haiku it’s 4,096 tokens, which is why the 4,777-token prefix cached and the one-line prompt did not. Below the minimum, cache_control does nothing: no error, no warning, and you are billed full price as if you had never added it. So caching pays off for genuinely large stable prefixes, and the discipline is to verify cache_creation_input_tokens is non-zero rather than assume a breakpoint took effect. The Foundations cost chapter hit this exact trap; it is worth carrying into every design that leans on caching.

One more rule follows from how caching matches: the cache keys on an exact byte prefix. Anything before the breakpoint must be identical to hit. So the stable content goes first, and the variable content goes after the breakpoint — the customer’s turn, per-request IDs, timestamps. A single changing byte in the prefix invalidates the cache and forces a rewrite. Order the request stable-first, and caching works; interleave a timestamp into the system prompt, and it never engages.

The second lever: modular prompts

Caching rewards a stable prefix; modular prompts are how you keep the prefix stable across many call sites. Rather than a bespoke prompt per step, compose each prompt from shared parts — a common role-and-rules base, plus the task-specific slice — exactly the templating from the previous chapter. The reuse is twofold. Operationally, one edit to the shared base reaches every step. Economically, because every step opens with the same base text, that base is a cacheable prefix shared across the whole platform, not a different prompt each time. Modularity and caching are the same structural choice seen from two angles: build prompts from stable, ordered pieces and both the maintenance win and the cost win fall out.

The third lever: Skills

The third form of reuse is for procedures. Sometimes the reusable thing is a multi-step method: how to run a refund end to end, how to apply a house style, how to work a domain workflow. Packaging it into the system prompt bloats the fixed context with instructions that only matter some of the time. A Skill solves this: a procedure authored as a SKILL.md (plus optional scripts and files) that Claude discovers and loads only when the task calls for it. Its short description sits in context by default; the full body is read on demand. That progressive disclosure is precisely a context lever. The fixed prompt stays small, and the detailed procedure enters the window only when relevant, instead of riding along on every unrelated call. Skills are the Foundations agentic-customization chapter’s answer to “reusable know-how,” and here they double as a way to keep the standing context lean.

The three levers are complementary, not competing: caching makes the stable prefix cheap to re-send, modular prompts keep that prefix stable across call sites, and Skills keep occasional procedures out of the fixed context until they are needed.

Applied to the bookshop platform

The design writes itself from the run above. The large stable prefix sits first in every request, behind a cache breakpoint: the shared role, the standing rules, the PII and escalation policy, the governing policy text the assistant answers from. It is written once and read at a tenth of its price on the thousands of turns that follow. The small per-request user turn goes after the breakpoint, uncached, billed at full rate — the 15 live tokens riding on the 4,777-token cached base. The shared prefix itself is assembled from modular parts. A change to the escalation rule is one edit that every step inherits, and it keeps the prefix byte-stable enough to keep hitting cache. And the refund procedure — the multi-step eligibility-and-approval workflow — lives in a Skill, loaded when a refund is actually in play rather than weighing down every FAQ lookup. Lean window, cached prefix, and just-in-time procedures: the platform pays full price for its policy exactly once and carries only what each turn needs.

Final thoughts

The context window is a budget every request spends, and it fills as conversations grow and retrieval stuffs documents. Trim it — bound the history, retrieve tightly, and shape tool output to the fields the model needs rather than dumping raw JSON. Then reuse the stable parts. Prompt caching takes a large stable prefix from full price to about a tenth — 4,777 tokens read from cache while 15 live tokens billed normally. But it is silently skipped below a per-model minimum, so verify cache_creation is non-zero and keep the cached prefix byte-stable, with variable content after the breakpoint. Modular prompts keep that prefix stable across every call site, and Skills keep occasional procedures out of the fixed context until they are needed. Spend the budget deliberately and the same platform that would have grown slow and costly instead pays once and runs lean.

Next: integration protocols — how the platform actually talks to the model, the RAG layer, and the tools, and the protocol choices an architect makes at the seams.