Let the Numbers Decide: A/B Testing and Diagnosing Claude Failures

The previous chapter built the instrument: a dataset, a scoring method, and a tracked metric. An instrument that only tells you “you are at 85%” is a dashboard. The value comes when you use it to act — to choose the better of two designs, and to find out why the other one lost. Those are the two jobs of this chapter. Both are things you cannot do responsibly without an eval. And both are where an architect earns the title, because the decisions stop being matters of taste and start being matters of measurement.

A/B testing: let the eval pick the winner

You will constantly face a fork: this system prompt or that one, this model or a cheaper one, retrieval on or off. The wrong way to settle it is to try each on a few examples and go with the one that “felt” better. Small samples and confirmation bias will lie to you, and you will ship the loser convinced it was the winner. The right way is an A/B test: hold everything constant except the one variable you are testing, run both arms over the same eval set, and compare the metrics. The eval is the referee, and the referee does not have a favorite.

The discipline is in the word same. Same cases, same judge, same rubric, one changed variable. If you change the prompt and swap the model, a difference in score tells you nothing about which change caused it. Change one thing, measure, then change the next.

Here is a real fork from the bookshop platform: should the support assistant answer from its own knowledge of “a bookshop,” or should every answer be grounded in the retrieved knowledge base? It is the RAG-or-not decision made concrete, and it is exactly the kind of question people argue about in meetings. Run it through the eval instead:

import anthropic
client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"

KB = ("Returns: 30 days for a full refund. Shipping: express is a flat $12, next "
      "business day, US and Canada only. Membership: Prime is $49/year. "
      "Order records are retained 7 years.")

VARIANTS = {
    "A_ungrounded": "You are the support assistant for an online bookshop. "
                    "Answer the customer's question helpfully and specifically.",
    "B_grounded":   "You are the bookshop support assistant. Answer ONLY using the "
                    "knowledge base below; if it is not there, say you don't know.\n" + KB,
}

def answer(system, q):
    r = client.messages.create(model=MODEL, max_tokens=120, system=system,
        messages=[{"role": "user", "content": q}])
    return "".join(b.text for b in r.content if b.type == "text").strip()

# (judge() and DATASET are the ch12 harness, reused unchanged)
def run(name, system):
    rs = [judge(c["q"], answer(system, c["q"]), c["must"], c["forbid"]) for c in DATASET]
    n = len(rs)
    passed, mean = sum(bool(r["passed"]) for r in rs), sum(r["score"] for r in rs) / n
    print(f"{name}: pass {passed}/{n}  mean {mean:.2f}/5")
    return passed, mean

scores = {k: run(k, s) for k, s in VARIANTS.items()}
print("winner:", max(scores, key=lambda k: scores[k]))

The numbers, run against claude-haiku-4-5 over the seven-case set:

Variant	Pass rate	Mean score
A — ungrounded	2 / 7	2.43 / 5
B — grounded	7 / 7	5.00 / 5

Grounding wins, and not narrowly. What makes the result instructive is how the ungrounded arm lost. On the questions it could not know from general knowledge — shipping cost, membership price, retention period — it either refused or guessed. And on “Do you ship to the UK?” it answered, with total confidence, “Yes, we do ship to the UK!” The real policy is US and Canada only. That single answer is the whole case for grounding: an ungrounded model does not know what it does not know, and it fails fluently. The eval caught it because the rubric checked for the specific facts, not for a confident tone. Had you A/B-tested by vibes, the ungrounded answer’s polish might have won your vote.

Two details make the result trustworthy rather than lucky. The judge scored against the specific facts in the rubric, so a fluent wrong answer could not earn a pass on tone alone. The ungrounded arm’s polish did it no good. And both arms ran the identical seven cases through the identical judge, so the 2/7-to-7/7 gap is attributable to the one thing that differed: grounding. Change two variables and you learn nothing; change one and the number means something.

When is A/B the wrong tool? When your dataset is too small or too noisy for the gap to be real — a one-case difference on a seven-case set is noise, not a signal. A 5-case gap on a 200-case set is a decision. Size the dataset to the size of the difference you need to detect, and treat a within-noise result as a tie, not a win. The corollary: a big, obvious gap like the one above is safe to act on even on a small set, because it is far outside the noise. A narrow gap on the same set is not.

Diagnosis: reading a failure to its cause

The other job is debugging. When a case fails, “the model got it wrong” is not a diagnosis — it is a shrug. A production Claude system has several independent things that can produce a wrong answer, each with a different signature and a completely different fix. Naming the cause is the whole skill, because fixing the wrong layer wastes days. The four causes to hold in your head:

• Prompt failure. The instructions are ambiguous, contradictory, or missing a constraint, so the model does something reasonable that you did not want. Signature: the answer is a sensible response to what you literally asked. Fix: the prompt, not the model.
• Hallucination. The model states something not supported by its input — a fabricated policy, an invented figure. Signature: confident, specific, and unsupported by any source. Fix: grounding, retrieval, and a “say you don’t know” instruction, which is precisely the A/B above.
• Model mismatch. The task genuinely exceeds the chosen model’s capability — subtle multi-step reasoning on the cheapest model. Signature: failures cluster on the hard cases while easy ones pass, and a stronger model fixes them. Fix: route to a bigger model for that path.
• Retrieval failure. In a RAG system, the model answered faithfully — but from the wrong retrieved context. Signature: the answer is well-grounded in a chunk that should not have been retrieved, or a chunk that is stale. Fix: the retrieval and indexing pipeline, and not the model or the prompt.

These causes look alike from the outside: every one of them shows up as “a wrong answer.” But they have nothing in common under the hood, and the fix for one does nothing for another. Rewriting the prompt will not fix a stale index; swapping the model will not fix an ambiguous instruction. So the discipline is to resist the reflex to “improve the prompt” or “try a bigger model” and instead read the failure for its signature first. Is the answer a sensible response to what you literally asked (prompt), confident and unsupported (hallucination), clustered on the hard cases (model), or faithfully grounded in the wrong chunk (retrieval)? The signature tells you which layer to touch, and touching the wrong one wastes the afternoon.

That last cause is the one architects miss, because every instinct points at the model when the model is innocent. The exam presses on it directly.

The stale-document scenario, worked

Here is the exam’s sample question 3. A RAG system that was answering correctly starts returning confident-but-wrong answers. You check: the model version is unchanged, latency is unchanged, and the code that calls Claude is unchanged. The one thing that happened recently is a document refresh — the knowledge base was re-ingested. Where do you look first?

Not the model. The model version did not move, so the model did not cause a new failure. Latency is flat, which rules out a timeout or a truncated call. The prompt is the same. The variable that did change is the corpus and its index. And “confident-but-wrong” is the exact signature of a retrieval failure: the model is faithfully answering from a chunk that is stale or mismatched. The first place to look is the retrieval and indexing step. The usual culprits: stale chunks that were not re-embedded, a botched refresh that indexed a draft or superseded document, an embedding-model version skew, or a chunk-boundary change that split a policy across two chunks so the right one no longer ranks. The model is doing its job perfectly on bad inputs.

To make the failure mode concrete, here is a minimal RAG loop where the model and prompt never change and only the index goes bad:

import re, anthropic
client = anthropic.Anthropic()

def retrieve(corpus, query, k=1):
    q = set(re.findall(r"[a-z]+", query.lower()))
    return sorted(corpus, key=lambda c:
        len(q & set(re.findall(r"[a-z]+", c.lower()))), reverse=True)[:k]

def grounded_answer(context, query):
    r = client.messages.create(model="claude-haiku-4-5", max_tokens=80,
        system="Answer ONLY from the provided context.",
        messages=[{"role": "user", "content": f"<context>{context}</context>\n{query}"}])
    return "".join(b.text for b in r.content if b.type == "text").strip()

Q = "What is the return window for books?"
good = ["Returns policy: books may be returned within 30 days for a full refund.",
        "Shipping: express is a flat $12, next business day."]
stale = ["DRAFT returns policy (superseded): books may be returned within 14 days.",
         "Shipping: express is a flat $12, next business day."]

print("before:", grounded_answer(retrieve(good, Q)[0], Q))
print("after :", grounded_answer(retrieve(stale, Q)[0], Q))

Run against claude-haiku-4-5, before the bad refresh the answer was “books may be returned within 30 days of delivery for a full refund”; after a refresh that re-ingested a superseded draft, the very same model with the very same prompt answered “books may be returned within 14 days of delivery.” Confident, well-formed, and wrong — and the model never changed. If you had spent the afternoon swapping models or rewriting the system prompt, you would have moved the two things that were not broken and left the one that was. Baseline the retrieval first: log what got retrieved for the failing query and read it. The bad chunk is usually sitting right there in the logs, wearing a “DRAFT (superseded)” label nobody looked at.

The discipline underneath both jobs

A/B testing and diagnosis are the same discipline seen from two angles: change one variable and measure. A/B changes one variable on purpose and reads the score to choose. Diagnosis finds the one variable that changed by accident and reads the failure to explain it. In both, the enemy is moving two things at once — it destroys your ability to attribute the result. And in both, the rule is baseline first. A number that is systematically wrong points at a confounder you have not controlled; a number that is noisily wrong points at sample size. Establish the clean baseline, change exactly one thing, and let the eval tell you what happened.

Final thoughts

Once you have an eval you can do two things a spreadsheet of opinions never could. You can A/B two designs on identical cases and let the metrics choose. Grounding beat ungrounded 7/7 to 2/7 here, and the loss was a fluent fabrication the vibe test would have missed. And you can diagnose a failure to its actual cause rather than blaming the model: prompt, hallucination, model mismatch, or the retrieval failure that makes a healthy model answer confidently from stale context. When answers go wrong after a document refresh and nothing else moved, look at retrieval first — the model is usually innocent, and the evidence is already in the logs.

Next: optimizing for tokens, latency, and cost — once the system is correct and measured, making it fast and cheap enough to meet the platform’s SLAs.