Tokens, Sampling, and the Thinking Modes

Every decision you’ll make about a Claude application comes down to a handful of facts about how the model actually works: what it costs, how much fits, how repeatable or creative it is, how hard it thinks. Get those facts right and the choices in the chapters ahead are almost mechanical. Get them from folklore and you’ll size a prompt against the wrong number, or treat a “deterministic” setting as a guarantee it never made.

This chapter is those fundamentals. How the model represents text as tokens, how it generates one token at a time by sampling from a distribution, what’s genuinely deterministic and what only looks it, and the thinking modes you can switch on to trade latency for reasoning. It’s grounded in live behavior, including two results that contradict common lore. (It opens Arc 4, Model Selection & Optimization, and everything that arc optimizes is measured in the units below.)

Tokens: the unit of everything

The model doesn’t see characters or words; it sees tokens — subword chunks, very roughly 3–4 characters of English each. Everything you care about is measured in tokens: what you’re billed, how much fits in context, how fast a response streams. A rule of thumb is ~750 words per 1,000 tokens, but it’s only a rule of thumb. Punctuation, code, and non-English text tokenize differently, so you count, you don’t estimate — with the count_tokens endpoint from chapter 1, passed the whole request.

One version-sensitive fact: tokenization changes across model generations. The same text can tokenize to a different count on a newer model. So a prompt sized against one model should be re-counted against the model you actually deploy. Token counts are model-specific, not universal.

The context window

Every model has a context window: the maximum number of tokens it can consider at once, spanning both your input and the generated output. This is a hard ceiling, not a soft one. If input plus generation would exceed it, the request fails — recall the model_context_window_exceeded stop reason from chapter 1. Two practical consequences follow. Long conversations eventually hit the wall, which is why context management, Arc 5, exists. And max_tokens reserves output room inside the window. The window is a budget you spend across the whole request, and running out is a failure mode you design around, not a warning you can ignore.

Sampling: how the next token is chosen

Claude generates one token at a time, autoregressively: given everything so far, it produces a probability distribution over the next token, picks one, appends it, and repeats. “Picks one” is where sampling comes in, and it’s what temperature controls:

• Low temperature (near 0) concentrates probability on the most likely tokens — focused, repeatable output. Right for extraction, classification, code, and tool use.
• High temperature flattens the distribution so less-likely tokens get chosen more often — varied, creative output. Right for brainstorming and drafting.

top_p and top_k are alternative truncations of that distribution (adjust temperature or top_p, not both). The mental model that makes this stick: the model always produces a distribution; temperature decides how boldly you sample from it. Understanding generation as next-token sampling from a distribution is what makes the rest — non-determinism, thinking, streaming — make sense.

Is temperature 0 deterministic?

Common lore says two opposite things, so it’s worth checking directly. The claim on one side: “temperature 0 is deterministic.” On the other: “LLMs are non-deterministic even at temperature 0.” Run three times at temperature=0 with the same prompt, the outputs were byte-identical all three times.

So in practice, temperature 0 gave fully repeatable output here. But “identical in three runs” is not a guarantee. Anthropic documents that even at temperature 0, output is not strictly guaranteed to be identical. Factors outside your control — hardware, batching, floating-point — can introduce variation. The honest statement: temperature 0 makes output as deterministic as you can practically get, and you should design as if small variation is still possible. Don’t build a system that requires two temperature-0 calls to be bit-for-bit equal; do expect them to be, nearly always. Treating “near-deterministic” as “guaranteed deterministic” is the trap.

The thinking modes

Modern Claude models can think before answering — generate internal reasoning that improves hard tasks (math, multi-step logic, careful analysis) before the final response. This is an output mode (distinct from multi-format input, chapter 3), and it comes in two forms whose availability differs by model:

• Enabled (manual) thinking: you set thinking={"type": "enabled", "budget_tokens": N}, giving the model a token budget for reasoning. On Haiku 4.5, the response comes back with a thinking content block followed by a text block (['thinking', 'text']). The reasoning is visible, and the final answer is correct (“17 × 23 = 391”). The thinking block precedes the answer; you render or discard it as you like.
• Adaptive thinking: newer models let the model decide how much to think via an effort level, rather than a fixed budget. But it is not universal: thinking={"type": "adaptive"} on Haiku 4.5 returns 400 — "adaptive thinking is not supported on this model." Adaptive is a capability of the newer/larger models; older ones use the manual enabled form.

That split is a live example of “capabilities differ across models” (chapter 14): the same thinking parameter that works on one model is rejected by another. There’s also a fast mode on some models that trades a little quality for lower latency — the inverse lever from thinking. So thinking is a dial. You trade latency and tokens for reasoning quality, and which form is available depends on the model you chose.

Shot-based prompting

A fundamental from the same domain: how many examples you give the model.

• Zero-shot — instructions only, no examples. Fine for simple, unambiguous tasks.
• One-shot / few-shot (multi-shot) — you include one or several worked examples of the desired input→output. This is the single most reliable way to fix inconsistent format and steer judgment on ambiguous cases.

In the companion Architect work, on a formatting task zero-shot produced 0 of 4 outputs in the required format. A two-example few-shot prompt produced 4 of 4. Examples teach the model to generalize a pattern to cases you didn’t show. The depth is Arc 5’s prompt-engineering chapter; the fundamental is that more well-chosen examples buy more consistency.

A note on the technical plumbing

The SDKs wrap a REST API. Underneath client.messages.create is an HTTPS POST to a JSON endpoint; underneath streaming is server-sent events over that same connection. The SDK gives you types, retries, and accumulation. But nothing stops you from calling the REST endpoint directly from a language without an SDK — the wire protocol is the contract. Knowing the SDK is a convenience over HTTP keeps you from being mystified when you need to debug a raw request or integrate from an unusual runtime.

Final thoughts

The fundamentals under everything. Text is tokens: count them, since they’re model-specific and shift across generations. The context window is a hard input-plus-output budget you design around. Generation is next-token sampling from a distribution that temperature shapes. Temperature 0 was byte-identical across three runs — practically deterministic, though not a contractual guarantee. Thinking trades latency and tokens for reasoning, and its form is model-dependent: enabled worked on Haiku, adaptive was rejected outright. And more well-chosen examples buy more consistency. These aren’t trivia; they’re the model of computation every optimization decision in the next two chapters is made against.

Next: choosing a model — Opus, Sonnet, and Haiku, and the quality-latency-cost triangle you’re really deciding.