"LLM Reasoning Modes (1/6) — Why Thinking Costs: Test-Time Compute and Reasoning Tokens"

Claude's low·medium·high·xhigh·max, OpenAI's reasoning_effort. This series dissects what that dial actually controls and why it behaves the way it does, from primary sources. Part 1 establishes the one premise every dial rests on — thinking costs.

Modern models don't answer immediately. Claude and GPT alike, given a hard question, first emit a block of thinking and only then begin the answer. That stretch — shown as "thinking…" or a long pause — is the model spending extra tokens, separate from the answer you see. Whether it's called effort or reasoning_effort, a reasoning mode is ultimately a knob over how many of these extra tokens to spend.

To understand this series you have to understand the thing the knob controls before the knob itself. That thing is test-time compute.

In One Paragraph

A reasoning model unrolls its chain-of-thought as tokens before the final answer. This is test-time compute — spending more compute at inference instead of scaling training — and it lifts accuracy on hard, multi-step work: math, logic, coding, agentic loops. But those thinking tokens are billed as output tokens and add latency. So every vendor exposes a per-call dial over how much to think (Claude effort, OpenAI reasoning_effort). The raw chain of thought is generally not returned to the caller — only summaries.

1. What a Reasoning Model Is: Tokens Before the Answer

A standard model takes input and generates answer tokens one by one. A reasoning model inserts a step in between: before producing the answer, it generates thinking (reasoning) tokens, then uses that thinking as context to write the final response.

Those thinking tokens are a chain-of-thought (CoT) — intermediate reasoning that breaks the problem into steps — spelled out as a token sequence. It plays the same role as a human jotting down intermediate work on paper, except the model does it by generating tokens.

The key point: this is not free. Thinking tokens are real generated tokens, so they consume compute, time, and money. "Think harder" is literally "spend more tokens."

2. Test-Time Compute — Spending on Inference, Not Training

There are two broad ways to raise model performance.

  • Train-time scaling: grow parameters, data, and training compute. Make the model itself bigger and smarter. The cost is paid once, at training.
  • Test-time scaling: leave the model as is, and spend more compute solving each individual question. Generate longer chains of thought, explore multiple paths, self-verify. The cost is paid per question.

Reasoning modes control the latter. With a single fixed model, you can think briefly (or not at all) on some questions and deeply on others — i.e., allocate compute per call to match how hard the question is. Tuning compute to "how hard is this one" without swapping models is the essence of the dial.

Let test-time compute be \(C_{\text{test}}\). On hard problems, accuracy generally rises as \(C_{\text{test}}\) grows — but not without bound; it's a curve of diminishing returns. The shape of that curve is what makes the cost-quality tradeoff in Part 6 of this series.

3. Why More Thinking Helps — and Where It Stops

Where extra thinking helps and where it doesn't splits cleanly.

High payoff — problems that need multi-step reasoning: - Math and logic (you must accumulate intermediate steps to reach the answer) - Hard coding and debugging (trace causes, test hypotheses) - Agentic loops (plan the next tool call, interpret results) - Long-form analysis (weave many facts into a conclusion)

These have a long path to the answer. When the model lays that path out in tokens, it makes fewer mistakes than when it tries to jump straight to the answer.

Little payoff or actively harmful — narrow output spaces: - Simple classification, extraction, format conversion - Short rewrites, single-answer lookups

When the answer is effectively fixed, more thinking doesn't improve it. Excess reasoning instead becomes overthinking — burning tokens needlessly, sometimes second-guessing a correct answer into a wrong one. Even Anthropic's own effort documentation notes that on structured-output tasks the top setting (max) "can lead to overthinking."

So the correct use of a reasoning mode is not "always maximum" but match the dial to task difficulty. This principle recurs throughout the series.

4. The Cost of Reasoning Tokens: Billed as Output Tokens

This is where the dial stops being a quality knob and becomes a cost knob.

Claude's thinking tokens and OpenAI's reasoning tokens are both billed as output tokens. Output tokens are far pricier than input tokens (e.g., Claude Opus 4.8 is 5 USD input / 25 USD output per 1M). Increasing thinking means spending more of the most expensive token class.

The cost structure:

Segment What Billed at
Input prompt and context input rate (cheap)
Thinking thinking / reasoning tokens output rate (expensive)
Answer the visible response output rate (expensive)

Thinking tokens also raise latency. The model must finish thinking before the first answer token appears, so a high reasoning setting increases time-to-first-token. Cost and latency moving together is the first thing you feel when operating reasoning modes.

5. Thinking Is Invisible — Hidden CoT and Summaries

Thinking tokens are generated and billed, yet the raw chain of thought is generally not returned to you verbatim.

  • Claude: current models never return the raw chain of thought. They optionally provide a summary (thinking.display: "summarized" returns a readable summary; "omitted" leaves the thinking block's text empty). Thinking happens and is billed the same regardless of the display setting.
  • OpenAI: reasoning tokens are not surfaced verbatim (summaries only), yet are still billed as output tokens.

Why hide it? The raw chain of thought exposes model behavior directly and carries safety/misuse exposure risk. What matters practically for you is twofold — (1) cost includes tokens you can't see, so account for them, and (2) to show progress in a UI you must explicitly enable summaries (the default is often the hidden side).

6. So a Dial Was Born — effort / reasoning_effort

In one line: thinking raises quality but increases cost and latency, and its payoff varies by task. The obvious conclusion — thinking should be tunable per call. Both vendors expose exactly that knob.

  • Claude — effort: low / medium / high / xhigh / max. Unusually, it controls not just thinking depth but all response tokens (explanations, tool calls included) with a single dial. Default is high.
  • OpenAI — reasoning_effort: minimal / low / medium / high (GPT-5; later generations add none and xhigh). Default is medium. Output length is controlled separately by a verbosity parameter.

That the two designs differ — Claude one dial for thinking + output, OpenAI thinking and output split — is central to the comparison later in this series.

What Comes Next

The series decomposes the dial in this order:

  • Part 2 — Claude's Thinking: the shift from a fixed budget (budget_tokens) to adaptive thinking, thinking blocks and display, hidden raw CoT.
  • Part 3 — Claude effort in full: behavior of each level low~max, per-model support and defaults, effect on tool calls, preamble, and thinking.
  • Part 4 — Claude Code's effort in practice: the five CLI levels, /effort and env vars, session persistence, what ultracode actually is.
  • Part 5 — OpenAI and Codex reasoning_effort: minimal~xhigh, GPT-5 / 5.5 / 5.2-codex differences, config.toml, separation from verbosity.
  • Part 6 — Comparison, tradeoffs, practical guide: Claude vs OpenAI summary, the cost-latency-quality curve, benchmark figures, matching the dial to task difficulty.

Next, we look at how Claude's thinking moved from a fixed budget to an adaptive mode the model sets for itself.

This article is grounded in Anthropic and OpenAI primary documentation. Exact parameter values, defaults, and per-model support are tabulated in Parts 3-5.

Series overview: Series index

๋Œ“๊ธ€

๋Œ“๊ธ€ ์“ฐ๊ธฐ

์ด ๋ธ”๋กœ๊ทธ์˜ ์ธ๊ธฐ ๊ฒŒ์‹œ๋ฌผ

"ML Foundations (9/9) — PyTorch vs TensorFlow, and the Road to Local LLMs"

์ด๋ฏธ์ง€
← 8/9 Deep Learning Architec… ๐Ÿ“š Series Index (series end) The final part. So far we've focused on models. Now we focus on the tools that actually run them . We'll lay out the philosophical difference between PyTorch and TensorFlow, and trace how Hugging Face transformers , llama.cpp , MLX , and Ollama built the bridge to running large language models on your own machine. By the end you should have the full mental model of "download a pretrained LLM and serve it locally." 0. Learning Objectives Explain the eager-vs-graph difference between PyTorch and TensorFlow. Explain, in graph terms, how autograd automates the backward pass. Use Hugging Face transformers ' AutoModel , AutoTokenizer , and pipeline abstractions. Describe llama.cpp's GGUF quantization, INT4 inference, and CPU-first flow. Describe Apple MLX's use of unified memory and how it differs from PyTorch. Run a local LLM with Ollama and call it through an OpenAI-compatible API. ...
Read more »

Agent Memory Engine (2/10) — Building an AI Agent Memory System with SQLite Alone

← 1/10 SQLite Memory Engine f… ๐Ÿ“š Series Index 3/10 memcore Completed → A memory engine that runs without daemons, without dependencies, anywhere ํ•ต์‹ฌ ์š”์•ฝ This post covers how to design an AI agent memory engine (memcore) on a single-file SQLite backend. - Structure: 25 modules, ~6,464 lines, 22 DB tables - Principles: no daemon, single-file portability, no required external dependencies, host-agnostic - Core techniques: 4-layer hybrid search (topic · vector · FTS5 · LIKE), U-tag dialectic-based confidence evolution, ingestion-layer bias rejection rules What This Post Covers One approach to building AI agent memory without a server or cloud vector DB. Specifically: (1) the limitations of text-file-based memory, (2) vector DB alternatives compared and selection criteria, (3) how to layer search tiers, (4) how to evolve confidence scores over time, and (5) how to block biased memory accumulation at the ingestion stage. Design Principles: Why Single-File SQLite Text-file-based ...
Read more »

"ML Foundations (8/9) — Deep Learning Architectures: CNN, RNN, Attention"

์ด๋ฏธ์ง€
← 7/9 Deep Learning Training ๐Ÿ“š Series Index 9/9 PyTorch vs TensorFlow → What did we keep adding on top of an MLP? Vision went CNN, sequences went RNN/LSTM, and eventually both converged on Attention and the Transformer. Each architecture is easier to remember if you read it as what it gave up to gain something else . This part is the one-line summary of the decisive inflection points: LeNet → AlexNet → ResNet → LSTM → Attention → Transformer. 0. Learning Objectives Explain why CNN's convolution, pooling, and weight sharing match images so well. Trace what got unblocked from LeNet → AlexNet → VGG → ResNet as depth grew. State BPTT for RNNs and the vanishing/exploding gradient problem. Write the LSTM gate equations (forget, input, output) with intuition. Write the attention formula in query/key/value form and the scaled dot-product variant. State the precise reasons Transformers replaced RNNs (parallelism + long-range dependencies). 1. ํ•ต์‹ฌ ์š”์•ฝ CNN : weight sharin...
Read more »

"RAG Core Study (14/26) — Evaluation Sets with RAGAS & DeepEval"

์ด๋ฏธ์ง€
Series overview: Series index ๐Ÿ“š Series Index Without an evaluation set, every RAG improvement is a story, not evidence. RAG systems fail in more than one place: retrieval, context selection, answer generation, and grounding. That means "looks better" is not a metric. Part 14 explains how to build a golden dataset , how RAGAS and DeepEval fit into the workflow, and why teams must separate retrieval quality from answer quality if they want tuning decisions to hold up. 0. Prerequisites Part 13 reranking — system quality now depends on multi-stage ranking. Part 1 grounding — a correct-looking answer is not enough. Part 12 Hybrid — multiple retrieval settings need comparison on fixed data. 1. Learning Objectives Build a small but useful golden dataset for RAG. Distinguish retrieval metrics from answer metrics . Use RAGAS and DeepEval in the right roles. Avoid the common traps in judge-model-based evaluation. 2. ํ•ต์‹ฌ ์š”์•ฝ An evaluation set for RAG is ...
Read more »

"ML Foundations (7/9) — Deep Learning Training: Optimizers, Regularization, Initialization"

← 6/9 Neural Networks ๐Ÿ“š Series Index 8/9 Deep Learning Architec… → Building an MLP, as we did in Part 6, is not the same as getting it to train well . The right optimizer, regularization, initialization, and learning rate — when these line up, deep networks converge. When they don't, the network refuses to learn at all. This part is the catalog of those crafts, with formulas, paper citations, and code in one place. 0. Learning Objectives Compare and write the update rules for SGD, Momentum, Nesterov, and Adam. Explain how Dropout, BatchNorm, and LayerNorm work and where they belong in a model. Derive the variance formulas for Xavier and He initialization and match them to activations. Implement step, cosine, and warmup learning-rate schedules in PyTorch. Explain why gradient clipping is effectively required for RNNs and Transformers. Diagnose the most common training failures (NaN, plateau, overfitting) and apply first-line fixes. 1. ํ•ต์‹ฌ ์š”์•ฝ SGD : \(w \leftarro...
Read more »

AI Agents I Built (5/7) — Building an Automated Blogger API Publishing System

← 4/7 My Life Organizer Agent ๐Ÿ“š Series Index 6/7 Dual → Designing a Markdown-Based Auto-Publish Pipeline with Blogger API v3 + OAuth 2.0 Key Summary Blogger API v3 + OAuth 2.0 enables a pipeline that converts Markdown files into publishable HTML posts blogger_publish.py handles the full flow — frontmatter parsing → body cleanup → HTML conversion → CSS injection → API post — in a single script For bulk publishing, quota management (delay, retry, TOC suppression) is the critical design concern Platform Selection Criteria API access is mandatory for AI agent-driven content publishing. A comparison of platforms: Platform API Cost AdSense Decision WordPress.com REST API Paid plan required Paid only Cost overhead Ghost REST API Self-hosted or paid Manual setup Ops overhead Blogger v3 API Free Native integration Adopted Blogger is free, exposes a REST API, and integrates natively with AdSense. For personal blog automation, this combination i...
Read more »

"ML Foundations (6/9) — Neural Networks: From Perceptron to MLP"

์ด๋ฏธ์ง€
← 5/9 Regularization & Model… ๐Ÿ“š Series Index 7/9 Deep Learning Training → A neural network is logistic regression from Part 4 stacked into layers. Each layer passes through a new nonlinear activation, expanding the family of functions the network can represent. In this part we put how a neural network learns onto one page. Perceptrons, XOR, MLPs, backpropagation, activation functions, and the universal approximation theorem all meet here. 0. Learning Objectives Write the perceptron's learning rule and explain why it cannot solve XOR. Express an MLP's forward pass as one line of matrix products. Derive backpropagation by hand on a two-layer MLP. Compare the definitions, derivatives, and trade-offs of sigmoid, tanh, ReLU, and GELU. State the universal approximation theorem (Cybenko 1989 / Hornik 1991) precisely. Train an MLP end-to-end in PyTorch. 1. ํ•ต์‹ฌ ์š”์•ฝ Perceptron (1958): \(\hat{y} = \mathrm{step}(w^\top x + b)\). Converges only on linearly separabl...
Read more »