Recap

Part 3 built the harness around the agent.

Retry and fallback kept one provider error from ending a run. A model router picked the model, and hard limits capped model and tool calls. Guardrails redacted PII, fenced injected instructions, and screened the request with Jev.

A permission matrix enforced roles at the tool boundary, and an approval interrupt paused destructive calls for a human.

Part 1 opened with a failure that none of this was built to catch. A model asked for services/checkout/flag.py, but the real file is flags.py. The tool reported the error, and the scripted model ignored it and answered anyway. The loop saw no more tool calls and returned the answer as a success.

Everything we built so far let it through, and there was a good reason. No permission was violated, because reading a file is allowed. No limit was hit, because the run took one tool call. The injection guard found nothing to fence.

The Answer schema was satisfied, because evidence is a string and a file path is a string.

The harness governs what each call may do. It does not know what the task was, so it cannot tell whether the task is done. The stop rule from Part 1 still gives that decision to the model.

Here, in this part, the agent's work becomes a LangGraph loop with an explicit plan. The Part 3 agent works one checklist item at a time, with every Part 3 control still in force. A verifier that the model does not control decides whether each item is done.

The loop repairs, replans, or stops, and every way it can end is a named terminal state. By the end, a careless model ends in a stuck status and nothing is written to memory.

Prerequisites: Parts 1 to 3 of this series. You can read them below:

First Steps Towards a Production-Ready Agent with LangChain

Part 1: Tools, structured output, runtime context, and tracing, in one small agent.

Daily Dose of Data ScienceAvi Chawla

Middleware-Driven Context Engineering in LangChain

Part 2: Dynamic prompts, memory, compression, and isolation with middleware

Daily Dose of Data ScienceAvi Chawla

Agent Harness with LangChain Middleware and Jev

Part 3: Retries, fallbacks, routing, guardrails, permissions, and the first human-in-the-loop interrupt.

Daily Dose of Data ScienceAvi Chawla

Let's begin!

Introduction

Every agent so far has used the same loop. Call the model. If it asked for tools, run them and call the model again. If it did not, stop. The stop rule is the last line, and it hands the decision to the model.

That rule has a known weakness. Huang et al. studied intrinsic self-correction. That is a model reviewing its own answer using only its own judgment.

On reasoning tasks, models struggled to correct themselves without external feedback. Sometimes the attempt made their performance worse. Asking a model whether it is finished relies on the same judgment and carries the same blind spot.

So what we need is a control cycle designed around the agent, so that "done" is established rather than claimed. Here we add things the default loop lacks:

• A goal the loop can check against, stored as data rather than implied by a prompt.
• A completion check the model does not control, grounded in tool output.
• Progress detection, so a loop that repeats itself is noticed.
• Hard stops, so every run ends in a known state with a report.

Every loop has the same basic parts, but control can sit in different places. A trigger starts it, a goal defines success, and the agent acts and observes. Then something decides whether to continue.

The loop stops because the goal is met or a budget has run out. In the default loop, the model owns that decision. Here, code does.

Note: Verification costs time and model calls. A verified loop makes more calls than a single agent run, and planning adds a call before any work happens. Thus, add complexity only when it improves outcomes you can measure.

For example, the support agent's answers can trigger a rollback. Preventing a plausible but wrong answer is worth the extra calls.

Note on the reference project:

The code and project setup are attached below as a zip file. Extract it and run uv sync to get going.

Download the zip file below:

Recap

Part 2 gave the support agent a context engine. Each call assembled its prompt from the tenant, environment, and role.

The middleware removed stale tool results from the model's view, moved large outputs to a store, and summarized long runs. It also kept memories inside tenant namespaces.

The next failures sat outside context management:

1. The prompt told a viewer not to suggest write actions, but no code blocked an attempted call of a possible write tool.
2. We described tool results as data, but an injected instruction still reached the model.
3. One transient provider error could still crash the run.

In each failure above, the context was correct. The model had the intended instructions and data. The missing code sat around the model call, where the application decides what may happen regardless of what the model requests. LangChain calls that code the harness and this part builds it.

Prerequisites: Parts 1 and 2 of this series. You can read them below:

First Steps Towards a Production-Ready Agent with LangChain

Part 1: Tools, structured output, runtime context, and tracing, in one small agent.

Daily Dose of Data ScienceAvi Chawla

Middleware-Driven Context Engineering in LangChain

Part 2: Dynamic prompts, memory, compression, and isolation with middleware

Daily Dose of Data ScienceAvi Chawla

Let's begin!

What a harness is

Part 1 used a simple definition, an agent is a model calling tools in a loop, and the harness is the code around that loop. Now that the surrounding code is the subject, the definition needs more precision.

So what we need to understand is, the harness connects the model to its environment, data, memory, and tools. The model-and-tool loop is familiar. The surrounding controls vary by application.

Hence, harness engineering is engineering the execution environment and policies surrounding an agent loop.

create_agent already provides a harness. Even without custom middleware, it has a prompt, tool registry, fixed loop shape, and structured-output handling. This part replaces its implicit defaults with components chosen for specific failures.

Middleware (contd.)

Part 2 used some of the hooks for context management. In this part, we'll explore the complete execution model.

The six hooks

The agent loop has two steps, a model call and a tool call, plus a start and an end. Middleware exposes hooks at each.

Hooks of the same kind run in list order for before_* and reverse list order for after_*. Wrapper hooks nest with the first item on the list as the outermost wrapper. So a before_model hook always runs before every wrap_model_call hook, regardless of their positions in the middleware list.

Composition

Two ordering relationships matter most here.

Retry and fallback both wrap the model call. With [fallback, retry], fallback is outermost. We call the primary model up to three times. If every attempt fails, the exception reaches fallback, which calls the second model.

Note on reference project:

The code and project setup are attached below as a zip file. You can extract it and run uv sync to get going.

Download the zip file below:

When an agent product integrates one harness directly, its backend starts depending on that harness’s task format, event stream, session model, and filesystem behavior.

Unlike model routing, adding another harness is not a config/parameter change because it adds another runtime that plans work, calls tools, manages files, and decides when a task is complete.

And supporting more than one runtime is critical since harnesses are built differently.

The right choice can depend on the task, required tools, provider access, available models, cost, latency, or usage limits. Like models, a product should be able to add or replace a harness without rebuilding the feature around it.

Model routing does not solve that problem. It chooses which model handles an inference request, while the same agent loop remains in place. Harness routing chooses the agent runtime that owns the complete task.

Today, let’s talk about the engineering behind that second form of routing. We will understand why products may need different harnesses, what must be standardized across them, and how the Unified Harness Protocol defines that contract.

We will then use HarnessRouter, an open-source UHP implementation, to run the same workflow through Codex and Claude Code.

Why a product may need another harness

Harnesses make different choices about tool execution, context management, permissions, skills, and supported models. The same harness may not provide the right tools or model support for every product feature.

A product may also serve customers with different provider access. One customer may already use OpenAI models, while another is set up around Anthropic. A quota or provider outage can also make one backend unavailable.

Evaluation is another reason. Running the same task through two harnesses makes their cost, latency, tool use, and output easier to compare. The comparison is difficult when each backend requires different task and event code.

The goal is not to switch harnesses during every session but rather to keep the product from depending on one runtime.

A team can select a harness per feature, offer a supported choice to customers, or replace a backend without changing the product workflow.

A model call does not run the whole task

A model API accepts input and returns generated output. The application still runs requested tools, starts follow-up model calls, and manages files, permissions, retries, and task state.

An agent harness includes this control loop. It can inspect files, call tools, edit a project, run commands, review the output, and decide when to stop.

Codex and Claude Code can use similar models and still behave differently. Their instructions, tools, context handling, permissions, and completion rules affect the run.

Model routing vs. harness routing

Model routing happens inside the inference layer. It chooses an endpoint based on price, latency, context length, availability, or measured quality. The application or harness still owns the agent loop.

Harness routing selects the runtime that owns the task. That choice can change the tools, skills, workspace, permissions, and event stream.

A product may use both routing layers. In UHP, model chooses the model, while metadata.harness_id chooses the configured harness.

What the product must handle for each harness

The first integration often starts as a small adapter that launches a process and reads its output. A product UI also needs live progress, session continuity, file uploads, download links, and access rules.

Cancellation needs a defined path because local processes, remote jobs, and sessions stop differently. Errors also need enough structure to separate provider failures, permission denials, time limits, missing files, and cancelled tasks.

Adding Claude Code beside Codex means building and maintaining these mappings again. The product needs one task lifecycle that both integrations can follow.

The task lifecycle defined by UHP

The Unified Harness Protocol defines the HTTP contract between an application and a harness server. It covers task creation, progress, sessions, files, cancellation, and errors without defining how the harness plans.

UHP uses an API shaped like the OpenAI Responses API. A client starts work through POST /v1/responses. The request includes the task input, model, configured Harness ID, and optional execution limits.

A response records one task. A session groups related responses that share conversation state and a working directory. A configured harness combines a base harness with its model, instructions, skills, tools, MCP servers, and limits.

Live tasks use Server-Sent Events, or SSE. Each event has a type and sequence number, so the client can process the stream in order.

The events report text output, tool activity, files, and task status. The application displays them without searching raw terminal lines for each event.

UHP accepts small files in the request and larger files through uploads. Generated files become artifacts that the application can list and download. The files specification defines those operations.

Follow-up tasks use previous_response_id. The server continues the same session with its conversation, working directory, files, and configured harness.

You can still hand off work to another harness. If Codex starts a task and you want Claude Code to take over, start a new session with Claude Code and pass it the relevant context, such as a summary of the work so far, the next instructions, and any files it needs. Think of it as handing a task to another developer: the work can continue, but Claude Code is not resuming Codex’s session.

For comparison, the application starts two sessions and gives both the same PDF and prompt. Each harness receives clear input instead of hidden state created by another runtime. The session specification documents this rule.

The current UHP version is 2026-09-12. The project publishes the written specification, OpenAPI 3.1 schema, JSON Schema definitions, and runnable conformance checks.

How HarnessRouter runs a task

HarnessRouter (GitHub Repo) is a self-hosted UHP server. It receives a UHP request, reads the Harness ID, and starts the selected runtime. As the harness works, HarnessRouter records its output as UHP events, responses, sessions, files, and errors.

The Docker image contains three services.

The Console provides the browser UI. The Gateway handles the UHP API and harness configuration. The Runner starts harness processes inside session workspaces.

A named /data volume stores the database, installed harness CLIs, provider integrations, session files, and workspaces. Only the Console port is published. The Gateway and Runner listen on loopback inside the container.

The self-hosting guide lists Codex, Claude Code, Hermes, DeepSeek Harness, Gemini CLI, OpenCode, Qwen Code, Cline, Goose, and other backends.

A built-in harness can run without a custom configuration. A custom harness starts with one base harness and adds a default model, instructions, tools, skills, and optional MCP servers. HarnessRouter gives the saved configuration a Harness ID.

The product sends this ID with each new task. It does not need to launch the harness CLI or parse the harness output itself.

Running HarnessRouter locally

Community Edition requires Docker, about 4 GB of disk, and an API key for a supported model provider.

Start the container.

docker run -d --name harnessrouter \
  -p 127.0.0.1:3000:3000 \
  -v harnessrouter:/data \
  harnessrouter/harnessrouter

The 127.0.0.1 binding keeps the Console on the local machine. The named volume preserves its data across container restarts.

Follow the startup logs.

docker logs -f harnessrouter

The first run installs the enabled harness CLIs:

Wait for this message:

[harnessrouter] ready on :3000

Open http://localhost:3000.

Next, open Integrations in the left panel and select Add Integration. Choose a model provider and enter its API key:

This key authorizes model requests. It is separate from the Console password and the API key later used by the application.

Open Agent harnesses in the left panel. You’ll see 11 built-in harnesses, including Codex, Claude Code, and Hermes. Select Hermes to create a task.

Select a model, create a task, and follow the run in the Console.

Files created by the harness remain attached to that session:

Configuring the paper workflow

Create two custom harnesses in the Console.

Paper Explainer with DeepSeek Harness
Paper Explainer with Claude Code

The first configuration uses DeepSeek Harness as its base harness. The second uses Codex. Both receive the same instructions and the same paper-explainer skill.

The skill defines the output expected by the paper application.

Read the uploaded research paper and identify its main mechanism.
Build a self-contained interactive explainer in index.html.
Include one controllable simulation with a short explanation beside it.
Use Three.js only when 3D interaction improves the explanation.
Create README.md with the source, assumptions, and run instructions.
Verify both files before completing the task.

The following video depicts this setup:

Upload the same paper to two new sessions. Run the same prompt through both configurations.

Turn the uploaded paper into an interactive visual explainer.
Recreate its main mechanism as a controllable simulation.
Return index.html and README.md.

The prompt, skill, and required files stay the same. Codex and DeepSeek Harness can still choose different plans, tools, and implementations.

• The first part of the video above shows the HarnessRouter dashboard and how the application selected and ran the harness.
• The final part of the video above shows what the agent produced.

Calling a harness from the application

After the task works in the Console, open http://localhost:3000/keys. Create a HarnessRouter API key and store it in backend code.

Set the local API address and the API key

export HARNESSROUTER_BASE_URL=http://localhost:3000/api/harness
export HARNESSROUTER_API_KEY=sk-hr-...

Send a task through the Responses-compatible endpoint.

curl --fail-with-body -sS "$HARNESSROUTER_BASE_URL/v1/responses" \
  -H "Authorization: Bearer ${HARNESSROUTER_API_KEY:?}" \
  -H "content-type: application/json" \
  -d '{
    "input": "Reply with exactly: it works.",
    "metadata": {"harness_id": "codex"},
    "model": "gpt-5.4-mini",
    "stream": false
  }'

Here’s that request running against the local endpoint, with the response printed in the terminal:

This invocation uses the built-in codex identifier from the repository quickstart. A custom configuration uses the Harness ID shown in the Console.

For another new task, changing metadata.harness_id selects another configured harness.

Here’s the demonstration with Hermes:

curl --fail-with-body -sS "$HARNESSROUTER_BASE_URL/v1/responses" \
  -H "Authorization: Bearer ${HARNESSROUTER_API_KEY:?}" \
  -H "content-type: application/json" \
  -d '{
    "input": "Reply with exactly: it works.",
    "metadata": {"harness_id": "hermes"},
    "model": "gpt-5.4-mini",
    "stream": false
  }'

As depicted above, the application keeps the same endpoint and event format. The same client can also continue sessions, retrieve files, and cancel work.

Session and security rules

The Unified Harness Protocol (UHP) standardizes the task lifecycle, but it does not make the harnesses behave the same. Different runtimes can choose different plans, tools, files, and models. The protocol gives the product one way to submit the work and inspect the result.

The local deployment also has a clear isolation limit. Sessions use separate operating-system users and workspaces. They do not run in separate containers. The Runner keeps the permissions needed to create those users, while each harness process runs as its assigned session user.

Provider credentials should stay outside environments that agent tools can read. Application API keys should never reach browser code. A deployment outside loopback also needs TLS and the controls described in the UHP security specification.

The two API calls above show why this matters when an agent becomes part of a product. Users may want to run a task with Codex, Claude Code, or Hermes.

Without a shared task interface, supporting each harness means writing another integration for progress, sessions, files, and cancellation.

HarnessRouter lets the product offer that choice in its own UI.

It sends the task through UHP, selects the harness, and receives the result through the same API.

The team can test those workflows locally before putting them in front of users. As harnesses change, the product has a way to support more of them without rebuilding its task flow each time.

HarnessRouter is Apache 2.0 licensed. The repository contains the self-hosted implementation, UHP schemas, and conformance tests.

GitHub Repo: github.com/HarnessRouter/harnessrouter.

(don’t forget to star it ⭐️)

Good day!

TypeSafe AI released Jev on September 15, 2026. It cannot hold a conversation, write code, or generate a useful paragraph. The narrow interface is the product.

Software makes thousands of small judgments: Is this ticket urgent? Which model should handle this request? Is this shell command dangerous? Teams often send each one to a general-purpose LLM, then parse and validate the generated answer.

Jev takes a different interface. Unstructured state goes in. Typed answers and probabilities come out. The relevant question is whether some LLM calls should have been bounded decisions in the first place.

Let’s dive in to learn more!

Generation is expensive control flow

Tool calling and structured outputs removed a lot of brittle parsing, but the model underneath is still generative.

Even when the answer is one word, such as “billing”, it produces tokens sequentially.

Now put that inside an agent loop.

while not done:
    action = llm(context)
    result = run_tool(action)
    context += result

One agent run may call the model repeatedly to select tools, detect risk, judge results, and decide when to stop. These calls require judgment, not prose. Jev targets them directly.

The interface: state, typed questions, and distributions

Jev is a semantic decision engine. You send it two things:

• State, which is the text or JSON that describes the current situation.
• Questions, which are the decisions you want it to make about that state.

Every question declares its answer shape in advance. Jev supports three primitives:

• Choice picks one option from a list and returns a probability for every option.
• Score places the input on an ordered scale such as low, medium, and high.
• Noul answers a yes-or-no question by returning the probability that it is true.

Noul is TypeSafe’s name for its Boolean decision type. It returns a number between 0 and 1.

{
  "model": "jev-latest",
  "state": "The deploy failed twice and customers are seeing 500s.",
  "questions": {
    "urgent": {
      "type": "noul",
      "instructions": "Does this need attention right now?"
    },
    "owner": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "engineering": "Product failures and outages",
        "billing": "Charges, invoices, and refunds",
        "sales": "Pricing and new accounts"
      }
    }
  }
}

The response contains an urgency probability and a distribution over the three teams. The schema prevents undeclared answers, but it cannot repair a bad taxonomy. If billing and engineering overlap in the rubric, Jev must choose between ambiguous categories.

Your program keeps control:

if urgent > 0.9 and owner == "engineering":
    page_on_call()
elif confidence < 0.6:
    send_to_human_review()
else:
    add_to_queue(owner)

Code owns the branches. Jev estimates which branch matches language that cannot be reduced to exact conditions.

TypeSafe says Jev evaluates questions in parallel. That helps when the questions are independent. If one answer determines the next question, keep that dependency in code rather than evaluating unused branches.

The company reports latency of 70 to 500 milliseconds and a price of $0.042 per million input tokens, with no charge for output. Its claims of up to 200 times lower latency and 400 times lower cost come from favorable internal comparisons, not production capacity data.

Scoring fixed outcomes avoids token generation, output parsing, and some retries.

Why confidence exists in control flow

Suppose Jev routes a ticket to billing. The label tells you what won. The distribution tells you how close the race was.

{
  "choice": "billing",
  "probabilities": {
    "billing": 0.52,
    "technical": 0.46,
    "sales": 0.02
  },
  "confidence": 0.18
}

Billing won by six points. Automatic routing would be reckless.

The application needs an abstention policy in addition to a winning label:

• Act automatically when confidence is high and the consequence is small.
• Ask for confirmation or call a stronger model when confidence is middling.
• Send the case to a person or gather more information when confidence is low.

Thresholds belong in code and should reflect consequence. A dashboard label can tolerate weaker evidence than a command that deletes data.

TypeSafe trains Jev using Reinforcement Learning for Calibrated Decisions, or RLCD, so predicted confidence should track observed accuracy. That relationship must hold on your traffic. Measure it separately for each decision type and recheck it after changes to the model, questions, or input distribution.

TypeSafe also says Jev cannot hallucinate. More precisely, it cannot break the declared output schema. It can still choose the wrong valid option with high confidence. Type safety prevents malformed output, not bad judgment.

Using Jev inside an agent

Jev belongs beside an LLM. The LLM plans, writes, explains, and uses tools. Jev handles repeated decisions around that work.

1) Model routing

A simple lookup does not need the same model as an architecture review. Jev can score a request and choose the least expensive model likely to complete it.

route = jev.choice(
    state=user_request,
    options={
        "fast": "Lookups, extraction, and small local edits",
        "powerful": "Architecture, ambiguity, and high-stakes work",
    },
)

model = fast_model if route == "fast" else powerful_model

The router chooses a model. It does not answer the request.

Pre-execution tool risk classification

Before an agent runs a shell command, Jev can classify it as read-only, reversible, or destructive. Separate questions can check whether it deletes files, changes Git history, touches production, or leaves the repository.

High-confidence read-only actions can continue. Destructive or uncertain actions pause for approval. LangChain’s Jev integration implements this check in middleware.

Post-execution semantic checks

Jev can check whether an agent is repeating itself, whether an output follows policy, or whether a result needs review. It should never replace a hard test. Use it only where the rule depends on meaning.

Problems Jev can solve

A workload is ideal when its output set is closed, a reviewer can label examples quickly, and the decision occurs often enough to justify another model dependency.

1) Request classification and routing

Jev can classify intent, urgency, department, spam, and customer frustration. One request can score several properties of the same ticket, while code combines the results into the routing policy.

2) Retrieval relevance and citation support

Embeddings retrieve related text. They do not establish that a passage answers the query or supports a claim. Jev can score those conditions before generation and remove irrelevant context.

Embeddings are excellent at finding semantically related text. Jev can make the narrower decision of whether a particular passage is useful for this question.

3) Semantic checks at enforcement boundaries

Jev can screen for prompt injection, policy violations, and risky tool calls. Its scores can inform an enforcement boundary, but they should not become one. Permissions, sandboxes, allowlists, and tests must enforce exact rules.

4) Corpus-scale semantic enrichment

Jev can label documents, products, and customer messages or turn free text into features for another model. Low per-call cost can make semantic scoring practical across every row of a dataset.

5) Real-time interfaces

Jev can choose among known browser actions or actions from structured game state. It is text-only, so the environment must first be represented as text or JSON.

Failure modes and exclusion criteria

Reject Jev when the task cannot be expressed as a stable set of outcomes.

• It cannot write, summarize, generate code, or explain its reasoning.
• Do arithmetic, counting, date comparison, and exact string manipulation in code.
• If a decision needs several hidden reasoning steps, split it into smaller questions or use a reasoning model.
• It cannot extract an unknown value directly. Find candidates first, then let Jev choose among them.
• Irrelevant context can reduce accuracy. Send only the state needed for the decision.
• Closed weights, early access, and limited independent calibration data make blind trust premature.

If deterministic code already solves the problem, keep it. An if statement is faster, cheaper, and easier to test than any model.

Deployment procedure and operational controls

A cheap model becomes expensive when mistakes cause retries, reviews, or incidents. Measure expected cost per decision, including model calls, escalations, false approvals, false blocks, and recovery work.

A production rollout needs evidence at each boundary:

1. Choose one bounded, low-risk decision with clear possible answers.
2. Write the rubric before calling the model. Define what belongs in every option.
3. Collect representative examples with expected answers, including ambiguous and adversarial cases.
4. Run Jev in shadow mode beside the current workflow without letting it change behavior.
5. Plot accuracy against confidence and set thresholds from your data.
6. Automate the safest branch first. Keep a person or stronger model for uncertain cases.
7. Pin or log the model version, questions, criteria, and thresholds so you can replay changes against the same evaluation set.

Questions and criteria are program logic. Version, review, and test them.

The architectural trade

Jev gives up generation in exchange for fixed answer types, explicit uncertainty, and code-owned branching. The LLM produces a plan, explanation, or code. Jev routes the request, gates risky actions, checks the result, and escalates uncertain cases.

Generation and judgment are different workloads. Forcing both through a text-generation API adds latency and turns control flow into prompt parsing. Jev exposes semantic judgment as a separate component.

Start with one bounded decision currently handled by a slow LLM call or a brittle regex. Send the minimum state, define the answers, and log Jev’s probabilities beside the current result. Automate one low-risk branch only after the evaluation supports it.

An if statement handles explicit values. Jev estimates the meaning needed to choose among them. The application decides whether that estimate is safe to act on.

Good day!

A dense transformer applies the same feed-forward network to every token. Batching changes the input matrix dimensions, but it does not change which weights execute.

An MoE layer replaces that feed-forward network with multiple experts. A router scores those experts for each token. Only the selected experts execute for that token.

This reduces executed expert computation per token. Although it does not determine which weights the next token will need, so the serving system must account for every selectable expert.

The Qwen3-30B-A3B model card reports 30.5 billion total parameters and 3.3 billion activated parameters. Each token selects eight of 128 routed experts.

The server must still make the full model available. It needs attention weights, embeddings, expert weights, temporary buffers, and a growing key-value cache. The 3.3 billion figure roughly describes active computation. It does not describe the deployment’s memory footprint or expert weights alone.

Activated parameters estimate the executed path for one token. Resident parameters determine the weight memory required by the deployment

In this article, we will follow a token through the MoE serving path. We will examine how the router selects experts, how the runtime groups tokens into expert batches, how local kernels execute them, and how activations move between GPUs.

We will then cover expert placement, load imbalance, capacity limits, quantization, and the measurements that reveal whether computation, memory bandwidth, or interconnect traffic is limiting performance.

To learn building an MoE-based Transformer from scratch, we covered LLaMA 4, using nothing but Python and PyTorch here →

It demystifies the architecture powering modern LLMs like LLaMA 4 and Mixtral, and gives you hands-on insight into how experts, routers, and sparse activation work in practice.

We’ll walk through every stage of implementation:Character-level tokenization,Multi-head self-attention with rotary positional embeddings (RoPE),Sparse routing with multiple expert MLPs,RMSNorm, residuals, and causal masking,And finally, training and generation.

MoE layer structure and token routing

A transformer layer applies attention before feed-forward computation. Attention mixes information across token positions. The feed-forward network then transforms each token position independently.

An MoE layer replaces one dense feed-forward network with multiple experts. Each expert is a separate feed-forward network.

Attention still processes every token. The router controls only the expert branch. Sparse expert activation therefore does not make the attention path sparse.

Consider a toy layer with four experts. Its router scores every expert for one token vector. The layer selects the two highest-scoring experts.

Those two experts process the same token vector independently. The layer then combines their outputs. Model-specific routing weights determine each contribution.

The weighted expert output can be written as follows.

Here:

• x is the token’s input vector.
• S(x) is its selected expert set.
• E_e is expert e, while w_e(x) is its routing weight.
• The output vector is y.

The sum covers only selected experts. Each expert transforms the same input vector. Its routing weight scales that expert’s contribution before addition.

The router selects several expert functions. The layer weights and sums their outputs. Architectures use different normalization rules for those weights.

Token-to-expert routing occurs inside each MoE layer. This is different from application-level model routing which chooses an LLM for an entire request.

The toy layer uses top-two routing for readability. Qwen3-30B-A3B selects eight of 128 experts per token. Its model card also reports 48 layers.

Some architectures include shared experts. These experts process every token regardless of router scores. DeepSeekMoE uses shared expert isolation alongside routed experts.

Shared experts execute for every token. Routed experts execute conditionally. Only models designed with shared experts use this split.

Do not assume every model has shared experts. Check its architecture and serving implementation first.

The router changes the feed-forward computation inside each MoE layer. Batched inference must then schedule token-expert assignments with different destination experts and matrix sizes.

How a batch becomes expert-specific matrices

The router makes a decision for each token. The GPU cannot efficiently run one tiny expert operation at a time. The serving engine therefore converts routing decisions into larger expert-specific matrices.

Consider four tokens entering a four-expert layer with top-two routing. The router makes these choices:

• T1 selects E1 with weight 0.7 and E3 with weight 0.3.
• T2 selects E1 with weight 0.4 and E3 with weight 0.6.
• T3 selects E1 with weight 0.7 and E2 with weight 0.3.
• T4 selects E2 with weight 0.8 and E4 with weight 0.2.

The runtime now has eight assignments, not four. Each assignment contains three things. It carries the token vector, the selected expert, and the routing weight.

The runtime groups these assignments by expert. E1 receives T1, T2, and T3. E2 receives T3 and T4. E3 receives T1 and T2. E4 receives T4. Those groups become matrices with three, two, two, and one rows.

Each expert processes its own matrix. The runtime keeps the original token ID beside every row. That ID tells it where each result must return.

• T1 receives one result from E1 and another from E3.
• The runtime multiplies them by 0.7 and 0.3.
• It then adds both contributions to produce T1’s final expert output.
• The same combine step applies to every token.

This grouping step is called dispatch. The return step is called combine. Both occur on one GPU. Multiple GPUs add network transfers between them.

An expert batch counts assignments, not unique source tokens. With top-two routing, every token contributes two rows across the expert matrices.

Prefill processes many prompt tokens in one pass. Experts can therefore receive matrices with many rows. Larger matrices usually use the GPU more efficiently.

Decode adds one new token for each active request. Low concurrency can leave an expert with one or two rows. Launching a matrix kernel for so little work wastes GPU capacity.

Higher concurrency creates more assignments in each step. Grouped general matrix multiplication, or grouped GEMM, runs several expert matrices in one launch. It reduces launch overhead, even when row counts differ.

Some engines pad expert matrices to fixed sizes. Fixed shapes make kernels easier to schedule. The trade-off is computation on padded rows that contain no token.

Prefill and decode do not have fixed performance profiles. Prompt length and request count change their expert matrix sizes. Hidden width and hardware determine how efficiently those matrices run.

Grouped GEMM packs existing expert work into fewer launches. It cannot create work for an expert that received no token.

In summary, routing creates assignments. Dispatch groups them into expert matrices. Combine returns the results to their original tokens. The next section follows those matrices through one GPU.

Local MoE execution pipeline

An MoE layer performs more than expert matrix multiplication. In fact, it must select experts, rearrange token rows, execute the experts, and restore token order. Each stage can consume meaningful time.

The router first produces one score for every eligible expert. A top-k operation keeps the selected expert indices and their weights. The runtime then rearranges token rows so each expert receives a contiguous matrix.

This rearrangement is called permutation. It reads token vectors and writes them in expert order. It also records an inverse mapping for the combine step. Small decode batches can spend as much time moving rows as multiplying them.

Grouped GEMM executes several expert matrices in one kernel launch. The matrices may have different row counts. Grouping reduces launches without changing their contents.

The expert then applies its activation and output projection. The runtime uses the inverse mapping to restore token order. It applies routing weights and adds contributions from matching token IDs.

Grouped GEMM does not change assignment counts. It executes the existing expert batches with fewer launches.

Kernel fusion joins operations that would otherwise launch separately. A fused path may avoid writing intermediate activations to GPU memory. It can therefore reduce launch cost and memory traffic.

Fusion support is conditional. A kernel may support only certain data types, batch shapes, or quantization formats. Unsupported combinations must use a more modular path.

Activation quantization creates another choice. The runtime can quantize before dispatch and send fewer bytes. It can also dispatch at higher precision and quantize before expert computation. The first option saves bandwidth but adds conversion work earlier.

Decode often gives each expert a small matrix. Weight reads, row movement, and kernel startup may then dominate. Prefill usually supplies more rows, which makes matrix efficiency more important.

A fused path removes supported boundaries between operations. It does not increase expert batch size. It also does not change router choices or assignment counts.

Measure routing, permutation, expert GEMM, and combination separately. One combined MoE timer cannot identify the slow stage.

In summary, expert GEMM is only one part of local execution. Small batches often expose the cost of movement and launches.

Model weight residency and runtime memory

MoE deployment uses three different parameter counts. Total parameters describe the checkpoint. Activated parameters describe one token’s executed path. Resident parameters describe the weights currently available to the serving system.

These counts need not match. One token touches a small expert subset. The next token can choose another subset.

A fully resident deployment keeps every selectable expert on its GPUs. It also stores attention, embeddings, normalization, and output weights. Sparse routing does not remove any of these weights.

Storing Qwen3-30B-A3B’s 30.5 billion parameters at two bytes each requires about 61 GB, or 56.8 GB, for the weights alone.

The following expression gives the weight-storage lower bound.

• B(weights) is the stored weight bytes.
• P is the total parameter count.
• B(stored) is the number of bytes used for each stored parameter.

The expression multiplies parameter count by bytes per parameter. For Qwen’s figures, that gives 61 billion bytes. Decimal gigabytes divide this number by one billion. The result covers weights only.

The deployment also needs temporary activations and communication buffers. Memory allocators reserve additional space. The key-value cache stores attention state for active sequences. None of that appears in the 61 gigabyte estimate.

Never replace 30.5 billion with 3.3 billion in this estimate. Activated parameters do not determine the checkpoint’s stored weight bytes.

Weight quantization stores each parameter with fewer bytes. Sharding divides the weights across GPUs. Neither changes how many parameters the checkpoint contains.

Sparse activation reduces expert computation without reducing total weight storage. The model may still require weight sharding despite modest active FLOPs.

Expert offloading keeps some experts in CPU memory. The runtime transfers one to a GPU when routing selects it. This saves GPU memory while adding a slower transfer path.

An absent expert can stall decoding while its weights move. Caching helps when the same experts are reused. Prefetching helps only when future selections are predictable.

There is no fixed GPU count implied by these figures. Precision, sharding, reserved memory, and KV capacity change the answer.

In summary, activated parameters estimate computation. Resident parameters determine weight memory. Sharding solves capacity by spreading weights, but routed activations may then cross GPUs.

Expert-parallel dispatch and combine

Expert parallelism assigns different experts to different GPUs. Consider four experts across two GPUs. GPU A owns E1 and E2. GPU B owns E3 and E4.

A token begins on GPU A and selects E1 and E3. Its E1 assignment stays on GPU A. Its E3 assignment must travel to GPU B.

The network record contains the token vector and routing metadata. The metadata identifies the token, destination expert, source GPU, and routing weight. The destination needs these fields to return the result correctly.

Each GPU groups local and received assignments by expert. It executes one matrix for each active local expert. Remote results then travel back to their source GPUs.

The source GPU restores token order using saved identifiers. It applies each routing weight and sums matching contributions. This completes the combine step.

With many GPUs, each GPU may send assignments to several peers. This produces an all-to-all communication pattern. Combine sends the expert results back after computation.

Expert weights normally remain on their assigned GPUs. The network moves token activations and metadata, not expert weights. Padding and temporary buffers can increase the transferred bytes.

The following upper bound estimates the dispatch activation payload.

• B(dispatch) is the dispatched activation payload.
• T is the token count entering the MoE layer.
• K is the selected experts per token.
• H is the hidden width.
• Finally, b is bytes per activation value.

The product counts one full hidden vector per assignment. It is an upper bound before considering locality. Assignments that stay on one GPU consume no network bandwidth.

Real traffic depends on how many assignments are remote. Metadata and padding add bytes. Expert replication can keep some assignments local. Combine adds traffic in the return direction.

The estimate covers activation payload only. It excludes protocol overhead and cannot predict latency without topology and message-size measurements.

Every MoE layer repeats this exchange. A small dispatch delay can accumulate across the model. Overlap hides some delay when independent computation is available.

DeepEP provides separate communication paths for large and small batches. Its throughput path targets larger transfers. Its low-latency path targets decode steps where startup time matters more.

Kernel bandwidth is not end-to-end serving speed. A full request also includes attention, scheduling, sampling, cache work, and every model layer.

In summary, expert parallelism keeps weights in place and moves assignments. It increases distributed weight capacity but adds repeated activation traffic.

Network topology and expert placement

Not every GPU link has the same cost. GPUs inside one server may use NVLink or NVSwitch. GPUs in different servers may use InfiniBand or Ethernet.

A cross-server transfer usually costs more than an in-server transfer. Expert placement decides which GPU stores each expert. That choice determines how often routing crosses slower links.

One common layout keeps frequent expert traffic inside an NVLink domain. Another parallel layout distributes layers across servers. The best boundary depends on the model and cluster.

Placement changes physical destinations without changing router outputs. A frequently selected expert can move closer to its token sources. The runtime can also create physical replicas of one logical expert.

Routing constraints are different because they limit the model’s choices. DeepSeek-V3 uses node-limited routing. Each token can reach experts on only a limited number of nodes. That rule belongs to the trained architecture.

Node-limited routing is part of that model’s trained architecture. Adding a similar restriction after training can change selected experts and model quality.

Placement needs routing traces and topology measurements. Routing traces show which experts receive work. Topology measurements show the cost of each destination. Expert popularity alone cannot capture link cost.

Placing popular experts together may reduce network traffic. It can also overload one server. Spreading them balances computation but may increase cross-server traffic. Replication helps only when spare GPU memory exists.

Measure bytes and time by communication domain. A single cluster-wide traffic counter cannot distinguish cheap intra-node movement from expensive cross-node movement. Also record which expert pairs produce that traffic, since placement needs a stable pattern to exploit.

In summary, placement tries to keep frequent traffic on fast links. It must also distribute work across GPUs. Routing traces reveal whether one layout can satisfy both goals.

Tensor, expert, and data-parallel execution layouts

Three parallel layouts divide different kinds of work. Tensor parallelism splits one matrix operation across GPUs. Expert parallelism assigns different experts to GPUs. Data parallelism gives different requests to replicas.

Tensor parallelism divides a weight matrix into shards. Each GPU computes a partial result. The GPUs exchange partial results to finish the layer.

Expert parallelism distributes whole experts or expert shards. Token assignments move toward their selected experts. Its traffic therefore depends on routing and placement.

Data parallelism sends different requests to separate replicas. Some engines replicate attention components inside data-parallel groups. They distribute experts across a larger group.

Real deployments combine these layouts. Attention layers and expert layers can use different GPU groups. One parallelism label cannot describe the whole deployment.

Consider tensor parallel size two and data parallel size four. Attention runs in four groups with two GPUs each. Expert layers can use all eight GPUs as one expert-parallel group.

Real deployments combine these layouts. Attention and expert layers can use different rank groups. One label rarely describes the entire model.

The vLLM expert parallel deployment guide documents this mixed approach. With expert parallelism enabled, its EP group size is the tensor-parallel size multiplied by the data-parallel size. Attention uses tensor parallelism within each data-parallel group when TP exceeds one.

For example, TP equal to two and DP equal to four create four attention groups. Each attention group spans two ranks. Expert layers can use an eight-rank EP group.

The TP2 and DP4 arrangement is a vLLM configuration example. Other engines may define different communication groups.

Tensor parallelism communicates inside each matrix operation. Expert parallelism moves routed activations. Data parallelism consumes memory for replicated components. Each layout pays a different cost.

Low concurrency can make expert-parallel transfers inefficient. Weak interconnects can produce the same result. Tensor parallelism can also struggle when collective communication dominates.

In summary, tensor parallelism divides operations. Expert parallelism divides expert ownership. Data parallelism divides requests. The useful combination depends on memory, traffic, and interconnect speed.

Expert load imbalance and rank-level tail latency

Routing rarely divides work perfectly. Suppose one expert receives half the assignments. Its GPU must process a larger matrix than the others.

Other GPUs may finish early and wait. The layer cannot finish until every required contribution returns. The slowest participating GPU therefore determines step latency.

Training can encourage balanced routing across a large dataset. One production batch can still be skewed. Language and request mix can change which experts are selected.

Expert imbalance occurs inside one model replica. Request imbalance occurs between replicas that receive different workloads. These problems need different measurements and fixes.

A balanced daily histogram can hide slow individual steps. Measure assignment counts and time at the serving-step level.

Record assignment counts for every serving step. Also record dispatch time, expert execution time, combine time, and GPU idle time. These measurements show whether one expert or link sets the tail.

Placement can separate experts that are frequently selected together. A hot expert can move to a less busy GPU. These changes keep router decisions intact.

Replication creates several physical copies of one logical expert. The router still selects the same logical expert. The runtime sends its assignments to an available copy.

Replication consumes GPU memory and leaves less space for the KV cache. Moving an expert also consumes bandwidth. A short traffic spike may end before rebalancing completes.

Some runtimes rebalance experts using recent load measurements. They may also support redundant expert copies. Both mechanisms need enough memory and a workload stable enough to learn from recent history.

👉 Rebalancing changes physical destinations. It does not change which logical experts the model selected or their intended outputs.

Average utilization cannot explain MoE tail latency. Measure each step and each GPU. Placement and replication can reduce the tail, but they consume memory and control bandwidth.

Inference optimizations and model-correctness boundaries

MoE optimizations fall into two groups. Some change only execution. Others change numerical values or the selected computation.

Fusion, grouped GEMM, placement, and communication overlap preserve logical expert selection. They change how the runtime performs the work. Small floating-point differences may still occur.

Quantization changes stored weights or transmitted activations. It can reduce memory and network traffic. It also introduces numerical error. Quality must therefore be tested on the target workload.

Reducing top-k changes the executed expert set. It removes expert contributions from the weighted sum. Renormalizing the remaining weights changes the result again.

Suppose the model normally selects eight experts. A runtime executes only four. The output now contains fewer expert functions. Renormalization also changes their combined scale.

The paper Training-Free Halving of Activated Experts studies this distinction. It executes fewer experts while using a larger reference set for normalization mass.

On Qwen3.6-35B-A3B, standard reduction from eight experts to four lost 4.65 MMLU points. Its adjusted reference mass reduced the measured loss to 0.35 points. The paper reports halved routed-expert compute for that setting.

The same method lost 0.55 MMLU points on Qwen3.5-397B-A17B when reducing ten experts to five. The authors also found that perplexity and task accuracy preferred different settings.

These are research results for specific checkpoints and evaluations. They do not establish a general rule for halving top-k.

Expert parallelism changes where selected computation runs. Lower top-k changes which computation runs. They are not equivalent optimizations.

Capacity limits require similar care. Some training systems cap tokens assigned to an expert. Particular runtimes may implement inference limits too.

MoE inference does not automatically drop tokens when an expert is busy. A capacity or dropping rule belongs to a specific model or runtime. Check the implementation before assuming one exists.

Optimize unchanged computation first. Treat quantization and top-k changes as model changes requiring quality tests.

Execution changes need performance tests. Quantization needs performance and quality tests. Top-k changes need the same tests plus careful output comparison.

Benchmark design and bottleneck diagnosis

A useful benchmark changes one serving decision at a time. Keep the checkpoint, precision, prompts, output limits, and request arrivals constant. Then change the parallel layout or kernel path.

Start with the smallest supported layout that fits the model. Add a same-server layout next. Test cross-server expert parallelism only after measuring those baselines.

Memory capacity may prevent one or more configurations from loading the model. Backend support may also exclude a kernel or communication layout.

Test low and high concurrency. Use short and long prompts. This separates small decode matrices from prompt-heavy prefill work.

Record prefill and output throughput. Measure time to first token and time per output token. Also record inter-token latency and p95 request latency. Tail measurements reveal delays caused by one hot GPU.

GPU-level measurements explain those request metrics. Record bytes by network domain, assignment counts, expert time, and idle time. Split routing, permutation, dispatch, GEMM, combine, and unpermutation. Also record resident weights and remaining KV capacity.

Interpret each stage separately. High dispatch time points to topology, locality, or message size. One persistently slow GPU points to routing skew or poor placement.

The diagnostic table maps common symptoms to initial measurements. Hardware topology and backend behavior can produce other causes.

High memory use with low expert compute indicates a weight-capacity problem. Little memory remaining for KV cache confirms the same constraint. More concurrency may improve matrix sizes while worsening cache pressure.

In summary, compare identical request traces and change one variable. Measure memory, computation, and communication separately. The largest measured cost determines the next experiment.

Engineering checklist

Qwen3-30B-A3B has 30.5 billion total parameters and 3.3 billion activated parameters. The total count drives weight storage. The activated count estimates the executed path for each token.

An MoE serving step includes routing, token permutation, dispatch, grouped expert computation, combine, and unpermutation. Multi-GPU deployments add communication domains, expert placement, and rank-level load imbalance.

Validate a deployment in this order. First, confirm weight and KV-cache capacity. Next, profile every stage of the MoE path. Then measure assignment skew and bytes by communication domain. Finally, compare placement, kernels, quantization, or top-k changes against the same request trace.

Placement and kernel changes target runtime overhead. Quantization changes numerical values. Top-k changes the executed expert set. Keep those experiments separate so performance gains and model-quality changes remain attributable.

MoE inference becomes easier to diagnose once three quantities remain separate. The deployment must store model weights, execute selected experts, and move routed activations. Measure each one independently before changing the serving layout.

To learn building an MoE-based Transformer from scratch, we covered LLaMA 4, using nothing but Python and PyTorch here →

It demystifies the architecture powering modern LLMs like LLaMA 4 and Mixtral, and gives you hands-on insight into how experts, routers, and sparse activation work in practice.

We’ll walk through every stage of implementation:Character-level tokenization,Multi-head self-attention with rotary positional embeddings (RoPE),Sparse routing with multiple expert MLPs,RMSNorm, residuals, and causal masking,And finally, training and generation.

👉 Over to you: Would you investigate memory, expert computation, or communication first in your deployment?

Good day!

Recap

At the end of Part 1, the support agent could answer a question with evidence.

It had four read tools scoped to a tenant. Its Answer schema forced every claim to carry evidence. Runtime context told tools who was asking, and tracing was on from the first run.

create_agent had replaced the hand-written loop. We had drawn the two-node graph it compiles to.

Now, before writing any new code, we first need to understand the limitations of the project as it stood at the end of Part 1:

• The agent's system prompt is the same for every tenant, every role, and every environment. An engineer in production and a viewer in dev get the same instructions. The agent had that information in its runtime context. It never reached the model.
• The message list grows without bound. A long investigation will overflow the context window, or become expensive long before that.
• And nothing learned in one run survives to the next.

The code behaved as written. The missing piece was control over the model's context. LangChain's documentation frames agent failures in the same way: a model call takes the wrong action because either the model wasn't capable enough, or it received the wrong context.

This article (Part 2) adds that control:

• Each model call gets a prompt assembled for the current requester.
• Middleware trims stale context, offloads large results, and summarizes long conversations.
• The agent writes useful findings to a store, then retrieves them on later runs without crossing tenant boundaries.

Prerequisite: Part 1 of this series. You can read it below:

First Steps Towards a Production-Ready Agent with LangChain

Part 1: Tools, structured output, runtime context, and tracing, in one small agent.

Daily Dose of Data ScienceAvi Chawla

Let's begin!

Context engineering

A model call has one input, the tokens it receives. Those tokens carry everything the model knows about the task, the user, the tools, and the work so far.

Context engineering decides what belongs in that input on each call.

Anthropic defines it as choosing and maintaining the best set of tokens during inference. That includes everything placed in the window beyond the prompt itself.

LangChain describes the job as giving the model the information and tools it needs, in a usable format.

Why agents make it hard

A single chat completion has a fixed input. An agent does not. Every tool call adds a request and a result. Every model turn adds a reply. Hence, the model's context is an accumulator.

Longer input does not produce uniform performance. Two studies show where it breaks down:

• In 2023, Liu et al. showed that model accuracy on retrieval and question answering depends on where the relevant information sits in the input. Accuracy is highest when it is near the start or the end. It drops when it is in the middle. The result is a U-shaped curve. Hence, position matters, not only presence.

• In July 2025, Chroma tested eighteen production models, including GPT-4.1, Claude 4, and Gemini 2.5. The tasks kept a fixed difficulty while only the input length grew. Every model degraded as the input got longer, even on simple retrieval and copying tasks. The degradation was worse when the relevant text did not obviously match the question, and distractors made it worse still.

The phenomenon is known as context rot. Since attention is a finite budget, every token added competes for it, and the model's ability to recall any given fact declines as the total grows.

Hence, the practical rule is simple, find the smallest set of high-signal tokens that maximizes the chance of the outcome you want.

The four operations

In June 2025, Lance Martin in his blog organized the practices agents use into four useful operations:

• Write: Save information outside the context window so it is available later. Scratchpads during a task and memories across tasks.
• Select: Pull the right information into the window for a step. Retrieval, memory lookup, tool selection, etc.
• Compress: Keep only the tokens the task needs. Trimming, summarization and pruning of stale results helps.
• Isolate: Split context so that different parts do not interfere. Separate agents, separate state, quarantined inputs, etc.

An agent loop keeps adding to the model's context, and model performance declines as that context grows. Write, select, compress, and isolate give us practical ways to control it.

Three kinds of context, three data sources

Before touching the agent, we need to separate what can be controlled from where the underlying data lives. The project follows LangChain's model for both.

What you can control

The docs divide context into three types, according to where in the agent loop it acts:

• Model context: What goes into each model call. Instructions, message history, tools, response format. It is transient.
• Tool context: What tools can read and write. State, store, runtime context. It's a persistent context type.
• Lifecycle context: What happens between model and tool calls. Summarization, guardrails, logging. This one is also persistent.

The transient versus persistent distinction decides most of the design in this part. For example:

• Trimming the message list for one call is a model-context operation. It should be transient. Nothing is lost; the full history is still in state.
• Summarizing old messages is a lifecycle operation. It is persistent by design. The originals are gone and the summary stands in for them.

Both are compression, but one is reversible and one is not.

Where the data comes from

The docs list three data sources the agent reads from and writes to:

Part 1 introduced runtime context and state. The store is new in this chapter.

The store

A store is a key-value database that outlives any single conversation.

LangGraph ships an interface for it and several implementations, from in-memory to Postgres.

Values are dictionaries. Keys are strings. Every key lives inside a namespace, which is a tuple of strings that works like a folder path.

There is no global lookup that crosses namespaces by accident. The project uses this to guarantee isolation, and a test proves it.

The store supports put, get, search, and delete. Search takes a namespace prefix and returns items under it, optionally filtered.

With an embedding index configured, search also accepts a natural-language query and ranks by similarity. Without one, it is a filtered listing.

Where a piece of information belongs

The project uses the following defaults to decide where information belongs.

The scratchpad row is the "write" operation inside a run. A planner that produces a checklist should put it in a state field instead of a chat message, so later nodes can read it as data. We'll be exploring more about this in a later part.

The final row refers to handling tool results that are too large for the conversation. The full output goes to the store, while state keeps a short stub and lookup key.

So overall, model context is transient; tool and lifecycle context persist. Runtime context, state, and the store each have a different scope. Keeping those boundaries explicit prevents accidental persistence and gives tenant namespaces a clear place to enforce isolation.

Note on reference project:

The code and project setup are attached below as a zip file. You can extract it and run uv sync to get going.

Download the zip file below:

An agent can look complete surprisingly quickly:

• The model has a question and a few tools.
• Each tool result gets sent back to the model.
• And stop when the model produces an answer.

The loop fits on one screen of Python.

The first time running it, it feels like the hard part is already over. Ask why checkout is slow, and the agent searches logs, reads a runbook, and returns a useful answer.

Then, say, you change the question.

The agent asks for services/checkout/flag.py. The real file is flags.py, so the tool reports that the file does not exist. The agent ignores the error and confidently gives an answer. And because it asks for no more tools, the loop declares the task complete.

Nothing crashes, and that is what makes the failure dangerous.

This article is the first step toward understanding failures like that. It builds the loop twice: first by hand, then with LangChain's create_agent. Along the way, we will see what an agent loop actually is, what the framework adds, and why "the model stopped" is a weak definition of success.

Let's begin!

The agent loop and the harness

Before writing code, we need two definitions that the rest of the course leans on.

Agent

LangChain's documentation defines an agent as a model calling tools in a loop until a task is complete.

The model reads the conversation so far. It either answers or asks for a tool. If it asks for a tool, the tool runs and its result goes back into the conversation. Then the model reads again.

Harness

The second term is harness. LangChain's documentation defines a harness as everything around that loop: the prompt, the tools, and any middleware that shapes the model's behavior.

These two definitions organize this course. An agent is a loop, and a harness surrounds it.

Note on reference project:

The code and project setup are attached below as a zip file. You can extract it and run uv sync to get going.

Download the zip file below:

Loading an LLM is only the first part of GPU memory planning.

Model weights stay roughly fixed during inference. The KV cache does not. It grows with every token in the sequence, and by default, every active sequence needs its own cache.

For Llama 3.1 70B, one 128K-token sequence requires about 40 GB of BF16 KV cache. Four full-length sequences would add another 160 GB, before accounting for model weights and serving overhead.

That creates two separate costs. The cache occupies GPU memory, and the attention layers must repeatedly read it while generating new tokens.

The techniques described as KV cache optimizations address different parts of this problem. Some change what the model stores. Some retain fewer tokens or use fewer bits. Others leave the logical cache intact and improve how the serving engine reads, allocates, shares, or moves it.

This article organizes twelve techniques by what they reduce, when they can be applied, and what trade-off remains. Each section includes a small runnable example that explains the main idea.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.

Start here →

The cache formula

Each attention layer stores one key and one value for every retained token.

The raw cache size for one sequence is:

The leading 2 counts keys and values. BF16 uses two bytes per value, while FP8 uses one.

That formula also gives us a clean way to group the techniques:

• GQA and MQA reduce the number of KV heads.
• Cross-layer attention reduces the number of unique cached layers.
• Sliding windows and eviction reduce the number of retained tokens.
• MLA reduces the width of the stored representation.
• Quantization reduces the bytes used by each value.
• Hybrid recurrent layers replace a growing cache with fixed-size state.
• Paging and prefix reuse reduce allocation waste and duplicate blocks.

Sparse attention needs a separate line. It may read fewer tokens without removing them from memory, so it can cut attention work while leaving cache capacity unchanged.

This distinction matters when the GPU is full. A faster attention kernel doesn’t create room for another sequence unless it also stores fewer bytes.

Technique map

The visual below puts all 12 techniques in one place, showing what each one changes and whether its main saving comes from fewer heads, layers, tokens, dimensions, bits, or duplicated memory.

The table above represents what each technique tries to tackle.

The rest of the article works through the full table in order.

1. GQA and MQA cache fewer heads

The cache formula above contains a term for KV heads. Reducing that term is one of the largest architectural savings available.

Inside each transformer layer, every token produces three representations:

• The query represents what the current token needs.
• The key describes what each earlier token can match.
• The value carries the information returned when that match receives attention.

During generation, the current token’s query is compared with cached keys.

Those comparisons produce attention weights, which combine the cached values. The layer keeps keys and values for later decoding steps. It does not keep the queries.

Attention splits these representations into smaller parts called heads.

A query head is one slice of the token’s query representation. It is not the user’s prompt or a separate model request. Different heads can learn to focus on different relationships in the same sequence.

• Standard multi-head attention, or MHA, gives each query head a matching key head and value head. A layer with eight query heads therefore produces eight key heads and eight value heads. Every attention layer performs this projection independently and keeps its own KV cache.
• Multi-query attention, or MQA, keeps all eight query heads but produces only one key head and one value head. Every query head reads from that shared KV pair. The sharing does not cross transformer layers.
• Grouped-query attention, or GQA, sits between these two layouts. It divides the query heads into groups and assigns one KV pair to each group. With eight query heads and two KV heads, four query heads share each KV pair.

All three methods operate inside each attention layer. They differ only in how many KV heads that layer stores. Cross-layer attention, covered next, is a separate idea that shares cached information between layers.

The GQA paper described it as an intermediate point between MHA and MQA.

The authors converted existing MHA checkpoints and used 5 percent of the original pretraining compute for recovery training. Their uptrained GQA models stayed close to MHA quality while approaching MQA speed.

The following example uses eight query heads. It shows which KV head each query head reads. It also counts the cached scalars for six tokens:

query_heads = 8
tokens = 6
head_size = 8

for name, kv_heads in [("MHA", 8), ("GQA", 2), ("MQA", 1)]:
    queries_per_kv_head = query_heads // kv_heads
    mapping = [head // queries_per_kv_head for head in range(query_heads)]
    cached_scalars = 2 * kv_heads * tokens * head_size

    print(name, "mapping:", mapping)
    print(name, "cached scalars:", cached_scalars)

# Output:
"""
MHA mapping: [0, 1, 2, 3, 4, 5, 6, 7]
MHA cached scalars: 768
GQA mapping: [0, 0, 0, 0, 1, 1, 1, 1]
GQA cached scalars: 192
MQA mapping: [0, 0, 0, 0, 0, 0, 0, 0]
MQA cached scalars: 96
"""

The mapping lists one KV head number for every query head. MHA uses a different KV head each time. GQA shares KV head 0 across four queries, then does the same with KV head 1. MQA maps every query head to KV head 0.

The cache count includes both keys and values. It then multiplies by six tokens and eight scalars per head.

Llama 3.1 70B uses 64 query heads and 8 KV heads. At 128K context, that produces a 40 GB BF16 cache instead of the 320 GB required by the same shape with 64 KV heads.

This code proves the storage ratio. It doesn’t test quality, because head sharing changes what the model learns and must be evaluated after training.

2. Cross-layer attention

GQA shares keys and values across heads within one layer. Cross-layer attention shares them across adjacent layers.

So a sharing factor of two implies two attention layers use one set of cached keys and values. The model now stores half as many unique layer caches.

The Cross-Layer Attention (CLA) paper trained 1B and 3B models and reported another 2x KV cache reduction while keeping accuracy close to the MQA baseline.

The next script represents two layers sharing one cache allocation:

tokens = 32
kv_heads = 2
head_size = 8

one_layer_cache = 2 * tokens * kv_heads * head_size
shared_cache = {"stored_scalars": one_layer_cache}
layer_caches = [shared_cache, shared_cache]

separate_total = 2 * one_layer_cache
shared_total = shared_cache["stored_scalars"]

print("layers point to same cache:", layer_caches[0] is layer_caches[1])
print("two separate layer caches:", separate_total, "scalars")
print("one shared layer cache:", shared_total, "scalars")

# Output:
"""
layers point to same cache: True
two separate layer caches: 2048 scalars
one shared layer cache: 1024 scalars
"""

One layer would store 1,024 scalars here. The leading 2 counts keys and values. Two independent layers would therefore store 2,048 scalars.

Both items in layer_caches point to the same Python object. The physical storage remains 1,024 scalars even though two logical layers use it.

Cross-layer sharing compounds GQA because it reduces a different term. GQA stores fewer KV heads inside each layer. CLA stores fewer distinct layer caches across the model.

A standard checkpoint learns separate KV projections for every layer. Each layer expects the cache created by its own weights. Redirecting it to another layer’s cache changes the model’s computation.

A CLA model trains with the sharing pattern already present. Training lets each dependent layer adapt to the shared cache. The serving engine must then reproduce the same ownership pattern.

The checkpoint and engine must agree on which layer owns each cache. This is why CLA cannot be enabled as a generic runtime flag.

3. Sliding windows

CLA reduces the number of stored layer caches. Sliding-window attention reduces the tokens stored inside selected layers.

In full self-attention, each new token can attend to every earlier token. Each attention layer must therefore keep keys and values for the full sequence. As the sequence grows, every layer’s KV cache grows with it.

Sliding-window attention limits a layer to recent tokens. Suppose the window contains 1,024 positions. That layer stores KV entries for only the latest 1,024 tokens. Each new entry removes the oldest one after the window fills. Its cache then stays fixed at 1,024 positions.

We call this a local layer because it sees nearby context. A global layer can still attend to every earlier token. Its cache keeps growing until the sequence ends.

Many models mix both layer types. Local layers provide most of the memory saving. Occasional global layers preserve access to distant parts of the prompt. The total cache still grows, but much slower than full attention everywhere.

For instance, Gemma 3 uses a repeating pattern of five local layers followed by one global layer. Its local window is 1,024 tokens, while the global layer supports the full 128K context:

The following example tracks the positions retained by a four-token window:

window_size = 4
retained_positions = []

for position in range(10):
    retained_positions.append(position)

    if len(retained_positions) > window_size:
        retained_positions.pop(0)

    print(f"after token {position}: {retained_positions}")

print("maximum retained positions:", window_size)

# Output:
"""
after token 0: [0]
after token 1: [0, 1]
after token 2: [0, 1, 2]
after token 3: [0, 1, 2, 3]
after token 4: [1, 2, 3, 4]
after token 5: [2, 3, 4, 5]
after token 6: [3, 4, 5, 6]
after token 7: [4, 5, 6, 7]
after token 8: [5, 6, 7, 8]
after token 9: [6, 7, 8, 9]
maximum retained positions: 4
"""

The list fills with positions 0 through 3. When position 4 arrives, the list becomes too long. Removing its first item drops position 0.

The same step repeats for every later token. Production engines usually use a ring buffer, which overwrites old slots without moving the remaining entries.

The first four tokens fill the available slots. Token 4 then removes token 0, so the window shifts to positions 1 through 4. Every later token shifts that range forward by one. After token 9, only positions 6 through 9 remain. The allocation stays fixed at four slots throughout.

The ratio also changes with context length. Below 1,024 tokens, both layouts retain the same number of positions. The saving appears only after the sequence exceeds the local window.

4. Multi-head latent attention

Multi-head latent attention, or MLA, doesn’t cache a complete key and value for every head. It compresses the hidden state into a smaller latent representation and stores that instead.

During decode, the model uses learned projections to recover the information needed by attention.

Implementations can absorb parts of those projections into the query and output paths, which avoids reconstructing every full KV tensor in memory.

DeepSeek-V2 introduced this design. The paper reported a 93.3 percent KV cache reduction compared with DeepSeek 67B and a 5.76x increase in maximum generation throughput:

An actual MLA layer uses learned matrices to create and use the latent representation. Those matrix operations hide the basic memory difference in a small example.

The following example counts storage for one token. The dimensions are intentionally small. They make the difference between both layouts visible.

kv_heads = 4
head_size = 8

standard_keys = kv_heads * head_size
standard_values = kv_heads * head_size
standard_total = standard_keys + standard_values

mla_latent = 8
mla_rotary_key = 4
mla_total = mla_latent + mla_rotary_key

print("standard cache:", standard_keys, "key +", standard_values, "value")
print("MLA cache:", mla_latent, "latent +", mla_rotary_key, "rotary key")
print("stored scalars per token:", standard_total, "->", mla_total)
print(f"example reduction: {standard_total / mla_total:.2f}x")

# Output:
"""
standard cache: 32 key + 32 value
MLA cache: 8 latent + 4 rotary key
stored scalars per token: 64 -> 12
example reduction: 5.33x
"""

Four KV heads with eight scalars each produce 32 key scalars. The values require another 32. Standard attention therefore stores 64 scalars per token.

The illustrative MLA layout stores eight latent scalars and four positional scalars. Its total is 12. Dividing 64 by 12 gives the example ratio.

The 5.33x figure belongs only to this example. Real MLA models choose different latent and positional dimensions.

MLA is not an inference flag for a standard checkpoint. Its projection weights and cache layout belong to the trained model. Converting another model still requires training.

5. Replace growing cache with fixed state using Hybrid models

Mamba and recurrent linear-attention layers don’t store one key and value for every old token. They update a fixed-size state as the sequence grows.

Pure recurrent models keep memory nearly constant with context, but full attention remains better at exact retrieval in many settings. Recent models combine both.

Qwen3-Next has 48 layers arranged as 12 repeated blocks. Each block contains three Gated DeltaNet layers and one full-attention layer. Only the 12 full-attention layers create a standard growing KV cache.

The model card reports 16 query heads, 2 KV heads, and a head dimension of 256 for those attention layers.

The next example compares their memory growth. The attention layer stores one 64-value key and value per token. The recurrent layer keeps one fixed 64-by-64 state.

width = 64
bytes_per_value = 4

attention_values_per_token = 2 * width
recurrent_state_values = width * width

for tokens in [128, 4096]:
    attention_kb = tokens * attention_values_per_token * bytes_per_value // 1024
    recurrent_kb = recurrent_state_values * bytes_per_value // 1024

    print(f"{tokens} tokens")
    print("  attention KV:", attention_kb, "KB")
    print("  recurrent state:", recurrent_kb, "KB")

# Output:
"""
128 tokens
  attention KV: 64 KB
  recurrent state: 16 KB
4096 tokens
  attention KV: 2048 KB
  recurrent state: 16 KB
"""

The attention calculation multiplies storage per token by sequence length. The recurrent calculation does not use tokens at all. Its state remains the same size in both cases.

The example tracks memory only. It does not implement Mamba or Gated DeltaNet. Those architectures use learned update rules to decide what the fixed state retains.

The attention cache grows 32x between the two sequence lengths. The recurrent state remains 16 KB. Hybrid models pay linear cache growth only in their full-attention layers.

For the Qwen3-Next attention shape, 12 full-attention layers require 3 GB of BF16 KV at 128K context.

Making all 48 layers full attention would raise that growing portion to 12 GB.

Jamba shows the same design choice with a different ratio. It interleaves one attention layer with seven Mamba layers. At 256K context, the Jamba paper reports a 4 GB KV cache, compared with 32 GB for Mixtral.

This layout belongs to the trained model. A serving engine cannot replace arbitrary attention layers with recurrent ones at runtime.

6. Compressed sparse attention

Full attention does two expensive things. It stores a separate KV entry for every earlier token, then reads every stored entry when generating the next token.

Sparse attention changes only the second part.

It reads fewer entries for each new token, but it may still keep the complete cache in memory. That can make generation faster without making the resident KV cache smaller.

Compressed sparse attention focuses on optimizing both parts.

• First, it combines several neighbouring token entries into one compressed entry. It stores that shorter sequence instead of one long-range entry per token.
• Then, for each new token, a selector chooses only the compressed entries that appear most relevant. Attention reads that selected subset.

DeepSeek V4 uses this design.

If the compression factor is m, every m token entries become one stored entry. So a context of n tokens produces roughly n / m compressed entries.

The model then selects k of those entries for attention. A small sliding window preserves finer detail for the most recent tokens.

At one million tokens, DeepSeek reports that V4-Pro uses 10 percent of the KV cache and 27 percent of the single-token inference FLOPs required by DeepSeek V3.2. V4-Flash goes further at 7 percent of the cache and 10 percent of the FLOPs

These are whole-model results. They include V4’s hybrid attention layout and precision choices, so they should not be read as the isolated effect of compression alone.

The next block isolates the two reductions. Compressing 32 long-range entries in groups of four leaves eight stored entries. If attention selects two of those eight, it reads only two compressed entries for the current step.

full_token_entries = 32
tokens_per_group = 4
selected_entries = 2

compressed_entries = full_token_entries // tokens_per_group

print("full token entries:", full_token_entries)
print("compressed entries stored:", compressed_entries)
print("compressed entries read:", selected_entries)
print("storage reduction:", f"{full_token_entries / compressed_entries:.0f}x")

# Output:
"""
full token entries: 32
compressed entries stored: 8
compressed entries read: 2
storage reduction: 4x
"""

Dividing 32 tokens into groups of four creates eight compressed entries. That is the storage reduction. Selecting two of those eight entries is the separate attention-work reduction.

The cache now holds eight long-range entries instead of 32. Attention reads two of them. A small local window separately preserves recent token detail.

This technique has to be built into the model and training process. A serving engine cannot convert an existing full-attention model into compressed sparse attention by changing a cache setting.

7. Query-aware sparse reads

The previous technique reduced both storage and attention work.

Sometimes, changing the model architecture is not an option. We may still want to reduce how much cache each decode step reads.

Recall what happens while the model generates a token. The new token produces a query, which acts like a search signal. Full attention compares it with every stored key. A long prompt therefore forces the GPU to load a large cache repeatedly.

Quest is a technique that reduces those reads without deleting any KV entries.

• It divides the cache into pages.
• A page is a small group of neighbouring token entries.
• Each page also stores the minimum and maximum key values found inside it.

For every new query, Quest scores these short page summaries first. It then loads only the pages with the highest scores. The full cache remains stored because a page ignored now may matter later.

The Quest paper reports up to 7.03x lower self-attention latency. Its complete inference system reached up to 2.23x speedup.

The example below uses four pages. Each page contains two keys. A key has only two numbers here, which keeps the scoring visible.

import numpy as np

pages = np.array([
    [[1, 0], [2, 1]],
    [[5, 1], [4, 2]],
    [[0, 6], [1, 5]],
    [[-2, -1], [-1, 0]],
])
query = np.array([1.0, 0.5])

page_min = pages.min(axis=1)
page_max = pages.max(axis=1)
page_scores = np.maximum(query * page_min, query * page_max).sum(axis=1)
selected_pages = np.argsort(page_scores)[-2:]

for page, score in enumerate(page_scores):
    print(f"page {page} score: {score:.1f}")

print("selected pages:", np.sort(selected_pages).tolist())
print("KV entries read:", len(selected_pages) * 2, "of", pages.size // 2)
print("KV entries still stored:", pages.size // 2)

# Output:
"""
page 0 score: 2.5
page 1 score: 6.0
page 2 score: 4.0
page 3 score: -1.0
selected pages: [1, 2]
KV entries read: 4 of 8
KV entries still stored: 8
"""

The pages array contains eight keys in total. The next two lines create one minimum and one maximum summary per page. Quest uses similar summaries to estimate a page’s best possible attention score.

The query prefers large values in both key dimensions. Pages 1 and 2 receive the highest scores, so the example reads their four keys. It leaves the other four keys untouched during this step.

If you notice the last line, the step reads four entries, but all eight remain resident. Quest reduces cache traffic and attention work. It does not reduce the memory needed to hold the full cache.

In practice, Quest uses many-dimensional keys and dedicated GPU code. It also repeats the selection for every query.

8. Quantization

Quest keeps every KV entry and skips some reads. Quantization takes another route. It keeps every entry but stores each number with fewer bits.

Keys and values normally contain floating-point numbers. BF16 stores each number with 16 bits. FP8 uses 8 bits, so its raw payload is roughly half as large. A 4-bit format cuts the payload to 25%.

Quantization maps many original numbers onto a smaller set of allowed values. A scale controls the gap between those values. The engine stores the small integer code and enough scale information to interpret it later.

The choice of scale affects accuracy. One extreme value can make a shared scale too wide. Most ordinary values then get rounded too aggressively.

KIVI handles keys and values differently. It groups key values by channel, which means the same feature across tokens. It groups value-cache numbers by token. The KIVI paper reports 2.6x lower peak memory and up to 4x larger batches with its 2-bit cache.

The small example below shows the basic mapping. It converts six BF16-style values into signed 4-bit codes. It then converts those codes back into approximate values.

import numpy as np

values = np.array([-1.0, -0.55, -0.10, 0.25, 0.60, 1.0])
bits = 4
largest_code = 2 ** (bits - 1) - 1
scale = np.max(np.abs(values)) / largest_code

codes = np.rint(values / scale)
codes = np.clip(codes, -largest_code, largest_code).astype(np.int8)
restored = codes * scale

print("original values:", values.tolist())
print("4-bit codes:", codes.tolist())
print("restored values:", np.round(restored, 3).tolist())
print("raw payload:", len(values) * 16, "bits ->", len(values) * 4, "bits")

# Output:
"""
original values: [-1.0, -0.55, -0.1, 0.25, 0.6, 1.0]
4-bit codes: [-7, -4, -1, 2, 4, 7]
restored values: [-1.0, -0.571, -0.143, 0.286, 0.571, 1.0]
raw payload: 96 bits -> 24 bits
"""

Signed 4-bit storage provides codes from -7 through 7 in this example. Dividing by scale maps each original value to a code. Multiplying by the same scale produces the approximate value used during attention.

As you may have noticed, the restored values are close, but not identical. That rounding difference is quantization error. The final line counts only the six stored values. A real cache also stores scale metadata.

The example uses one scale for all six values. KIVI uses smaller groups because each group can fit its own numerical range. This usually reduces rounding error, though it requires more scale metadata.

If you use vLLM, it exposes FP8 cache storage directly:

MODEL_ID=Qwen/Qwen2.5-7B-Instruct
vllm serve "$MODEL_ID" --kv-cache-dtype fp8

The first line defines the model name. The second starts a vLLM server and asks it to store the KV cache in FP8. This command keeps running because it launches the inference server.

Recent vLLM versions can also leave selected layers in their native type. This is useful when a sliding-window layer proves more sensitive:

MODEL_ID=google/gemma-3-4b-it
vllm serve "$MODEL_ID" \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window

This version leaves sliding-window layers in their original format. It quantizes the remaining cache layers. The option helps when those local-attention layers lose too much accuracy under FP8.

Quantization preserves every token position. The next technique reduces memory by removing positions instead.

9. Eviction

Quantization keeps all token positions. That may still be too large for a long context. Eviction sets a fixed position budget and discards entries outside it.

The difficult question is which positions should not be evicted. Recent tokens often matter because they contain the current conversation. Older tokens may still carry instructions, facts, or tool results needed later.

H2O is a technique that keeps recent positions and older heavy hitters. A heavy hitter is a token that has accumulated high attention during earlier steps. Once H2O evicts another token, its KV entry is no longer available.

Other methods estimate importance differently. SnapKV studies attention near the end of the input prompt. It uses that observation window to choose prompt positions before generation.

PyramidKV gives lower layers larger budgets and higher layers smaller ones, based on the attention patterns reported by its authors.

SnapKV reports 8.2x higher memory efficiency and 3.6x faster generation with 16K-token inputs. PyramidKV reports matching full-cache performance while retaining 12 percent of the cache in its LongBench experiments.

The following example takes a snapshot after ten tokens. Their scores represent attention accumulated during earlier decode steps. The cache can retain only six positions, including the latest two.

scores = [9.0, 0.2, 4.0, 0.1, 0.3, 6.0, 0.2, 0.4, 0.1, 0.2]
budget = 6
recent_count = 2

positions = list(range(len(scores)))
recent = positions[-recent_count:]
older = positions[:-recent_count]

heavy_hitter_count = budget - recent_count
heavy_hitters = sorted(
    older,
    key=lambda position: scores[position],
    reverse=True,
)[:heavy_hitter_count]

retained = sorted(heavy_hitters + recent)

print("heavy hitters:", sorted(heavy_hitters))
print("recent positions:", recent)
print("retained positions:", retained)
print("cache entries:", len(retained), "of", len(scores))

# Output:
"""
heavy hitters: [0, 2, 5, 7]
recent positions: [8, 9]
retained positions: [0, 2, 5, 7, 8, 9]
cache entries: 6 of 10
"""

Positions 8 and 9 are retained because they are the newest. Four slots remain. The sort selects positions 0, 2, 5, and 7 because they have the largest scores among older tokens.

The union of those two groups produces the final six-position cache. Positions 1, 3, 4, and 6 are discarded.

Real H2O updates its heavy-hitter statistics as generation proceeds. This smaller example starts with the accumulated scores so the selection rule remains visible.

That said, eviction needs a cautious evaluation. Importance changes across turns. A tool result that receives little attention now may become necessary after several more calls. System instructions and delimiters can also be disproportionately important.

Eviction evaluations should therefore use complete agent traces and delayed references. They should also include structured prompts. A good Needle-in-a-Haystack score does not cover those cases.

10. Paging

So far, we have looked at what one request stores or reads. Paging solves a different problem: how the serving engine allocates cache memory across many requests.

A simple allocator may reserve one continuous memory region per request. It often reserves enough space for the request’s maximum length. Shorter requests leave part of that region unused. Free space also gets split into gaps as requests finish.

PagedAttention divides GPU cache memory into equal-sized blocks. Each block holds KV entries for a fixed number of tokens. A request can use blocks found anywhere in the shared pool, so its blocks do not need to lie next to each other.

When a request finishes, the allocator returns its blocks to the pool. Another request can reuse them immediately. The PagedAttention paper reports near-zero KV cache waste and 2x to 4x higher throughput than the systems tested.

A fixed pool can hide the effect of quantization. The engine may reserve the same 8 GB before and after enabling FP8. Smaller entries increase the number of tokens that fit inside that pool. The allocated memory shown by nvidia-smi may remain unchanged.

The example below begins with six free blocks. Request A takes three. Request B takes two. A then finishes and returns its blocks before request C arrives.

free_blocks = [0, 1, 2, 3, 4, 5]

request_a = free_blocks[:3]
free_blocks = free_blocks[3:]

request_b = free_blocks[:2]
free_blocks = free_blocks[2:]

free_blocks.extend(request_a)
free_after_a = free_blocks.copy()
request_c = free_blocks[:2]
free_blocks = free_blocks[2:]

print("request A blocks:", request_a)
print("request B blocks:", request_b)
print("free after A finishes:", free_after_a)
print("request C blocks:", request_c)

# Output:
"""
request A blocks: [0, 1, 2]
request B blocks: [3, 4]
free after A finishes: [5, 0, 1, 2]
request C blocks: [5, 0]
"""

After A finishes, the free pool contains block 5 and A’s returned blocks. Request C takes blocks 5 and 0. Those block numbers are not adjacent, but the block table records their order for C.

This is the central paging idea. The allocator reuses available blocks instead of searching for one large continuous region.

vLLM also tracks which request owns each block. Its attention code can read the non-adjacent blocks in logical sequence order. The short example isolates allocation and reuse.

Quantization still changes the capacity inside this pool. An 8 GB pool holds 26,214 tokens at 320 KB per token, compared with 52,428 tokens at 160 KB. Total allocated GPU memory remains 8 GB in both cases.

11. Prefix reuse

Paging reuses blocks after a request finishes. Prefix caching can share blocks while several requests are still active. It applies when those requests begin with exactly the same tokens.

This happens often with system prompts and tool definitions. Without prefix caching, the engine computes and stores another KV copy for every request. The repeated prefix consumes memory and repeats the prefill work that created it.

Automatic prefix caching stores each completed prefix block under a lookup key. When another request has the same token prefix, it points to the existing KV block. New computation begins only after the requests diverge.

vLLM builds each lookup key from the current block and its preceding blocks. It also includes details such as adapter identifiers when they affect compatibility. The vLLM prefix-caching design documents this structure.

The example below uses four-token blocks. Each lookup key contains the entire prefix up to that block. This is a readable stand-in for vLLM’s chained hashes.

def prefix_keys(tokens, block_size):
    block_ends = range(block_size, len(tokens) + 1, block_size)
    return [tuple(tokens[:end]) for end in block_ends]

first = [10, 11, 12, 13, 20, 21, 22, 23, 30, 31, 32, 33]
second = [10, 11, 12, 13, 20, 21, 22, 23, 40, 41, 42, 43]

cache = set()
first_keys = prefix_keys(first, block_size=4)
first_hits = sum(key in cache for key in first_keys)
cache.update(first_keys)

second_keys = prefix_keys(second, block_size=4)
second_hits = sum(key in cache for key in second_keys)

print("first request reused:", first_hits, "of", len(first_keys), "blocks")
print("second request reused:", second_hits, "of", len(second_keys), "blocks")
print("second request computes:", len(second_keys) - second_hits, "new block")

# Output:
"""
first request reused: 0 of 3 blocks
second request reused: 2 of 3 blocks
second request computes: 1 new block
"""

The first request has no cache entries to reuse. It adds keys for its first four, eight, and twelve tokens. The second request has the same first eight tokens, so its first two keys match.

The final four tokens differ. That changes the last prefix key, so the second request computes one new block.

Using the full prefix prevents a false match. Two requests cannot reuse a block merely because four tokens match somewhere in the middle. Their preceding context must match as well.

Prefix reuse depends on exact tokenized prefixes. A timestamp in the system prompt, reordered JSON keys in a tool schema, or different chat-template whitespace can break the match.

Stable prompt templates therefore improve the hit rate. The model and cache format can remain unchanged while duplicate storage drops.

12. Cache Offloading from GPU

Prefix reuse helps when another request begins with tokens the engine has already processed. Offloading handles a different problem. It moves KV blocks that are not currently needed from GPU memory to a larger but slower memory tier, usually CPU memory.

Those blocks may belong to a sequence paused by the scheduler or to a cached prefix retained for reuse. When the engine needs them again, it copies them back to the GPU.

There is one important limitation. A new OpenAI-compatible API request does not automatically resume the KV cache from an earlier request. Reuse happens only if the engine still tracks the sequence, recognizes an exact prefix-cache match, or restores the cache through an external storage system.

Offloading frees GPU memory without deleting the stored KV data. It does not reduce the total number of bytes held across GPU and CPU memory. The trade-off appears when the engine must transfer those blocks back before computation can continue.

The example below gives the GPU room for two sessions. Adding session C moves the oldest session, A, to CPU memory. Resuming A brings it back and moves B out.

gpu_cache = ["cache-a", "cache-b"]
cpu_cache = []

# Cache C needs GPU space, so cache A moves to CPU memory.
cpu_cache.append(gpu_cache.pop(0))
gpu_cache.append("cache-c")

print("GPU after cache C arrives:", gpu_cache)
print("CPU after cache C arrives:", cpu_cache)

# Cache A is needed again, so it returns to the GPU.
cpu_cache.remove("cache-a")
cpu_cache.append(gpu_cache.pop(0))
gpu_cache.append("cache-a")

print("GPU after cache A returns:", gpu_cache)
print("CPU after cache A returns:", cpu_cache)

# Output:
"""
GPU after cache C arrives: ['cache-b', 'cache-c']
CPU after cache C arrives: ['cache-a']
GPU after cache A returns: ['cache-c', 'cache-a']
CPU after cache A returns: ['cache-b']
"""

The first move leaves caches B and C on the GPU while cache A remains available in CPU memory. When cache A is needed again, the example brings it back and moves cache B out.

The lists contain cache names instead of tensors. A real engine performs this operation on KV blocks and tracks which request or reusable prefix owns each block.

The example assumes that the engine has retained enough information to identify cache A later. Sending an unrelated API request with the same label would not restore it automatically.

vLLM can enable a 16 GB CPU offloading buffer directly:

MODEL_ID=Qwen/Qwen2.5-7B-Instruct

vllm serve "$MODEL_ID" \
  --kv-offloading-size 16 \
  --kv-offloading-backend native

The first option sets the CPU buffer size in GB. The second selects vLLM’s native CPU backend. Like the earlier serving command, this process keeps running after startup.

Offloading trades GPU capacity for transfer time. Measure resume latency and CPU bandwidth before enabling a large offload tier.

How to use them together

Some of these methods compose because they reduce different terms. Architecture-level methods still depend on how the checkpoint was trained.

For instance, take the 40 GB GQA cache from the first example.

• CLA2 can cut the unique layers in half, producing 20 GB.
• FP8 can halve the bytes per value, producing roughly 10 GB.
• A 50 percent token budget would reduce the resident cache to roughly 5 GB.

That multiplication is useful for capacity planning, but it hides model quality and kernel support.

• CLA requires training.
• FP8 introduces quantization error.
• Eviction discards context.
• And the serving engine must also support the resulting layout.

These are some situations you may face, along with solutions:

• When choosing a model → Inspect KV head count, local and global layer ratios, latent-attention dimensions, and recurrent layers. These determine the starting cache before deployment.
• When serving an existing model → Test FP8 first, then standardize prefixes. Both preserve the token set. Add eviction only after workload-specific quality tests.
• When memory appears unchanged → Inspect cache capacity and block counts. A fixed pool can absorb the savings while total allocated GPU memory stays flat.
• When latency is the problem → Measure cache reads and attention time. Quest-style sparse reads may help even when they don’t free capacity.
• When inactive sessions occupy the GPU → Test offloading and scheduling. Moving cold blocks can matter more than compressing the active ones further.

KV cache reduction is easier to reason about once each technique has a specific target.

This table summarizes all of the techniques we cover, what they alter, the takeaway from it, and what part they don’t handle:

Start with the constraint you need to change, then pick the technique that handles it.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →
• Read Part 2 on understanding the core building blocks of LLMs →
• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →
• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →
• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →
• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →
• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →
• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →
• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →
• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →
• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →
• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →
• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →
• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

👉 Over to you: how do you reduce KV cache in production?

Good day!

Recap

In Part 13, we prefilled a corpus once and answered from the stored cache. In Part 14, we tried to make that cache smaller, and mostly could not.

Compression splits into two families.

• One shrinks the stored numbers, which is query-agnostic by construction and can therefore run offline, but it ran out of room between 2x and 4x on a real model.
• The other drops tokens judged unimportant, which reaches far higher ratios but has to decide what matters, and the dominant methods answer that using an observation window sitting exactly where the user's query sits.

That is fatal for preloading, since a score computed from the query cannot be computed before one exists.

We measured how much it matters and found the answer depends on depth, with kept token sets agreeing 89 percent of the time at layer 1 and only 37 percent at layer 16. Compressing with one question's selection and then asking a different question produced a fluent answer naming the wrong depot and the wrong person.

Both approaches in this part require a training run, and both escape that ceiling because of it. One changes the model so that independent blocks stop being a problem. The other abandons the cache the model produced and trains a much smaller one to behave as though the corpus were present.

We then close the series with what running any of this in production actually involves.

Let's begin!

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

Modular preloading

Naive preloading treats the corpus as one indivisible block. That works when the whole corpus fits, and every query needs all of it.

But neither of those two conditions is practically feasible for long documents.

The modular version prefills each passage separately, so passages can be assembled in any combination at request time. This is the same construction Part 12 examined and found broken, for three specific reasons.

There is a different way to address it. Instead of repairing the cache after the fact, change the model so the damage never occurs.

Making blocks independent by design

Block-Attention takes this route.

The input is divided into blocks, each block computes its keys and values without attending to any other block, and only the final block, containing the user query, attends across everything.

The model is then fine-tuned to operate under this attention pattern.

Because the training distribution now matches how the cache is assembled at inference, the mismatch that caused quality loss in Part 12 no longer exists.

Each passage becomes a genuinely reusable unit, and its cache can be computed once and dropped into any request.

The implementation involves three pieces, which are block segmentation, position re-encoding so each block's keys carry the position they occupy in the assembled input, and the fine-tuning step itself.

That middle piece is the same repair we performed by hand in Part 12.

A block prefilled on its own has its keys rotated for positions starting at zero. Dropping it into the middle of an assembled input means re-rotating those keys to the positions the block now occupies.

Block-Attention does not escape that correction in any way, but rather it moves it into the serving path so it happens on every assembly automatically.

The fine-tuning step addresses the other failure, which is the missing cross-attention.

Part 12 corrected position and found the answer still wrong, because a block prefilled alone never saw the blocks before it.

Selective recomputation solved that by recomputing a slice of tokens at request time:

Fine-tuning solves it by teaching the model that blocks are meant to be independent, so there is no longer a discrepancy left to repair.

If you see these two approaches side by side, the two routes split the problem identically and solve with different costs involved in both.

Position is fixed arithmetically in both.

Cross-attention is either bought back with computation on every single request, or trained away once and never paid for again.

Why the training step is necessary

The same paper has also provided a useful result about the training-free alternatives, which also answer a question we left open in Part 12.

The authors re-implemented Prompt Cache (the earlier approach that precomputes block caches without fine-tuning), and found its results identical to their own no-fine-tuning baselines.

They also tested Parallel Context Windows, a method that stretches the usable context by cutting the input into several windows which each attend only within themselves, while the test tokens attend across all of them.

Its purpose is extending context length rather than making retrieval cheaper, and when its attention pattern was applied directly to retrieval, the accuracy landed below those same weak baselines.

This is direct evidence that simply precomputing independent blocks and hoping for the best does not work in retrieval settings. Something has to give, either recomputation as in the selective approaches from Part 12, or a change to the model itself as here.

In summary, modular preloading buys order-independence, which naive preloading cannot offer. Getting it without quality loss means either paying recomputation at request time or paying a fine-tuning cost once.

Trained preloading

Every approach so far kept the cache the model naturally produced, either whole, in blocks, or with pieces removed.

The last approach in the spectrum trains a cache instead.

The idea comes from Cartridges paper at Stanford, which we discussed briefly in the last part.

Rather than storing the cache a corpus produces, they train a much smaller cache to behave as though the corpus were present.

The setup

The starting observation of the paper is the same as the one this chapter opened with.

Users issue many queries against the same corpus, whether that is a codebase, a set of filings, or a chat history. Any cost paid once per corpus is amortized across every query that follows.

That makes an offline training run affordable in a way it would not be per request. The authors point out that this training can run on idle or underutilized compute, such as overnight when user load is low.

The artifact produced due to training is called a cartridge, and at inference time it is loaded in place of the corpus cache.

Structurally, it is not a compressed copy of anything.

Instead, a cartridge is a set of trainable key and value vectors, and a size parameter fixes how many positions it holds.

That parameter is a hyperparameter you specify when deciding how much memory the artifact is allowed to occupy, and it is chosen up front.

The authors describe the construction as a simplified form of prefix tuning, which is the established technique of learning a short sequence of vectors that is positioned in front of the real input and steers a frozen model's behaviour.

Do note that the trainable key and value vectors mentioned above are not initialized randomly.

They start as the real key-value cache of the first stretch of the corpus, as many tokens as the cartridge has positions, so training begins from something the model already reads as text rather than from noise.

They did because when they measured the approach's effectiveness by starting with random initialization, it reached 29.9 percent on the LongHealth benchmark, against 55.3 percent when initialized from actual corpus tokens.

Why the obvious training approach fails

The natural way to train a small cache is next-token prediction over the corpus, which is how language models are trained in the first place.

Four things in an LLM stack store four different objects, and all of them get called caching.

The KV cache stores attention tensors for one request.

Prefix caching stores those same tensors on the server, keyed by a hash chain over token IDs.

Prompt caching is the provider’s billed version of that same lookup, at 0.1x the base input rate on a read against a 1.25x premium on the write.

A semantic cache stores finished response strings, keyed by cosine similarity over an embedding.

The first three are exact-match and correctness-neutral, so a miss costs you money and latency. The fourth is fuzzy-match, and it will hand you a wrong answer with a 200.

So today, let’s go through all four, what each one stores, and what quietly breaks it.

Everything here runs on one machine, CPU included, with a 360M parameter model. There is also one Anthropic API example and one small semantic cache built on sentence-transformers. Where a mechanism only exists inside a serving engine, we walk the logic in pseudocode instead of pretending it is reproducible on a laptop.

Also, the cache API changed shape in transformers v5, so the snippets below assume v5 or later. On v4, the equivalents are DynamicCache() with no config argument and torch_dtype= instead of dtype=.

pip install "transformers>=5.0" torch

# only for the quantized cache example
pip install optimum-quanto

# only for the semantic cache example
pip install sentence-transformers 

# only for the prompt caching example
pip install anthropic

1) The KV cache

During prefill, the model computes a key and value vector for every prompt token at every layer and stores them.

Decoding then attends over those stored vectors and appends one new pair per generated token, instead of recomputing the whole sequence each step.

Queries don’t get cached, and the reason is causal masking. A token’s query vector is used once, at the step that token is processed, and never read again. Its key and value are read by every token that comes after it, so those are the two most important vectors to save.

Without storing them, each decode step requires a matrix-matrix multiply over the full sequence that has been generated so far.

With it, the step becomes a matrix-vector multiply over one new token, which is far fewer FLOPs.

The video below depicts LLM inference with and without KV caching:

While this reduces the computation on each token, you have to load the entire cache from HBM on every single step, so decode is no longer compute-bound but rather becomes memory bandwidth-bound.

Attention kernels finish faster than the cache can be streamed in, and the GPU spends most of a decode step waiting on memory.

KV cache growth with each token

The transformers library exposes the cache as a first-class object, so you can hold it, inspect it, and pass it back in.

Here is a minimal code demo of it:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache

model_id = "HuggingFaceTB/SmolLM2-360M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

inputs = tokenizer("The capital of France is", return_tensors="pt")
inputs = inputs.to(model.device)

past_key_values = DynamicCache(config=model.config)

out = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    past_key_values=past_key_values,
)

>>> print(tokenizer.decode(out[0], skip_special_tokens=True))
"""The capital of France is Paris. It is the largest city in
France and the second-largest city in the European Union."""

>>> print("prompt tokens: ", inputs["input_ids"].shape[1])
"prompt tokens: 5"

>>> print("total tokens: ", out.shape[1])
"total tokens:  25"

>>> print("cache length: ", past_key_values.get_seq_length())
"cache length:  24"

Normally, you invoke the generate method and the cache is created and destroyed internally, invisible to you. Here we construct a DynamicCache ourselves and hand it in, which means we still hold a reference to it after generation finishes.

get_seq_length() then reports how many token positions the cache holds. When you run this, the output contains the prompt length plus the tokens generated, minus one.

The final token's key and value are computed but never attended over by anything.

This code shows the cache holds one entry per token seen, and it grows by exactly one entry per decode step.

DynamicCache is used as the default because it grows as generation proceeds rather than pre-allocating, so short requests don't reserve memory they will never use.

The cache decides how many requests can fit on a GPU. Its size is fixed by the model shape and grows linearly with token count, since every layer holds a key and value tensor for every KV head.

For a 70B model at BF16, a single 128K context holds around 40 GB of cache, comparable to the entire model at 4-bit weights.

These are some ways to reduce this. For instance, Grouped-query attention shares one key and value head across a group of query heads, which shrinks the cache and raises FLOPs per byte of data loaded.

Multi-head latent attention in the DeepSeek line compresses the whole thing into a latent vector.

Cache quantization trades a little numerical accuracy for roughly double the capacity, and transformers implements it:

# requires: pip install optimum-quanto
out = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    cache_implementation="quantized",
    cache_config={"nbits": 4, "backend": "quanto"},
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Two arguments replace the default cache with a quantized one.

The KV values are stored at reduced precision, which reduces memory at the cost of quantizing and dequantizing on every access.

The backend also requires the group size to divide the model's head dimension evenly, so an unusual architecture can reject the config outright.

On short contexts, that overhead can make things slower rather than faster, so it is best used when running low on memory.

The cache is freed with the request

Everything above happens inside one call. The engine frees those blocks when the request finishes, so a 20-turn chat prefills turns 1 through 19 again on turn 20, at full cost.

You can see the alternative by keeping the cache alive yourself across turns.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache

model_id = "HuggingFaceTB/SmolLM2-360M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

past_key_values = DynamicCache(config=model.config)
messages = []

questions = ["What is the capital of France?", "And its population?"]

for prompt in questions:
    # Add to the history
    messages.append({"role": "user", "content": prompt})

   # Tokenize
    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt", return_dict=True
    ).to(model.device)

    # Generate
    input_length = inputs["input_ids"].shape[1]
    outputs = model.generate(
         **inputs, do_sample=False,
         max_new_tokens=64,
         past_key_values=past_key_values
    )

    # decode
    completion = tokenizer.decode(outputs[0, input_length:], skip_special_tokens=True)

    # Append to message history
    messages.append({"role": "assistant", "content": completion})
    print(f"turn tokens in: {input_length} | cache now: {past_key_values.get_seq_length()}")

# Output:
"turn tokens in: 42 | cache now: 55"
"turn tokens in: 71 | cache now: 92"

• The past_key_values object is created once, outside the loop, and passed into every generate call. That way, the cache is not freed at the end of turn one and is still populated when turn two begins.
• On each turn, we rebuild the full message list and re-render it through apply_chat_template. The prompt sent on turn two contains everything from turn one plus the new question.
• Because the cache already holds the tokens from turn one, the model only prefills the new suffix. The printed input_length grows every turn while the actual prefill work does not.
• The completion is sliced off the generated IDs and appended back into messages. That is what makes the next turn's prompt a strict extension of the last one.

Reuse only works because turn two's token sequence starts with turn one's token sequence, absolutely identical, bit by bit. If you edit anything earlier in the history, the cache becomes invalid.​

In this code demo, the cache belongs to one Python variable in one process. In a serving engine, it belongs to a shared pool that thousands of requests look up against. Let's learn about that next.

To dive deeper, we built all of this from scratch (with implementation) in ​Part 12 of the RAG systems course​, including the prefill and decode split and the cache memory formula, with the tensors printed out at each step. ​Read it here →

2) Prefix caching

The shared pool discussed above comes from one change in behavior.

When a request finishes, the engine keeps its KV blocks in memory instead of freeing them, and leaves them indexed so a later request can find them. That is prefix caching.

The index has to enforce the same rule covered in the chat loop, where reuse is only valid if the earlier tokens are identical.

vLLM does that by storing the cache of 16 tokens by default and identifying each block by a hash over the parent block's hash plus the token IDs inside it.

Chaining the parent hash into the child turns a block lookup into a prefix lookup, since a block only matches if everything before it matched too.

The scheduler iterates over the incoming blocks in order and stops at the first miss. A hit increments that block’s reference count, which also pins it against eviction while a request is using it.

Everything from the miss onward gets fresh allocation and a fresh prefill.

The lookup code

vLLM runs this inside its scheduler, wrapped in the memory management that owns the actual tensors.

The code below keeps only the two parts that decide reuse, i.e., the function that turns a token sequence into block keys and the function that walks those keys to work out how much of the prefix it can skip prefilling.

BLOCK_SIZE = 16

def block_hashes(token_ids, salt=None):
    """Chain-hash a token sequence into per-block keys."""

    hashes, parent = [], hash(salt)

    # Only complete blocks are hashed. A partial tail block is skipped.
    for start in range(0, len(token_ids) - BLOCK_SIZE + 1, BLOCK_SIZE):
        block = tuple(token_ids[start : start + BLOCK_SIZE])
        parent = hash((parent, block))
        hashes.append(parent)

    return hashes

def schedule(token_ids, cache):

    """Return how many tokens are reusable, and allocate the rest."""

    matched_blocks = 0

    for h in block_hashes(token_ids):
        if h not in cache:
            break                      # first miss ends all reuse
        cache[h].ref_count += 1        # pin it against eviction
        matched_blocks += 1

    reused_tokens = matched_blocks * BLOCK_SIZE
    to_prefill = token_ids[reused_tokens:]

    return reused_tokens, to_prefill

• The block_hashes method slices the token sequence into fixed 16-token blocks. Each block's key folds in the previous block's key through hash((parent, block)), so key number five encodes blocks one through five rather than block five alone.
• The range stops at len(token_ids) - BLOCK_SIZE + 1, which drops any partial block at the tail. Those tokens are never indexed and get recomputed on every request that ends there.

• The schedule method iterates over the keys in order and stops on the first missing one. There is no attempt to resume matching later in the sequence, because a later block's key already depends on the earlier one that failed.
• ref_count += 1 marks the block as in use. Eviction only touches blocks whose count is zero, which is what stops a running request from having its own cache pulled out from under it.

Whatever gets matched becomes reused_tokens, and everything after it is prefilled fresh.

There's one more important thing in the code we just discussed:

BLOCK_SIZE = 16

def block_hashes(token_ids, salt=None):
    """Chain-hash a token sequence into per-block keys."""

    hashes, parent = [], hash(salt)

    # Only complete blocks are hashed. A partial tail block is skipped.
    for start in range(0, len(token_ids) - BLOCK_SIZE + 1, BLOCK_SIZE):
        block = tuple(token_ids[start : start + BLOCK_SIZE])
        parent = hash((parent, block))
        hashes.append(parent)

    return hashes

Notice the salt argument in the function above.

When two requests send identical text, they produce identical block keys, so they end up pointing at the same physical KV blocks in GPU memory. There is one copy of those tensors, and both requests read it.

That is the behavior you want when both requests come from the same application.

But it may need a decision when they come from different customers. So passing a per-tenant value as the salt changes the first parent hash, so identical text now produces different keys for each tenant and their requests never land on the same blocks.

This way, every tenant gets its own copy, which costs memory and hit rate but provides separation.

Implementation in transformers

transformers lets you prefill a prompt once and reuse the resulting cache across several different continuations.

import copy
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, StaticCache

model_id = "HuggingFaceTB/SmolLM2-360M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

SHARED_PREFIX = """You are a careful assistant. 
                   Answer in one short sentence."""

prompt_cache = StaticCache(config=model.config, max_cache_len=1024)

prefix_inputs = tokenizer(SHARED_PREFIX, return_tensors="pt")
prefix_inputs = prefix_inputs.to(model.device)

# Prefill the shared prefix exactly once. No token is sampled here.
with torch.no_grad():
    prompt_cache = model(**prefix_inputs, past_key_values=prompt_cache)
    prompt_cache = prompt_cache.past_key_values

questions = ["What is the capital of France?", "Name one ocean."]

for question in questions:
    inputs = tokenizer(SHARED_PREFIX + question, return_tensors="pt")
    inputs = inputs.to(model.device)

    # each request gets its own copy
    past_key_values = copy.deepcopy(prompt_cache)   

    outputs = model.generate(
        **inputs, past_key_values=past_key_values, do_sample=False
    )
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

• StaticCache is used instead of DynamicCache because we need a fixed allocation we can copy around.
• The model(...) call is a prefill. No token is sampled here. We run the shared prefix through the model purely to populate the cache, then keep the returned past_key_values.
• Inside the loop, each question is concatenated onto the same prefix. The full string is tokenized, so the token IDs for the prefix portion are identical every time, which is exactly the condition the engine's hash chain checks for.
• copy.deepcopy gives each request its own copy of the prefilled cache. Generation mutates the cache in place by appending, so without the copy, the first question would corrupt the prefix for the second. A production engine does not copy the tensors. Instead, it shares the physical blocks and tracks reference counts, which is what makes reuse nearly free instead of proportional to prefix length.

The impact of eviction on hit rate

As discussed above, only complete blocks get indexed, so a trailing partial block is recomputed every time.

This means the block size should be tuned appropriately

• Larger blocks imply fewer table lookups and better memory locality
• Smaller blocks imply finer-grained sharing and less waste at the tail.

Eviction reduces hit rates, as expected.

The cache and the running batch draw from the same GPU memory pool, so a larger cache leads to fewer concurrent sequences, and under pressure vLLM drops unreferenced blocks by least recent use.

Mixed traffic makes this worse, because long shared prefixes occupy the most blocks and are the ones whose loss actually hurts.

Before you turn this on, you should know two things

It saves prefill only, so decode time is unchanged and crediting a whole speedup to the cache will overstate it.

And the hashing itself costs something, so on traffic with genuinely unique prompts, benchmarks have measured a throughput regression rather than a gain.

There’s a third problem, which is workload dependent, and it impacts RAG the most.

A RAG prompt includes a system instruction, then retrieved chunks, then the query, and the chunks change per request and change order between requests. Two requests that retrieve the same documents in a different order share nothing at all under the chain hash.

Prefilling each chunk on its own and stitching the caches together does not work.

The stitched tensors carry the wrong positional encoding. No chunk ever attended to any other chunk. And every chunk contributes its own attention sink at what the model thinks is position zero. Making it work needs partial recomputation at the boundaries rather than plain concatenation.

Btw, the solution already exists in open source.

LMCache (open-source) implements CacheBlend, wherein, instead of gluing the chunk caches end to end, it reuses them at any position and recomputes only a small subset of tokens, chosen by where the precomputed values deviate most from what full attention would have produced.

That subset restores the cross-chunk attention and fixes up the positional encoding, so the output holds at full-prefill quality.

This leads to an improvement in the time to first token by roughly two to three times compared to recomputing everything, with the recompute cost pipelined against fetching the cached chunks from slower storage.

It plugs into vLLM and reads the chunk boundaries out of your prompt, so retrieval traffic gets reused even when the retrieved documents arrive in a different order each time.

Here's the repo: https://github.com/LMCache/LMCache

To dive deeper, the ​Part 12 of the RAG course​ covers the solutions to the problem. It also explains the problem with the obvious repair of prefilling each chunk on its own and stitching the caches together, and the problems with it. ​Read it here →​

3) Prompt caching

On a hosted model, you don’t get any block table or the eviction policy. Instead, you get a price sheet over the provider’s own prefix reuse, plus two knobs for control.

The cached object is still KV tensors, not your prompt text, and it still requires an exact prefix match on the fully rendered context.

The rendered context includes provider-side system content you never wrote, which is part of why the minimum lengths and the invalidation rules look arbitrary from the outside.

Here's a version of prompt caching demonstrated with code:

import anthropic

client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

# Must clear the model's minimum cacheable length or nothing is cached at all.
LONG_INSTRUCTIONS = "You are a precise technical editor. " * 400

def ask(question: str):
    return client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        system=[
            {
                "type": "text",
                "text": LONG_INSTRUCTIONS,
                "cache_control": {"type": "ephemeral"},   # everything above is cacheable
            }
        ],
        messages=[{"role": "user", "content": question}],
    )

for question in ["Summarize section 3.", "Now rewrite it for a beginner."]:
    resp = ask(question)
    u = resp.usage
    print(
        f"write={u.cache_creation_input_tokens} "
        f"read={u.cache_read_input_tokens} "
        f"uncached={u.input_tokens}"
    )

# Output:
"write=2823  read=0     uncached=14"
"write=0     read=2823  uncached=17"

Only one line in that snippet touches the cache.

Where you specify cache_control decides which part of the request gets an entry written for it, and the usage counters tell you whether a later call read that entry back.

• The marker is attached to the last block you want covered, not to a range. It writes one cache entry spanning everything from the start of the request up to and including that block.
• The user message sits below the marker, so it stays outside the cached region since it changes every call, so it must not be inside.
• The usage counters tell you what's happening under the hood. The first call reports a non-zero cache_creation_input_tokens and a zero read. The second reports the reverse, and the instructions are billed at a tenth of the input rate.
• If both counters come back as zero, the prefix was below the model's minimum cacheable length, and the request was processed with no caching at all. No error is raised for this.

Intuitively (and as discussed above), if we move cache_control down onto the user message, the read counter will always be zero, because the marked block changes on every call.

The economics of prompt caching

Anthropic charges 1.25x the base input rate to write an entry and 0.1x to read it, with a higher write multiplier if you want it for a longer time. OpenAI applies the same two multipliers on its current models.

The premium cost is recovered in subsequent requests since anything reused inside the TTL will avoid any recomputation.

A read can only find an entry that some earlier request wrote, and writes happen only at a breakpoint you placed.

Each call checks your breakpoint, and on a miss it walks backward through a limited number of blocks looking for an older write.

Anthropic caps that at 20 blocks, so adding more than 20 blocks of conversation between two calls pushes the last write out of range and the hits stop.​

All of this is about reusing whatever the provider happened to keep around.

The other thing you can do is to prefill your corpus once, deliberately, before any query arrives, and pay to store the resulting cache instead of paying to recompute it.

​Part 13 of the RAG course​ extensively works through that, including the storage arithmetic and how many queries a preloaded cache has to serve before the offline prefill pays for itself. ​Read it here →

4) Semantic caching

The three techniques above save prefill work and still run the model.

A semantic cache embeds the incoming prompt, runs a nearest-neighbor search over stored prompts, and returns a stored response outright when the similarity exceeds a threshold.

That’s why it saves output tokens as well as input. It’s also why every request must bear an embedding round trip, including every miss.

Here's a working semantic cache demo in a few lines of code:

# requires: pip install sentence-transformers
import numpy as np
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")

class SemanticCache:
    def __init__(self, threshold=0.95):
        self.threshold = threshold
        self.vectors = np.empty((0, encoder.get_sentence_embedding_dimension()))
        self.prompts, self.responses = [], []

    def _embed(self, text):
        return encoder.encode([text], normalize_embeddings=True)[0]

    def lookup(self, prompt):
        vec = self._embed(prompt)
        if len(self.prompts) == 0:
            return None, 0.0, vec
        scores = self.vectors @ vec           # cosine sim, vectors are unit length
        best = int(np.argmax(scores))
        if scores[best] >= self.threshold:
            return self.responses[best], float(scores[best]), vec
        return None, float(scores[best]), vec

    def store(self, prompt, response, vec):
        self.vectors = np.vstack([self.vectors, vec])
        self.prompts.append(prompt)
        self.responses.append(response)

cache = SemanticCache(threshold=0.95)

def answer(prompt, call_model):
    hit, score, vec = cache.lookup(prompt)
    if hit is not None:
        return hit, f"HIT  (score {score:.3f})"
    response = call_model(prompt)             # the expensive path
    cache.store(prompt, response, vec)
    return response, f"MISS (best {score:.3f})"

# Stand in for the model so this runs without an API key.
fake_model = lambda p: f"<answer for {p!r}>"

for q in ["How do I reset my password?",
          "How can I reset my password?",
          "Is the API rate limited?"]:
    _, status = answer(q, fake_model)
    print(f"{status}  {q}")


# Output:
"MISS (best 0.000)  How do I reset my password?"
"HIT  (score 0.961)  How can I reset my password?"
"MISS (best 0.112)  Is the API rate limited?"

Every method in the class above maps onto a decision you have to make in production:

• normalize_embeddings=True makes every vector unit length, which lets self.vectors @ vec compute cosine similarity as a plain dot product. If you skip the normalization, the scores cannot be compared across prompts of different lengths.
• lookup returns the embedding alongside the result, so answer can store it later without re-embedding. That matters because the embedding is paid on every request, hit or miss, and computing it twice doubles the standing cost of having a cache at all.
• The brute-force argmax is fine for a demo and wrong at scale. Once you are past a few thousand entries, this becomes an approximate nearest neighbor index, which introduces its own recall setting on top of the threshold.
• store is called only on the miss path, after the model has answered. Nothing validates that answer before it becomes the response for every future prompt that scores above the threshold. This highlights the biggest risk with this technique. The cache has no notion of whether the stored response was correct, only of whether the new prompt looks similar to the old one.

The code below depicts the last point:

pairs = [
    ("How do I reset my password?", "How can I reset my password?"),
    ("Is the API rate limited?",     "Is the API not rate limited?"),
    ("Refund policy for annual plans", "Refund policy for monthly plans"),
]

for a, b in pairs:
    va, vb = encoder.encode([a, b], normalize_embeddings=True)
    print(f"{float(va @ vb):.3f}   {a!r}  vs  {b!r}")

This is the output we get:

0.961   'How do I reset my password?'  vs  'How can I reset my password?'
0.952   'Is the API rate limited?'  vs  'Is the API not rate limited?'
0.887   'Refund policy for annual plans'  vs  'Refund policy for monthly plans'

• The first pair is a genuine paraphrase and should share an answer.
• The second pair differs by one negation and needs opposite answers.
• The third differs by one operational value and needs different answers.

Despite some mismatches, the scores for all three are close together. The paraphrase and the negation are separated by less than a hundredth of a point, which is far too thin a margin to hold across real traffic.

• If you increase it, the hit rate collapses while you keep paying for embeddings on every call.
• If you decrease it, the hit rate climbs alongside the rate of confidently wrong answers.
• Published defaults range from 0.75 to 0.97 depending on who you ask, which tells you it’s a property of your traffic rather than a value to copy.

This is not a fully reliable technique per se since some failures (as demonstrated above) can bypass any threshold value, because they come from what embeddings represent.

Recap of all four techniques

Three of the four techniques discussed above are correctness-neutral, so their misses show up in cost and latency and nowhere else.

The semantic cache works in a different way, so hit rate is not the right metric to report here.

There is a fifth, lesser-used layer as well. It's exact-match response cache that returns a stored answer when the request is byte identical. It saves input and output like a semantic cache and carries no false positive risk, because it does no similarity matching at all. You just measure your byte-identical repeat rate before reaching for embeddings. There are problems, of course, as you can probably identify by now. Post them in replies.

Takeaways for production

Every technique has some failure point that you should note before using them in production:

• If you have any variable in the front of the prompt, like A timestamp, request id or user name in the system prompt, this invalidates every block after it. Always put stable content first, variable content last, and a marker on the boundary.

• Tool schemas are usually placed ahead of the system prompt, so a reorder can invalidate the whole cache.
• Check the settings that get rendered into the prompt. On Anthropic, toggling web search, citations, thinking config, or tool_choice rewrites the prompt text and invalidates downstream blocks. A/B testing two reasoning efforts splits your cache in two.
• Summarizing history rewrites the prefix, so the next call pays full price on cold tokens. Truncating tool outputs in place keeps the prefix byte-identical and the cache alive.

• Cache entries are keyed to a model, so routing to a cheaper one still prefills the whole accumulated history at cold rates.

To determine exactly where two prompts stop matching, compare their token IDs directly rather than the text you logged. Here's a demonstration:

messages_turn_1 = [{"role": "user", "content": "What is the capital of France?"}]
messages_turn_2 = [{"role": "system", "content": "Today is Tuesday."},
                   {"role": "user", "content": "What is the capital of France?"}]

# tokenize=True is the default and returns a plain list of token ids
a = tokenizer.apply_chat_template(messages_turn_1)
b = tokenizer.apply_chat_template(messages_turn_2)

shared = 0
for x, y in zip(a, b):
    if x != y:
        break
    shared += 1

print(f"shared prefix: {shared} tokens of {len(a)} and {len(b)}")
print(f"first divergence at index {shared}: {a[shared:shared+8]} vs {b[shared:shared+8]}")

# Output:
"""
shared prefix: 3 tokens of 35 and 26
diverges at index 3
  turn 1: [2683, 418, 253, 11173, 9042, 14260] You are a helpful AI assistant
  turn 2: [11814, 314, 27758, 30, 2, 198] Today is Tuesday.<|im_end|>
"""

Two prompts that look identical in your logs can differ by a beginning-of-sequence (BOS) token, a trailing newline, or a re-serialized tool schema.

Comparing token IDs instead of rendered text finds the exact index where reuse stops, and decoding the few IDs on either side usually finds the exact text.

The run above shows a common one.

Turn one specified no system message, so the chat template filled in the model's default, and the two prompts looked different at index 3, so no reuse was possible.

The first three layers cover one idea, applied at three scopes.

• The KV cache holds attention state for the duration of a single request.
• Prefix caching keeps that state after the request ends so a later request can look it up.
• Prompt caching is a provider running prefix caching on their own hardware and charging a separate rate for the part you reuse.

The semantic cache works differently. It stores response text keyed by embedding similarity, so if there's a hit, it skips the model entirely and saves output tokens along with input tokens. A hit can also be wrong, and it returns with a normal success status when it is.

As further reading:

• ​RAG course Part 12 covers the prefill and decode split and why prefix caching underperforms on RAG, with implementations →​
• ​RAG course Part 13 covers preloading a corpus into a cache before any query arrives →​
• ​RAG course Parts 14 and 15 cover cache compression that has to run before a query exists, plus the training-based approaches and what production involves →

👉 Over to you: which of these four layers has cost you the most debugging time?

Good day!

The engineering layer above the model: retrieval, context management, evaluation, and the patterns that make LLM applications reliable.

There is a habit that spread quietly among people running AI agents this year. They leave the laptop lid half-open so the agent can keep working. When they close it, the run dies.

That habit belongs to agents that run on your own machine. The hosted ones solved the lid problem long ago, and over the past year they have been solving the memory problem too, in pieces.

For instance:

• Devin restores a saved machine snapshot at the start of every session, so anything you did not commit or capture is gone.
• Manus puts its sandbox to sleep between tasks and recycles it after enough inactivity, with an always-on cloud computer available.
• ChatGPT Work keeps browser cookies, so once you sign in to a site through the takeover flow, later runs pick it up.

Each of those persists one thing, for one agent, usually scoped to a session or a project. You still get a machine per task, and the roster of agents does not share anything.

SpaceXAI launched Grok Bot last week. Instead of a machine per task, you get a machine per person. It stays on, it is tied to your account, and the whole roster of Bots shares it.

This guide covers what a Bot actually is, how the system is put together underneath, how it compares to Hermes Agent and the other options, how to set it up on macOS, and which roles are worth building first.

By the end, you will have a working roster on your own machine, a saved skill, and a routine that runs while your laptop is closed.

Let’s begin!

What Grok Bot is

The one-line version is that a Bot is a named teammate you message, and the actual computing happens one layer below it.

The word “bot” suggests a separate program running somewhere. It is closer to a saved role with its own memory, sitting on a machine it shares with every other role you create.

A Bot is defined by three things you write:

• A short name
• One primary job
• And a description of how it should work.

In exchange, you get a persistent conversation thread, and the Bot keeps working context across turns.

As per the official docs, a Bot retains stable working preferences, important facts, and summaries from its work. It does not replay every prior message.

The computer is the other half. Every Bot on your account uses one persistent cloud computer with a browser, a filesystem, and a terminal. It is assigned to your user account, not to any individual Bot.

Each Bot gets its own screen on that shared machine. Several Bots can therefore drive browser and desktop tools at the same time, though one Bot runs one computer-use task on its screen at a time

In the docs and in the app, a Bot means one persistent named agent. It is not a separate machine, a separate container, or a separate model instance.

As a useful analogy, you can relate it to an office setting.

The cloud computer is the office, and each Bot is a coworker with their own desk. There are different people, jobs, and notebooks, but one filing cabinet and one set of keys.

That analogy also explains the trade-off, which we will return to in detail. Shared keys make handoffs free. They also mean the roster shares one set of consequences.

In summary, a Bot is a role plus a memory plus a thread. The machine underneath is singular, persistent, and shared.

How it is built

Before the setup, let’s look at how all the pieces fit together.

Specifically, there are five parts you should know about Grok Bot:

The persistent cloud computer

The computer runs in SpaceXAI’s cloud, not on your Mac. So closing the app or the laptop does not stop a background turn or a scheduled routine.

There’s a durability contract, which means that there is a shared workspace at /workspace, and files, browser state, and supported sign-ins are designed to survive normal computer updates and recovery.

Everything else is explicitly replaceable. Temporary directories, manually installed packages, and uncommitted application state carry no guarantee. Durable project files belong in the shared workspace.

SpaceXAI has not published the operating system, the image contents, or the machine specification. One widely shared report from a user who inspected the environment describes Debian with roughly eight virtual CPUs, around sixteen gigabytes of memory, and a disk in the hundred-gigabyte range, with no GPU.

The sharing model

As mentioned above, every Bot on your account uses the same computer, and the docs enumerate exactly what that means.

Browser cookies and signed-in sessions are shared. Files are visible to every Bot. Command-line credentials are shared. One Bot can continue from work another Bot saved.

This means that a research Bot can save a file, and a writing Bot picks it up with no upload or additional login. The handoff costs nothing because nothing has to be re-established.

Under this shared setting, you should not place a credential or a file on the computer if another Bot on your account should not use it.

The action layer

Grok Bot has two ways to interact with a service.

A connector gives a Bot a structured path into a supported service. Connectors appear as Plugins in the current app, and they are account-wide rather than per Bot.

The browser handles everything else like vendor portals, ad managers, older internal tools, anything without a clean programmatic interface.

Ideally, prefer a connector when one is available, because it is often more reliable than clicking through a website.

Since programmatic coverage of business software can be limited, it falls back to pixel-level control, which also means that it inherits every fragility of the underlying site.

So a layout change, a new consent prompt, or a session timeout can break a workflow that ran fine earlier.

Authentication and the takeover flow

Sessions persist on the shared browser, so you usually do not sign in again for each task. Signing in once for one Bot makes that session available to your other Bots.

For anything sensitive, the Bot hands you the machine, like in case of passwords, passkeys, two-factor codes, CAPTCHAs, payment or identity checks, and sites that explicitly require a human.

Practically, you open the Agent Computer, take control, complete only the blocked step, return control, and tell the Bot to continue.

For supported connections, there is a secure secret request instead. The value is masked, excluded from the transcript, and not shown to the model.

Never paste a password or a one-time code into ordinary chat. The takeover flow exists precisely so those values stay out of the transcript.

Agent Memory

Memory is maintained per Bot.

Duplicating a Bot copies its profile, settings, enabled skills, routines, and avatar, but not its conversation history or learned memory.

The mechanism behind that memory is not published. Whether it is a summary buffer, a file on disk, a retrieval index, or some combination is not stated anywhere in the documentation.

In summary, the engineering story is one durable machine, a connector-first action layer with a browser fallback, human-in-the-loop authentication, and per-Bot memory whose internals stay private.

Bots, skills, and routines

Just like with any other agent, you should run one real task first. Then correct the output until it is reliable. Once you are satisfied, save the method as a skill.

This video depicts that process:

Once you are satisfied, only then turn it into a routine.

Creating a Bot properly

Press Cmd and N or click on the “+” icon, choose Create new agent, then open Bot actions and Edit Profile to set the name, title, description, and avatar.

You can also chat with the bot and describe what you need to fill this info:

The description holds rules that should stay true, and the conversation holds task-specific instructions.

A description like “never send external messages without approval” belongs in the profile. “Draft follow-ups for these twelve accounts” belongs in the thread.

Like any other Agent, specific jobs beat general ones.

For instance, Talent Scout and Expense Manager give the Bot something to build context around. General Helper gives it almost nothing, and its saved context becomes hard to reuse.

An account can hold up to 50 Bots and group chats combined. Start with the smallest useful roster and add a Bot only when a job has a stable specialist role.

Skills

A skill is a reusable set of instructions for how to do a task. It captures the steps, the decision rules, the expected output, and the safety boundaries.

Skills are available across your Bots, though a Bot may still need the relevant connector or login to use one.

You reference a saved skill by typing a forward slash (/) in the composer, and you mention Bots, groups, routines, and connectors with the at sign.

An ideal skill should mention six things:

• When to use it
• The required inputs and access
• The sequence of work
• How to validate the result
• What to return
• And what requires approval.

Teaching a skill by demonstration

When the Teach a task control is available, you can show a browser workflow instead of describing it.

Open a one-to-one conversation and its computer view, choose Teach a task, describe the result, perform the workflow once, then stop the recording.

The video below depicts this in action:

Teaching records visible computer interaction for up to ten minutes. Do note that it does not record microphone audio.

Recording lowers the barrier for people who would never write a procedure by hand. It also moves the work of hardening that procedure onto the human.

Important: Re-test a demonstrated skill after any website, connector, or source format change. A workflow learned from one pass through a site has no defence against that site changing.

Routines

A routine assigns a workflow to one Bot and tells it when to run. Schedules are the common case, and event triggers are supported where the underlying integration exists.

You can create one by asking the owning Bot in plain language.

A good request names the schedule and time zone, the input source, the expected result, the approval boundary, and what should happen when a source is missing.

Event triggers come through Cursor account integrations, such as a Slack message or a GitHub notification. These are separate from the Slack and GitHub plugins and may need their own connection flow.

Use a test run before enabling anything. A test run performs real work, so keep the inputs safe and write actions behind approval.

A Bot can own up to 50 routines, and the app keeps the 20 most recent run records for each. Deleting a routine is permanent and does not support undo.

In summary, skills describe how, routines decide when, and the gap between a working task and a safe automation is filled by you.

Working with more than one Bot

One Bot is useful. But ideally, the reason to add a second is to have another stable specialist role.

Group chats hold two to six Bots and exist for handoffs that need to be visible. You select the Bots, describe the shared outcome, and say who owns the next step.

You can interact normally, and the participating Bots decide who answers.

Alternatively, you can use the “at” sign when one teammate owns the request, and reserve the everyone mention for genuine group updates.

Bots also message each other asynchronously outside groups. The receiving Bot wakes, handles the request, and replies later, and the handoff stays visible in the conversation.

One current limit is that Bot-to-group handoff messages are text-only, so a Bot that needs another to inspect an image should send it directly.

Ideally ask for a single bot at each stage. Too many parallel handoffs create duplicate work and noisy updates, which is the failure mode that shows up once a roster grows past four or five Bots.

A direct message from you outranks background work and can redirect the current turn.

Auto review

Where enforcement is available, Grok Bot evaluates tool calls and computer actions before they run.

• Require Approval rules always stop matching actions.
• Always Allow rules let matching actions proceed only when the automated review finds no other reason to stop.
• When both match, Require Approval wins.

Two important caveats to note:

• Auto Review is model-based, so the docs position it as a complement to least privilege rather than a replacement.
• Personal rules are stored on the current desktop and synced to its computer, which means a second installation needs its own check.

What happens to your local machine

By this point, it should be clear that the cloud computer and your Mac are different things.

A Bot only runs commands locally when that capability is enabled, and you approve it.

The setting is available under Settings, then General, then Agent, then Execution on Local Computer.

Set local execution to never allowed unless a Bot has a specific reason to work on your local files. This does not restrict the cloud computer in any way.

Grok Bot vs. Hermes Agent

Both give an agent a persistent computer. That is where the similarity stops.

Hermes is open source and self-hosted. You bring the machine, whether that is a Mac Mini at home or a virtual private server, and you run the agent on it. Grok Bot is a managed product where the machine is provisioned for you, and you never see it.

The difference shows up most sharply in memory and skills.

• Hermes exposes both as files you can read and edit. Identity lives in SOUL.md, facts live in MEMORY.md and USER.md with hard character limits, sessions are searchable in SQLite, and skills are Markdown files with structured frontmatter that the agent writes and revises itself.
• Grok Bot exposes none of that. Memory is described by behaviour rather than by format, and a skill is something you inspect through the interface rather than a file you open.

Hermes also ships maintenance machinery that Grok Bot has no equivalent for. The Curator prunes and archives unused agent-written skills in the background, and the companion GEPA pipeline evolves skills offline against execution traces rather than asking the agent to grade itself.

Grok Bot’s advantage runs the other direction. Multi-agent coordination is a first-class product surface rather than something you assemble. Group chats, asynchronous handoffs, ownership passing, and a shared browser session across the roster all work without configuration.

The choice is mostly about who absorbs the operations. Hermes gives you inspectable files and real isolation, and asks you to run and maintain the machine. Grok Bot removes the maintenance and takes the isolation away with it.

The broader landscape follows the same axis. ChatGPT Work and Claude’s computer use both operate software, but neither keeps a durable machine with persistent logins in the same way. Devin and Manus give a sandbox per task or project, which is exactly the model Grok Bot moved away from.

Setting it up on macOS

The whole setup runs in a few minutes.

All the instructions are specified here: https://docs.x.ai/grok-bot/get-started.

Essentially, open the Grok Bot access page and choose the matching download.

Open the disk image, drag Grok Bot to Applications, and launch it. Confirm with Open if macOS asks.

Choose Get started on the welcome screen, finish authentication in the browser window that opens, and return to the app. Organizations on single sign-on complete their usual flow here.

Onboarding then introduces Bots, the shared computer, and routines, and asks which tools you use. Answering shapes the first teammate suggestions and does not connect or modify anything by itself.

Your cloud computer provisions in the background while that runs, and the last step opens Meet a future teammate.

Next, pick a suggested teammate or choose Create your own, then give it a short name, one primary job, and a description of how it should work.

A strong first request has five parts

• The outcome you want finished
• The important sources
• The constraints on what it must avoid or ask about
• The deliverable
• And the point where it should stop for you.

Start with something that needs no login at all, like attaching a document and asking for a structured summary with citations gives a result in about five minutes and confirms the machine is healthy.

When a Bot reaches a service that needs authentication, it hands you the computer. Open Agent Computer from the conversation, take control, enter the password or passkey or two-factor code, then return control.

The session persists on the shared computer afterwards. Every other Bot on your account can now use it, which is the moment the shared boundary becomes concrete rather than theoretical.

For supported services, install a connector from Settings and then Plugins instead. It is the more reliable path where one exists.

Finally, once a result is repeatable, ask the Bot to save the method as a named skill. Include the source systems, the output format, and the rules about what requires approval.

Only after that should you create a routine.

As a takeaway, resist automating on the first good output. The documented order exists because a routine inherits every unstated assumption in the task it was built from.

Wrapping up

The design decision underneath Grok Bot is small to state and large in consequence. State moved out of the context window and into a machine that stays running.

That way, a handoff between two Bots costs nothing, and what lets a routine fire at eight in the morning with your laptop shut.

It is also why the documentation tells you not to treat separate Bots as a security boundary, because free handoffs and shared credentials are the same fact seen from two sides.

The practical path through it is the one the docs prescribe and most people skip. One narrow Bot, one real task, corrections until the output is reviewable, then a skill, then a routine, with anything consequential behind approval.

Compared to Hermes, you are trading inspectable files and genuine isolation for a machine you do not have to run. Which side of that trade is right depends entirely on what you are willing to sign into a computer you cannot see.

We will keep writing about Grok Bot as the beta develops, particularly once enterprise controls and any per-Bot isolation land.

👉 Over to you. What is the first role you would hand to a Bot?

Good day!

Here is something that confuses almost everyone who starts serving language models.

You rent a top-end data center GPU. The spec sheet says it can perform close to a thousand trillion arithmetic operations per second. You load a 70 billion parameter model onto it and start generating text. You watch the utilization monitor, and it reads high. Everything looks healthy.

Then you count the tokens. You are getting a few dozen per second from a chip rated for a thousand trillion operations. Almost none of that arithmetic capability is doing anything.

Nothing is broken. Your code is fine, your drivers are fine, and buying a bigger GPU will barely help. What you are seeing is the single most important fact about this hardware, and once you can see it clearly, most performance advice you have read stops being a list of tricks and starts being obvious.

This article builds that intuition from the ground up. We will look at why a GPU is shaped the way it is, why memory and compute are in constant competition, and what actually makes these machines go faster.

No CUDA is required. We will name a few real chips where a concrete number helps, but nothing here depends on memorizing them, and in every case the ratio matters far more than the figure it came from.

👉 Prerequisites. You should be comfortable with the idea of a neural network doing matrix multiplications, and you should know that model weights are numbers stored in memory. That is genuinely all.

Let's begin!

One asymmetry explains almost everything

Start with the single fact the whole design rests on.

Doing arithmetic is cheap. Fetching the numbers to do arithmetic on is expensive.

That statement feels wrong the first time you read it. Multiplication sounds like the hard part and moving a number sounds trivial. On modern hardware the opposite is true, and it is not a small gap.

A useful picture is a workshop. The arithmetic units are workbenches, thousands of them, packed into the middle of the room. The data lives in a warehouse, and the warehouse is connected to the workshop by one corridor.

The benches can consume material far faster than the corridor can deliver it. Add more benches and nothing changes, because the corridor was already the limit. That corridor is what people mean by memory bandwidth.

Every performance technique you will meet later is a way of getting more work out of each trip down that corridor.

Why the gap keeps widening

This is not a temporary engineering shortfall. It is a long-running trend.

Across recent generations of accelerators, arithmetic capability has grown several times faster than memory bandwidth. Each new chip can do far more math, and can fetch only somewhat more data to do it on.

So the imbalance gets worse with every generation, not better. Techniques that reduce data movement become more valuable over time, and raw arithmetic capability tells you less and less about how a chip will actually perform for you.

Why a GPU has thousands of simple units instead of a few clever ones

Now let's look at how the hardware responds to that asymmetry.

A CPU is built around making one sequence of instructions finish quickly. It spends most of its silicon on machinery for that. Large caches so data is already nearby. Prediction hardware so it can guess what comes next. Reordering logic so it can keep working while it waits.

All of that serves a single stream of work, and all of it is expensive in chip area.

A GPU makes the opposite bet, because it was designed for a workload with an unusual property. In graphics, every pixel runs the same short program on different data. In neural networks, every element of a tensor gets the same treatment.

When the program is identical across millions of data elements, you no longer need one controller per element. One controller can steer thousands of arithmetic units, all executing the same instruction on different values.

That is the whole trick. The GPU deletes most of the control machinery a CPU carries and spends the reclaimed area on arithmetic units instead.

The comparison in raw counts is stark. A high-end server CPU advances a few hundred threads at once. A data center GPU advances tens of thousands per clock tick, on a similar power budget.

The catch is that a GPU thread is a much weaker thing. It cannot go its own way, and it is not fast on its own. It is one lane in a very wide, very simple machine.

👉 One term worth learning, because you will see it everywhere. Threads are not scheduled one at a time. The hardware handles them in fixed groups of 32 that move in lockstep and share a single instruction, and a group of 32 is called a warp. That is the whole definition, and it is the unit the chip actually works with.

The one place this leaks

Because a warp shares a single instruction, the 32 threads in it cannot disagree about what to do.

If your code branches on the data, and some threads in a warp take one path while others take the other, the hardware runs both paths in sequence. It switches off the threads that are not on the current path, and they sit idle producing nothing.

The warp takes as long as both branches added together.

This only costs you when the disagreement happens inside a warp. Different warps taking different paths is free. In practice it means data-dependent branching in a tight inner loop is worth avoiding, and it is rarely something you hit when you are calling library operations rather than writing your own.

The GPU does not make waiting shorter. It makes waiting invisible.

We have said fetching data is slow. Both CPUs and GPUs face the same physical delay. They handle it in opposite ways.

A CPU tries to make the wait not happen, using caches and prediction to have the data ready before it is asked for.

A GPU accepts the wait and arranges to always have something else to do during it.

Here is the mechanism. The chip keeps far more work loaded than it can run at once. A single one of the chip's compute units might have sixty-four warps sitting resident, while only one gets to execute in a given tick.

When the running warp asks for data from main memory, it stalls. The scheduler does not wait with it. It picks another warp that is ready and runs that instead. When that one stalls, it picks a third.

The arithmetic units stay busy. Every individual warp spends most of its life waiting, and the machine as a whole never idles.

Switching between warps costs essentially nothing. On a CPU, changing which thread is running means saving one thread's state and restoring another's, which takes hundreds of cycles. On a GPU, every resident warp's state is already parked on the chip in dedicated storage. Switching means pointing at a different set of storage. It takes one tick.

This is why GPUs carry an enormous amount of fast on-chip storage. It is not there to make any one thread faster. It is there so thousands of threads can sit half-finished at the same time, ready to be picked up the instant their data arrives.

👉 This is the reason the utilization number in your monitoring is misleading. It usually reports whether any work was scheduled on the chip, not whether the arithmetic units were doing anything useful. A GPU that is 100% "utilized" while starved for data looks identical to one running at full throughput.

The practical version

If you take one operational idea from this section, take this.

A GPU needs a large pile of independent work to run well. Give it a small job and it will finish that job at roughly the speed of the memory system, with most of the chip idle. Give it a large batch of similar work and the same hardware suddenly looks fast.

This is not a quirk. It is the design working as intended, and it is why batch size shows up in every serving discussion you will ever have.

Memory is a ladder, and each level is far slower than the last

Now for the part that determines almost all of your performance.

Data does not simply live "in memory". It lives at one of several distances from the arithmetic units, and the difference between the nearest and the farthest is enormous.

There are four levels worth knowing.

1. Per-thread storage. The closest and fastest. Each thread holds a handful of values here while it works on them. Access is effectively instant.
2. On-chip scratchpad. A small pool of very fast memory attached to each compute unit, a few hundred kilobytes in size. Crucially, this one is under your control if you are writing low-level code. You decide what goes in it and how long it stays.
3. Shared cache. A larger pool, tens of megabytes, shared by the whole chip. Managed automatically by the hardware. You do not choose what lives here, though access patterns influence it.
4. Main memory. The big pool, tens of gigabytes, where your model weights and activations actually live. This is what people mean by VRAM. It sits physically outside the processing die, and reaching it is by far the slowest thing the chip does.

Two things change as you move down that list. The pools get bigger, and getting to them gets harder.

The gap between the two ends is not a modest one. Reading a value a thread already holds costs you nothing worth measuring. Reaching main memory costs hundreds of times more.

Your weights sit at the bottom level, which is the largest one and the only one big enough to hold them.

Two costs, and the hardware only handles one

Reaching memory costs you in two separate ways, and keeping them apart is what makes the rest of this article make sense.

The first cost is the wait. You ask for data, and some time passes before it arrives.

The hardware deals with this one for you, using exactly the mechanism from the previous section. It keeps thousands of requests moving at the same time, so while one warp waits, plenty of others are being served. The wait is real, and it is almost entirely hidden.

The second cost is the width of the path. Only so many bytes can move per second, and no amount of overlapping requests changes that.

Nothing hides this one. It is a hard ceiling, and it is the reason a chip advertises its memory bandwidth as a headline number.

That is why every calculation in the rest of this article is about bandwidth rather than waiting.

Where these levels actually sit

The ladder tells you how far away each level is. It helps to also know where they physically sit, because the layout explains why some of them are shared and others are not.

A GPU is not one big pool of arithmetic units. It is divided into a hundred or so independent units, and each one is a small self-contained machine. These are called streaming multiprocessors, almost always written as SMs, and it is worth using the real name because every tool and document you meet will use it.

Each SM carries its own copy of the top two levels. Its own register file for the threads living there, its own shared memory and L1 cache, and its own warp schedulers deciding which warp runs next. None of it is shared with the SM next door.

Work arrives as a thread block, which is a batch of threads handed to one SM that stays there until it finishes. Inside the block, the threads are handled as the warps we met earlier.

An example makes the split concrete. Say you launch a thread block of 256 threads, which is a common choice. The hardware divides it into 8 warps of 32, and those 8 warps are what the scheduler picks between from then on.

You never choose that division. You pick the size of the batch, and the grouping into 32s happens underneath you.

This is why batch sizes are almost always multiples of 32. Ask for 250 threads and you still get 8 warps, but the last one runs with only 26 of its 32 lanes doing anything, and the other 6 produce nothing while still taking their turn.

The largest thread block allowed is 1024 threads, which works out to 32 warps sitting on one SM.

This is the reason shared memory is useful, and where its name comes from. Every thread in the block is on the same SM, so they can all read the same shared memory, and cooperating with each other costs almost nothing.

Below all of the SMs sits the L2 cache, which every SM can reach. Below that, off the die entirely and across the memory bus, sits HBM holding your weights, your KV cache, and your activations. HBM is what people mean when they say global memory or VRAM.

The picture explains the trade-off you keep running into. Anything you can keep inside one SM is fast and private. Anything that has to be seen by the whole chip has to travel down to L2 or further, and the trip down is the expensive part.

That also gives the four levels their real names, which is what you will see everywhere else. Per-thread storage is the register file. The scratchpad is shared memory, sitting alongside L1. The shared cache is L2. Main memory is HBM, and it is the same thing people call global memory or VRAM.

The rest of this article uses those names.

The counterintuitive part

On a CPU, storage gets larger as it gets slower, in a smooth pyramid. Registers are tiny, caches are bigger, main memory is huge.

On a GPU the shape is distorted. The register files across the whole chip add up to roughly the same size as L2. That is unusual, and it follows directly from the previous section. Thousands of half-finished threads need somewhere to park their state, so the chip devotes an unusual amount of area to holding it.

In summary, the memory system is not one thing. It is a ladder, and performance work is almost entirely about keeping data on the upper levels for as long as possible.

Work per byte is the number that decides everything

We now have enough to state the central idea precisely.

Take any operation you run on a GPU. Count two things. How many arithmetic operations it performs, and how many bytes it has to pull from main memory to perform them.

Divide the first by the second. That ratio is the operation's work per byte, and it predicts your performance before you measure anything.

The formal name is arithmetic intensity, and you will see it used that way. The idea is simply how much value you extract from each trip down the corridor.

Two operations at opposite ends

The useful question is not how many bytes an operation reads. It is how many times it uses each value it reads.

Multiplying every element of an array by two uses each value exactly once. Fetch it, double it, write it back, never look at it again. One operation per value fetched.

Multiplying two matrices is the opposite. In a 1024 by 1024 product, every value you fetch from the first matrix gets multiplied against 1024 different values from the second. One fetch, a thousand operations.

That is a thousandfold difference in what a single fetch buys you, and it comes from reuse alone. Matrix multiplication is the one operation accelerators genuinely excel at, and that is why.

The break-even ratio

Every chip has a threshold, and it comes from one division.

Take the chip's peak arithmetic rate and divide it by its peak memory bandwidth. That gives you the work-per-byte ratio at which the two exactly balance.

For a current data center GPU running 16-bit precision, that number is around 300 operations per byte.

Take the H100 SXM5, which is the workhorse most people are renting. Its datasheet gives 989 TFLOPS of dense BF16 throughput and 3.35 TB/s of memory bandwidth.

Divide one by the other. 989 trillion operations per second over 3.35 trillion bytes per second comes to 295 operations per byte, which is where the 300 comes from.

That single number tells you a lot before you profile anything. An operation that does fewer than 295 operations per fetched byte cannot use the full chip, no matter how well it is written.

The threshold also moves with the hardware, and not in the direction you might expect. The H200 uses the same compute die as the H100, so its arithmetic ceiling is unchanged at 989 TFLOPS, but its bandwidth rises to 4.8 TB/s. That drops the threshold to 206 operations per byte.

A lower threshold is a good thing. It means more of your workloads clear the bar and become compute-bound, which is why a bandwidth upgrade with no extra arithmetic still speeds up inference.

Below 300, you are limited by memory. Adding arithmetic capability changes nothing, because the arithmetic units are already waiting.

Above 300, you are limited by arithmetic. Adding bandwidth changes nothing, because the data is already arriving faster than you can consume it.

The threshold belongs to the hardware. Where your workload sits relative to it belongs to you.

👉 The name for this picture is the roofline model. It is worth knowing the term, because it is the standard way performance engineers frame these conversations, and it is exactly the two-ceiling idea described above.

Why generating a token is the worst case imaginable

Apply that framework to language model inference and the opening puzzle resolves itself.

Generating text happens one token at a time. Each new token requires a full forward pass through the model. That pass reads every weight in the model, once.

Start with what a single weight actually does.

A weight sits between two numbers. It gets multiplied by one incoming value, and the result gets added into a running total that becomes part of the output.

That is the entire job of a weight. One multiply, one add, so two operations every time it is used.

Now scale that up. A 70 billion parameter model does about 140 billion operations to produce one token, because every weight does its two.

Next count the bytes. At 16-bit precision each weight takes 2 bytes, so reading all of them means moving 140 GB.

Put the two together. 140 billion operations funded by 140 GB of traffic, which is 1 operation per byte.

The break-even threshold was around 300. Serving a single request puts you roughly three hundred times below it.

That is why the arithmetic units sit idle. There is nothing wrong with your setup. The operation itself simply does not contain enough work to keep them fed.

The number this actually predicts

Once you accept that generation is limited by memory, your token rate stops being mysterious and becomes a calculation you can do yourself in a few seconds.

A 70 billion parameter model in 16-bit precision occupies about 140 gigabytes. A high-end GPU delivers roughly 3.3 terabytes per second from main memory.

That gives you two numbers and one division. 140 gigabytes of weights, and 3,300 gigabytes arriving every second, which works out to 0.042 seconds to read the weights once.

Call it 42 milliseconds. That is the time to produce one token, because at batch size 1 one full read of the weights is exactly what one token costs.

Now flip it into a rate. If each token takes 42 milliseconds, then one second holds 1,000 divided by 42 of them, which is about 24.

So roughly 24 tokens per second, and no amount of software cleverness moves that floor.

The floor is set by how many bytes you must read per token, divided by how fast bytes arrive. Everything else is detail.

Prefill behaves in the opposite way

There is a second phase, and it lands on the other side of the line.

When you send a prompt, the model processes all of its tokens at once before generating anything. That phase is called prefill, and because it handles many tokens together, each weight fetched gets used across all of them.

Work per byte climbs immediately. Prefill is usually compute-bound, which is why long prompts feel expensive in a different way than long outputs.

The same model has two phases with opposite bottlenecks. A lot of confusion in performance discussions comes from treating them as a single thing.

Every optimization you have heard of moves the same ratio

Here is where the framework pays off. Once you see performance as a single ratio, the techniques stop looking like a list of unrelated tricks.

There are only two things you can change. Increase the work done per fetch, or decrease the bytes fetched. Every technique below does one or the other.

Batching increases the work

Process several requests at the same time and you read each weight once, then use it for all of them.

Ten concurrent requests means ten times the work per byte, at no extra memory cost. Nothing else you can do for serving throughput comes close, and the only thing it costs you is a little latency for each individual user.

The break-even math is worth remembering. At 16-bit precision, you need roughly 300 concurrent sequences before generation becomes compute-bound. Below that, you have spare arithmetic capability going unused, which is exactly why serving systems work so hard to keep batches full.

Fusion decreases the bytes

Chained elementwise operations are the clearest case.

Run three operations separately and each one reads its input from main memory and writes its output back. That is six trips down the corridor for arithmetic that takes almost no time.

Fuse them into one operation and the intermediate values never leave the chip. One read, one write, identical arithmetic. You have cut memory traffic by two thirds.

This is why compilers that fuse operations produce large speedups on models full of small elementwise work, and why those same compilers do nothing for a pure matrix multiplication.

Keeping data close decreases the bytes

Shared memory exists precisely for this.

Load a chunk of data into shared memory, do every calculation that needs it, then move on. You paid for the trip once and spent it many times.

FlashAttention is the famous example. Ordinary attention builds a large intermediate matrix, writes it to main memory, then reads it back. FlashAttention computes attention in tiles that stay in on-chip memory, so that intermediate never touches main memory at all.

The arithmetic is essentially unchanged. The memory traffic collapses, and so does the runtime.

Quantization decreases the bytes directly

Store weights in 8 bits instead of 16 and every weight is half the size.

That halves the bytes you must read per token, which doubles the work per byte and roughly doubles your generation ceiling. The same 70 billion parameter model drops from 140 gigabytes to 70, and the theoretical floor moves from about 24 tokens per second to about 48.

The trade-off, however, is accuracy. Lower precision loses information, and how much that costs depends heavily on the model and the quantization method. This is a real engineering decision, not something you get for free.

Reading in order matters more than it should

One access-pattern detail is worth knowing even if you never write a kernel.

Memory does not move one value at a time. The hardware fetches fixed-size blocks. When the threads in a warp read neighboring addresses, their requests fall inside the same few blocks, every fetched byte gets used, and you get the bandwidth you paid for.

When they read scattered addresses, each thread pulls in a whole block to use a few bytes of it. You can end up fetching eight times the data you actually needed.

The practical consequence is that memory layout is a performance decision. Reading a matrix along the wrong axis, or using a data structure that scatters related values, can cost you most of your bandwidth with no change to the arithmetic.

Small jobs have a floor of their own

There is one bottleneck the ratio does not capture.

Sending work to a GPU has a fixed cost. The CPU has to prepare and dispatch each job, and that takes time whether the job is large or tiny.

If your model runs many small operations, you can end up spending more time dispatching than computing. You are neither memory-bound nor compute-bound at that point. You are overhead-bound.

The symptom is a GPU that looks idle while your CPU works hard. The fixes are fewer and larger operations, or capturing a whole sequence of operations so it can be replayed as one unit.

How to tell which situation you are in

The framework is only useful if you can locate yourself on it, and there is a simple way to do that.

Measure two things during a run. How many bytes per second you are actually moving from main memory, and how many arithmetic operations per second you are actually performing. Compare each against what the hardware can do.

Three outcomes cover nearly everything.

• Bandwidth near its ceiling, arithmetic far below. You are memory-bound. Reduce bytes moved. Batch harder, quantize, fuse, fix layouts. Adding arithmetic capability will do nothing.
• Arithmetic near its ceiling, bandwidth far below. You are compute-bound. This is a good place to be. Improvements now come from lower precision, better algorithms, or more capable hardware.
• Both far below their ceilings. You are overhead-bound or your work is too small to fill the chip. Look at how many separate operations you are launching and how large each one is.

👉 Note on where to look first. For language model serving, generation is memory-bound in essentially every realistic configuration. If you are optimizing token throughput and have not yet examined batch size, precision, and KV cache size, those three will almost always dominate anything else you try.

What stays true when the hardware changes

Specifications move quickly. It is worth knowing which parts of this you will have to relearn and which parts you will not.

What changes are the numbers. Memory capacity, bandwidth, and arithmetic throughput all grow with each generation, with arithmetic growing fastest of the three. New precision formats keep appearing, each smaller than the last, because shrinking bytes is the most direct attack on the bottleneck.

What does not change is the shape. Arithmetic remains far cheaper than data movement. The memory ladder keeps its levels. Work per byte keeps deciding which ceiling you hit.

If anything, the framework gets more useful over time. Because arithmetic capability grows faster than bandwidth, the break-even ratio keeps rising. Workloads that were compute-bound on older hardware become memory-bound on newer hardware without a line of code changing.

That is the direction to bet on. Techniques that reduce data movement will keep gaining value, and peak arithmetic figures on a spec sheet will keep being the least predictive number in the document.

Conclusion

In this article, we built an intuition for how a GPU works from a single asymmetry, which is that arithmetic is cheap and moving data is expensive.

We saw why that asymmetry produced a chip full of thousands of simple arithmetic units sharing one controller, rather than a few sophisticated ones. We saw that the GPU does not shorten the wait for data, it hides the wait by keeping far more work loaded than it can run, and swapping instantly between pieces.

We walked the memory ladder, from the register file through shared memory and L2 down to HBM, where each level is bigger than the last and harder to reach. We also saw where those levels sit, with the top two owned privately by each SM and everything below either shared across the die or off it entirely.

That gave us the central number, which is work performed per byte fetched. Every chip has a break-even ratio, around 300 operations per byte on current hardware at 16-bit precision. Below it you are limited by memory, and above it you are limited by arithmetic.

Applying that to language models explained the opening puzzle. Generating a token reads every weight to perform about two operations on each, giving a ratio near one, roughly three hundred times below break-even. Prefill sits on the opposite side of the line, which is why the two phases behave so differently.

Finally, we saw that every well-known optimization is one of two moves. Batching and tiling increase the work done per fetch. Fusion, quantization, and better memory layouts decrease the bytes fetched.

We deliberately left out the low-level details of writing GPU code, because the intuition has to come first. Kernel writing before this framework tends to produce code that is fast for reasons the author cannot explain, and slow for reasons they cannot find.

Upcoming pieces will build on this and get more concrete, including how to read profiler output against the break-even line, how the KV cache changes the memory picture as context grows, and what actually happens inside a serving engine when it decides how to batch.

The aim, as ever, is to develop a solid system-level perspective and to equip you with an adaptable engineering framework for building AI systems that are robust and maintainable.

As always, thanks for reading!

Recap

In Part 13, we moved the retrieval work earlier.

Instead of waiting for a query and then reading the corpus, we read it once, kept the key-value cache, and answered from that cache afterwards.

Preloading Knowledge Into a Model Instead of Retrieving It (Part A)

Part 13: Reading the corpus once before any query arrives, and the two limitations that decide whether it works in production, with implementations

Daily Dose of Data ScienceAvi Chawla

We placed the idea on a spectrum, from ordinary retrieval through to a trained cache artifact, with each step moving more work offline and keeping a different artifact.

We then built the system, prefilling a corpus once and restoring it per query. The cache memory formula predicted the artifact to within 0.09 percent on a real model, and all eight answers were identical whether the corpus was prefilled fresh or restored from disk.

We also saw that preloading has many more benefits than just speed. A retriever answered a multi-hop question with the wrong name until it was fetching 40 percent of the corpus, because the section holding the second fact shared no vocabulary with the question.

We also understood two limitations from that part.

The advertised context window is not the ceiling, since a fact written in the question's own words was found at every length we tried while the same fact reworded started failing at 512 tokens.

And renting the cache from a provider turns preloading into a break-even calculation, where holding time sets the bar and idle entries still bill.

In this chapter, we will look at what happens when you try to make that stored cache smaller.

A preloaded cache is large. The 636-token handbook from Part 13 produced a 15.6 MB artifact, which scales to roughly 2.46 GB for a 100,000-token corpus on the same small model. Shrinking it before storage seems like the obvious next move, and there is a substantial research literature on doing exactly that.

We'll start with the two families in the literature, one that shrinks every stored number and one that removes tokens entirely, and understand why only one of them can run before a query exists.

We'll then examine the methods that dominate the second family, namely SnapKV, PyramidKV, and H2O, and see that all three decide what to keep using the query itself.

From there, we'll look at what happens when these methods are evaluated in the setting preloading actually operates in, and what a genuinely query-agnostic method has to look like instead.

Finally, we'll ask how far compression can be pushed at all, and find a ceiling that two completely different approaches run into from opposite directions.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

Compressed preloading and the query-agnostic requirement

A preloaded cache is large.

Shrinking it before storage seems like an obvious next move, and there is a substantial research literature on doing exactly that.

Most of that literature cannot be used for preloading and understanding why that's the case is an important thing, because the reason is structural rather than a matter of tuning.

How cache compression usually works

Two broad families of approaches exist, and they behave differently under preloading.

The first shrinks each stored number.

More specifically, keys and values are held at reduced precision, so a cache kept at 8 bits instead of 16 occupies half the space.

Nothing is discarded, i.e., every token remains present, and the loss is arithmetic rounding spread evenly across the cache.

These types of approaches are naturally query-agnostic, since rounding a number does not require knowing at info about the query that may be asked.

The second family removes tokens entirely, keeping the cache entries classified as important and discarding the rest.

This reaches far higher compression ratios than precision reduction, because dropping a token removes all of its keys and values across every layer.

It also raises a question that the first family never faces, which is how importance gets computed in the first place.

Precision reduction is bounded by the format itself. Caches are normally held in float16, so dropping to 8-bit integers is a factor of two, and aggressive 4-bit schemes reach a factor of four before rounding error becomes visible in the outputs.

Since a number only has so many bits to give up, there's a ceiling on how far we can go before we start losing accuracy

Token eviction has no equivalent ceiling. Keeping one token in four is a factor of four, keeping one in ten is a factor of ten, and nothing in the method stops you going further.

This does not remove precision but rather evidence, so the quality question becomes whether the discarded tokens were ones some future tokens may have needed.

That question is where preloading runs into trouble.

The prominent methods that answer this do it usually in the same way.

SnapKV starts from an observation about how attention behaves.

Take the last few dozen tokens of the prompt, which the authors call the observation window, and look at where those positions send their attention.

Each attention head turns out to focus on a fairly consistent small set of earlier tokens, and it keeps focusing on that same set as generation proceeds.

That consistency helps make the window usable as a predictor.

Every position inside the window carries an attention weight over every earlier token, so adding those weights down the window gives each earlier token a score, which is how much the window collectively cared about it. Rank the prefix by that score, keep the top slice, and evict everything else.

Two details are important to decide to see if this works well.

• The scoring and selection happen separately for each attention head rather than once for a whole layer, because different heads attend to different things and one shared choice would serve none of them properly.
• The winning positions are also pooled into contiguous clusters rather than kept as scattered individual tokens, which stops the method from shredding a passage into disconnected fragments that no longer read as a span.

PyramidKV keeps that scoring machinery and changes how the budget is spread across depth.

Its starting observation is that attention is scattered widely across the input in the lower layers, consolidates as you move up the stack, and ends up concentrated on a few critical tokens in the highest layers.

The authors call this pyramidal information funneling.

If that is how attention behaves, then a uniform per-layer budget is wrong at both ends.

A high layer attending to a handful of tokens does not need a large allowance.

A low layer spreading its attention broadly is ideally where a small allowance throws away information that was genuinely in use.

PyramidKV therefore allocates more cache to the lower layers and progressively less as it climbs, which is the pyramid the name refers to.

H2O's (short for Heavy-Hitter Oracle) starting observation is that attention mass is distributed very unevenly.

In traditional ML inference, a batch is usually represented as a matrix.

If there are inputs of varying lengths, then each input is padded or truncated to the same length, stacked into one tensor, and processed through a single forward pass to generate one prediction per row.

Every row costs the same, every row finishes at the same moment, and the full shape of the work is known before the pass starts. Batching there is a tensor-packing problem.

LLM decoding does not work on those principles.

Under the hood, one forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count beforehand until the model emits a stop token.

Padding can't fix it, because the mismatch isn't in the input width but rather in how long each request occupies the GPU.

So a batch fixed at the start runs at the pace of its slowest member.

Continuous batching is an approach implemented in most serving engines to avoid this problem. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default.

The core idea is to decide batch membership at every forward pass instead of once when the batch starts. A finished request leaves at the next pass and a waiting request takes its slot right there, so no slot sits reserved for work that's already done.

The scheduler underneath continuous batching is at the core of handling this process end-to-end, i.e., the loop that decides at every forward pass which requests get tokens and how many.

So let's build it up from the problem, walk one scheduling step in order, and look at the measured gains once the mechanism is clear.

Scheduler in traditional ML inference

A classifier takes a fixed-width input and returns a label.

Sequence models pad to a maximum length and mask the padding. Either way, the tensor going in has a known shape and the tensor coming out has a matching row count.

The reason to batch at all is that loading model weights out of HBM is a fixed cost per forward pass.

So if you stack 64 rows/inputs into a single pass and all those 64 rows share one weight read, the throughput rises:

Serving that in practice is straightforward since you can simply fill a batch, run it, return every result, and start the next one.

No request carries state into the following pass, and no request outlives the batch it arrived in.

Nature of LLM inference

The decoding process in LLMs breaks three assumptions discussed above:

• Each pass generates a single token per sequence.
• The KV cache carries state forward into the next pass.
• The number of passes a request needs is only known when it produces a stop token.

So a request that finishes in 30 tokens sits in the same batch as one running to 400, and under a fixed batch it holds its slot for all 400 steps while producing nothing.

The GPU keeps paying the full weight read for a mostly empty batch.

The waste scales directly with how much the output lengths vary, which in real traffic can be a lot!

The solution

To solve this, we need to alter where the serving system and the execution engine talk to each other.

Essentially, instead of handing over a batch and waiting for it until all requests complete, the scheduler runs one iteration, gets control back, and decides again.

This is called iteration-level scheduling, and it describes the mechanism continuous batching uses under the hood.

A finished sequence leaves at the next iteration boundary rather than at the end of the full batch run. And a waiting request enters at that same boundary. The batch is rebuilt every forward pass.

This, in itself, is not sufficient since rebuilding the batch every pass creates a problem of its own.

For instance, a request that's prefilling 4,096 tokens together and a request decoding its 900th token don't share a tensor shape, so there's nothing to stack in the usual way.

Selective batching handles this, and it ensures that batching is applied only to the operations that can take it.

Every token scheduled in the step, whatever request it came from, is flattened into one long sequence of shape (total tokens * hidden size).

Layer norm, the QKV projections, and the feed-forward blocks all operate on each token independently, so they neither know nor care which request a token belongs to.

They run once across the whole flat stream at full efficiency.

But the attention operation can not work like this. A token may only attend to earlier tokens from its own request, and every request has a KV cache of a different length.

So the flat tensor is split at the attention boundary, attention runs separately for each request against that request's own cache, and the outputs merge back into the stream before the next batched operation.

One scheduling step, clearly explained

Between two forward passes the scheduler answers one question. Which requests run next, and how many tokens does each one get.

It answers that in four steps and we'll use vLLM's V1 scheduler as the reference implementation:

1. The scheduler fixes a budget for the step:

max_num_batched_tokens caps the total tokens the step may issue, and max_num_seqs caps how many sequences can be in flight at once.

2. Running requests get to claim compute budget first:

Each request carries a count of tokens already computed and a target count. The scheduler hands out tokens to reduce the gap between where a request currently is and where it wants to get to, spending from the budget as it goes.

A scheduler could reasonably keep two code paths here, one for prefill and one for decode, since prefill processes a whole prompt and decode produces a single token.

But in practice, there's no such split. Prefill and decode look like different jobs, but the scheduler sees the same number.

It takes the target, subtracts what's already computed, and hands out that many tokens if the budget allows.

So for a fresh 4,096 token prompt, that number is 4,096. For a request mid-generation, it's 1. It's the same line of code either way.

# simplified from vllm/v1/core/sched/scheduler.py

tokens_this_step = min(
    request.num_tokens_with_spec - request.num_computed_tokens,
    remaining_token_budget,
)

Chunked prefill and prefix caching need no extra handling either.

If a 4,096 token prompt doesn't fit in the budget, the scheduler hands out 2,048 now and the other 2,048 in the next step.

If 3,000 of those 4,096 tokens were already cached from an earlier request, only 1,096 are left to hand out.

3. The scheduler claims KV blocks for the tokens it just assigned.

It reserves that memory while deciding, not after.

A step is only valid if the cache can hold what it produces, so when there aren't enough free blocks, the scheduler takes them from the newest request in the running list.

4. Whatever budget is left goes to waiting requests.

Requests that haven't started yet get scheduled with whatever tokens are left over. If the running requests used the whole budget, nothing new starts this step.

The scheduler then hands the GPU a list of how many tokens each request runs and which KV blocks to write them into.

Token budget decides latency and throughput

A step takes as long as the tokens inside it, which makes max_num_batched_tokens a latency setting as much as a throughput one.

To recall, max_num_batched_tokens is the total number of tokens the scheduler may put into one forward pass, added up across every request in the step. It isn't a per request limit. A decoding request contributes one token, since one token is all it produces. A prefill contributes however many prompt tokens the scheduler hands it.

At around 2048 tokens, no step runs long, so inter-token latency stays tight and the GPU is often under-fed.

At around 16384, the GPU stays saturated and throughput rises, but every request in the batch waits out a longer step.

Size isn't the only thing this budget setting controls.

With chunked prefill on, which is the V1 default wherever possible, the scheduler fills the budget with pending decodes first and gives what's left to prefills, chunking a prefill that doesn't fit.

Preemption

When block allocation fails, the scheduler frees a running request's blocks, marks it preempted, resets num_computed_tokens to zero, clears its speculative tokens, and puts it back at the front of the waiting queue.

The reset to zero is an actual cost because, let's say a request had already prefilled 3,900 tokens, and it then comes back with none of that saved, it must compute all 3,900 again.

V1 made recompute the default preemption mode and dropped the older swap path entirely.

From the outside, this looks like the GPU running out of headroom. Latency climbs as traffic rises, and adding replicas seems like the fix.

What's actually happening is the same prefill being computed two or three times. Requests get preempted, readmitted, and preempted again, and every one of those cycles repeats work the GPU had already finished.

vLLM counts this in Prometheus as total_cumulative_preemption_cnt, and it's the first metric to check when p99 climbs without a traffic change.

For context, p99 is the latency your slowest 1% of requests see. Averages hide preemption because most requests are never preempted, so problems show up in the tail.

Three settings improve it:

• Raise gpu_memory_utilization so more VRAM becomes KV cache.
• Lower max_num_seqs so fewer sequences compete for it.
• Raise tensor_parallel_size so weights shard across GPUs and leave more room per device.

Wrapping up

Everything above comes down to one loop.

The scheduler rebuilds the batch at every forward pass, hands out tokens against a fixed budget, and reserves KV blocks while it decides, and those three moves determine what your GPU actually produces.

Anyscale's benchmark on OPT-13B found that as output lengths varied, static batching fell to around 81 tokens/s while vLLM reached 23x the throughput of naive Hugging Face serving on the same A100. None of that came from a faster model:

So if throughput or p99 is your problem, the scheduler is the layer to look at before the model.

The preemption counter is usually the cheapest place to start, since it tells you in one number whether your KV pool matches the concurrency you configured.

Recap

In Part 12, we found that RAG latency does not live where most people optimize it.

Why RAG Latency Is a Prefill Problem, Not a Retrieval Problem

Part 12: Why prefill dominates RAG latency, and how to reuse KV caches that share no prefix, with implementations

Daily Dose of Data ScienceAvi Chawla

Retrieval takes tens of milliseconds. The model then reads the retrieved chunks, and that reading step can take seconds.

We separated the two phases of inference and saw why.

• Prefill processes every input token in parallel and is compute-bound, producing the key-value cache.
• Decode then generates one token at a time, reusing that cache, and is memory-bandwidth-bound.

We then examined prefix caching, which is considered the most standard fix.

Serving engines split the cache into fixed-size blocks and hash each block together with its preceding blocks. That rule guarantees any reused block is bit-identical to a freshly computed one. It also means two requests retrieving the same documents in a different order share nothing at all.

Finally, we took apart the obvious repair, which is precomputing each chunk's cache once and stitching the pieces together.

It fails for three separate reasons. Position is encoded into the key vectors and must be corrected by rotation. Cross-attention is missing entirely, because a chunk prefilled alone never saw the chunks before it. Attention sinks appear at the start of every chunk instead of once at the start of the input.

We built all of this from scratch and watched it appear in real tensors. Layer 0 matched the joint prefill exactly, while deeper layers diverged, and 12 percent of tokens carried 67.3 percent of the total error.

That work assumed retrieval happens, then asked how to make reading cheaper.

In this chapter, we will explore what happens when we move the work earlier still. We'll understand how a knowledge base can be processed once, before any query arrives, and stored as a reusable artifact.

This is the first of three parts on preloading. Here we cover the idea itself, along with the two things that bound it, which are how much context a model can actually use and what a cache costs when you rent it from a provider.

Part 14 takes up compression, and why most published methods cannot run before a query exists.

Part 15 covers the two approaches that require training, and what running any of this in production involves.

As always, every notion will be explained through clear examples and walkthroughs, and each part ends with runnable experiments on a real model.

Let's begin!

The focus of this part

Part 12 optimized the reuse of work already done. A request arrived, the model read some chunks and processed them, and we tried to avoid processing them again.

There is an earlier point to intervene at this stage.

In several cases, a knowledge base often does not change between queries.

For instance, a product manual, a legal contract, a codebase, or an internal policy document is the same text for every user who asks about it.

Nothing forces the pipeline design to wait for a query before processing that text.

If the model reads the corpus once, before any user shows up, the resulting cache can be stored and reused. Every query then skips straight to decoding. Retrieval, chunking, and embedding disappear from the request path entirely.

This idea has a name in the literature, cache-augmented generation, and we will get to it shortly.

But treating it as a single technique misses what is actually going on. Preloading involves a family of approaches that differ in one respect, which is how much processing you do offline and what you keep afterwards.

The Mooncake team, who built the serving platform behind the Kimi chatbot, titled their FAST 2025 paper around exactly this trade. They describe it as trading more storage for less computation.

The spectrum

Before going into any single approach, it helps to see the whole range laid out. Each step down this list moves more work offline and keeps a different kind of artifact.

At the top, we have ordinary retrieval, where nothing is precomputed and every query costs a full prefill.

Below that, we have prefix caching, which keeps whatever happened to be computed for an earlier request. We covered this in Part 12 and saw its limitations.

Next comes naive preloading, where the entire corpus is prefilled once and its raw cache is stored. This is where cache-augmented generation belongs.

Then modular preloading, where each passage is prefilled separately so it can be assembled in any order. This requires either accepting the cross-attention loss from Part 12 or changing the model so the loss no longer matters.

Then compressed preloading, where the stored cache is shrunk before it is saved. This turns out to have a requirement that rules out most published compression methods, and that requirement is the most interesting finding in this chapter.

At the bottom is trained preloading, where the corpus is distilled into a small cache through an actual training run.

With the map in place, let's work through it from the top.

Naive preloading

The simplest version of the idea simply takes the entire knowledge source, run one prefill pass over it, and keep the resulting cache.

The approach was formalized in the following paper:

The authors describe preloading all relevant resources into the model's context and caching the runtime parameters, so that inference can proceed directly from the preloaded cache without retrieval.

The three phases

The lifecycle has three steps, and each one maps onto the details we discussed in Part 12.

Step 1: Preload

The corpus is formatted as a single input and passed through the model once. Every layer produces keys and values for every token, exactly as in any prefill.

The cost here is a normal prefill cost, and because attention is quadratic in sequence length, it is not small. The benefit is that it is paid once rather than per query since all queries use the same cache generated from prefill.

Step 2: Persist

The resulting cache is serialized and written to storage. The artifact is a set of tensors, one key tensor and one value tensor per layer, and its size follows directly from the cache memory formula we derived in Part 12.

Let's revisit the formula we discussed in the previous part because every storage decision in this chapter rests on it. It tells you how many bytes a preloaded corpus becomes.

Let's consider the symbols one at a time:

• $M$ is the size of the stored cache in bytes.
• The leading 2 counts the key tensor and the value tensor, since every cached token needs both.
• $L$ is the number of transformer layers, because each layer keeps its own cache.
• $H$ is the number of key-value heads and $d_{h}$ is the dimension of each head, so their product is how many numbers one token contributes at one layer.
• $T$ is the number of tokens in the corpus.
• Finally, $b$ is the bytes used per number, which is 4 for float32 and 2 for float16.

Looking at the structure, everything except $T$ is fixed the moment you choose a model.

All those terms collapse into a single per-token constant, and the total is that constant multiplied by corpus length. The formula is linear in how much text you preload, with no dependence on what the text says.

The intuition is that a preloaded cache charges a fixed cost per token of corpus.

Doubling the corpus doubles the artifact. Halving the precision halves it. A page of dense legal text and a page of filler will cost exactly the same, because only the token count matters.

There are two practical consequences of this formulation:

• The first is that the artifact can be sized before any code is written, by reading $L$, $H$, and $d_{h}$ off the model's config file.
• The second is that models using grouped-query attention, where many query heads share a smaller set of key-value heads, produce far smaller caches, because $H$ counts key-value heads rather than query heads.

We will put real numbers through this formula in the hands-on section, where a 512-token corpus produces an artifact the formula predicts to within 0.2 percent of its actual size on disk.

Step 3: Infer

When a query arrives, the stored cache is loaded back into memory, the query tokens are appended, and decoding proceeds. The corpus is never read again.

How preload can improve answers

Speed is the most obvious benefit when you preload the entire context. But the increase in answer quality is that's evident so let's understand this now.

A retrieval pipeline can fail by fetching the wrong passage.

That failure mode does not exist when the whole corpus is present in the context, because there is no selection step to get wrong.

The CAG authors position this directly against retrieval-based methods, noting that precomputed caching approaches which still rely on retrieval remain vulnerable to the retrieval failures inherent to those systems.

There is another effect that connects to the Part 12 of this course.

Chunking a document destroys the relationships between its parts.

But a preloaded corpus was processed in a single joint prefill step, so every token attended to every earlier token normally.

None of the cross-attention loss we measured in Part 12 applies here, because nothing was ever prefilled in isolation.

That makes preloading particularly suited to questions whose answers span distant parts of a document, which is exactly where chunk-based retrieval struggles.

The trade-off, however, is that all of this holds only while the corpus fits comfortably inside the context window.

Cost economics of preloading

At this stage, let's see how many queries a preloaded corpus needs to see before the offline work is worth doing.

A 70B parameter model in FP16 requires 140GB of memory for weights alone.

That exceeds any single GPU available today:

• An H100 carries 80GB
• An RTX 4090 carries 24GB

So serving the model in half precision means multi-GPU setups and tensor parallelism before the first token is generated.

Quantization reduces this footprint by storing weights in lower bits.

The standard scheme maps each weight from FP16 onto a small integer grid: compute the dynamic range of the tensor, divide it into 2^b levels (16 levels at 4-bit), round each weight to the nearest level, and store a scale factor to dequantize during inference.

At 4-bit, the weight memory drops 4x. The 140GB model fits in 35GB, within the range of a single 40GB or 48GB card.

That said, if rounding were lossless in practice, one method would suffice, and the diagram below would have had one row:

But it has five because weights and activations in large transformers are not uniformly important.

The LLM.int8() paper quantified this. Beyond roughly 6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20x to 100x larger than the rest.

These dimensions make up about 0.1% of the model’s features, but zeroing them out nearly destroys the model’s ability to predict text.

These outliers are also what break naive rounding. The quantization grid spans the range of the tensor, so a single activation of magnitude 60 among values near 1 inflates the scale factor and collapses the remaining values into a few levels, destroying their precision.

Each method in the visual shown earlier solves this saliency problem differently.

1) RTN (round to nearest) does not handle it at all. It applies the scale-and-round procedure directly with no calibration data, which makes it the cheapest baseline and the weakest at low bit widths, where accumulated rounding error is unrecoverable.

2) GPTQ corrects rounding damage as it happens. It quantizes the weights of a layer a few at a time, measures how much error the rounding just introduced, and nudges the not-yet-quantized weights to cancel out that error before moving on.

The adjustment is not uniform. GPTQ uses statistics from calibration data to work out which remaining weights can absorb the error with the least effect on the layer’s output, which is what separates it from blind rounding.

This runs fast enough to matter in practice. A 175B model quantizes to 4-bit in around four GPU hours. The known weakness is that fitting the calibration data this closely can tune the model to it, and the AWQ authors showed this hurts accuracy on inputs that look different from the calibration set.

3) AWQ protects the important weights before rounding instead of repairing damage afterward. It runs calibration samples through the model and watches which weights get multiplied by the largest incoming values, because a rounding error on those weights gets amplified the most in the layer’s output.

Only around 1% of weight channels turn out to matter this much. Rather than storing them at higher precision, AWQ multiplies them by a scaling factor before rounding, so they spread across more levels of the grid and lose less detail, and the math is rearranged so a matching inverse scale elsewhere keeps the layer’s output unchanged.

Every weight still ends up in plain INT4, with no special-case handling at inference time. AWQ also depends far less on which calibration samples were used than GPTQ does, which is a big part of why it became the standard choice in serving engines like vLLM.

4) LLM.int8() isolates the outliers instead of fighting them. When the model is loaded, it finds the dimensions carrying extreme values and splits every matrix multiplication into two:

• The outliers run in full FP16
• The other 99.9% of values run in INT8, and the two results are added back together.

Nothing needs to be computed ahead of time. That is why it is also available as the load_in_8bit flag in bitsandbytes.

The cost is speed. Splitting and merging every matrix multiplication is slower than running one optimized 4-bit kernel, so this path is common for local development and QLoRA fine-tuning, not for serving traffic at scale.

5) QAT modifies training rather than the quantization procedure. During a short fine-tuning run, every forward pass rounds the weights to INT4 before using them, so the model experiences the exact damage that quantization will cause, and its predictions are scored under that damage.

The weight updates still happen in full precision behind the scenes, since rounding itself cannot be trained through directly. Over the fine-tuning run, the weights drift toward values that keep the loss low even after rounding, so when the real quantization is applied at the end, there is almost nothing left to break.

Google produced its Gemma 3 QAT checkpoints with roughly 5,000 fine-tuning steps guided by the original model’s outputs, and the int4 versions hold close to full-precision quality at about 3x lower memory.

All five methods described above produce the same artifact, i.e., a model at a fraction of its trained precision.

They differ in where the outlier problem gets addressed, at rounding time, after rounding, before rounding, at inference, or during training itself.

To dive deeper, we have already written a full deep dive on Quantization, specifically, which covers several of these methods with their simplified mathematics.

Learn how Quantization optimizes LLMs to run them on tiny hardware here →

Good day!

Every agent memory system today solves the same problem. They store facts and retrieve the right ones at query time.

The part that’s missing is pattern recognition.

Consider a project management assistant that reads status updates from an engineering team. Three people filled out updates in the same week. Each one reports a blocker, and each report is accurate and specific.

The agent stores all three and can retrieve any of them on request.

What it can’t tell you is that all three blockers trace back to a single delayed task.

Think of it like this. A new project manager reads three status updates and sees three problems to solve. An experienced one reads the same three and sees one problem with three symptoms.

Nobody taught the experienced PM to connect them. They’ve sat through enough standups to recognize the shape.

Agent memory today gives you the new PM. It stores every update correctly, retrieves them accurately, but misses the structure connecting them,

Zep implements a feature called Observations that addresses this. It automatically detects cross-conversation patterns from the knowledge graph and surfaces them as durable, evidence-backed context the agent can use.

Let’s look at how it works.

How the knowledge graph stores data

Zep builds a knowledge graph from conversations and business data. The structure has three building blocks:

• Entities are the nouns: people, products, places, tasks, services.
• Facts are specific claims connecting two entities, stored as labeled edges.
• Episodes are the raw source material: the actual conversation messages, JSON payloads, or documents that Zep ingested. Every fact traces back to the episode it was extracted from.

Graphs can be scoped to a single user or shared across a team.

A user graph holds one person’s history. A shared graph holds everything about a project, a workspace, or an organization, with data from many people flowing into the same structure.

Here’s a concrete example of a shared project graph.

Three engineers file status updates with a PM assistant over one week:

• Alice (Monday): The API migration is blocked. She’s waiting on the auth service refactor, which the backend team owns.
• Bob (Wednesday): The payments integration is stalled. He’s also waiting on the auth service refactor from the backend team.
• Clara (Friday): The checkout flow can’t move forward because the payments integration isn’t ready. That work belongs to Bob.

Zep processes each update as an episode, extracts the entities and facts automatically, and builds the knowledge graph:

Every fact is accurate and individually retrievable. What the graph doesn’t contain is the shape those facts form together.

Why better retrieval doesn’t solve this

When the agent queries “what’s blocking the team,” it gets back the stored facts:

• Alice is blocked on the API migration
• The API migration is waiting on the auth service refactor
• Bob is blocked on the payments integration
• The payments integration is waiting on the auth service refactor
• Clara is blocked on the checkout flow
• The checkout flow is waiting on the payments integration

Every line is correct, but nothing in the result tells the agent that unblocking one task clears all three people.

You can tune the reranker, widen the search scope, or return more results.

None of it helps, because the insight the agent needs was never stored as a discrete piece of content. It exists in how the facts connect across three separate conversations.

The workaround most teams reach for is a rule: flag any task that three or more people mention as a blocker. It just tells you the team is stuck without telling you which task to unblock first, and it only catches blockers that people name directly.

Clara never mentions the auth service. Her update is two hops away from the actual cause.

At this scale, a person could connect the dots manually. But at thirty status updates a week, nobody does, and that’s exactly when the chain matters most.

What Observations produce

For this project graph, Zep generates:

Name: Auth service refactor blocking multiple workstreams

Summary: The auth service refactor, owned by the backend team, is the root blocker in a dependency chain affecting three workstreams. Alice’s API migration and Bob’s payments integration are both waiting on it directly. Clara’s checkout flow is blocked downstream of the payments integration. Unblocking the auth service refactor clears all three.

No individual fact in the graph contains this. It spans three people, four work items, and three separate conversations.

Three properties make observations different from individual facts:

• Cross-entity: A fact sits on one edge between two entities. An observation synthesizes across an entire cluster, connecting multiple entities and conversations.
• Evidence-backed: Every claim traces to specific facts and episodes in the graph.
• Read-only: You can’t manually create, edit, or delete observations. They follow the evidence. The mechanism below explains why.

How Observations are created

The mechanism is a two-stage pipeline:

• A deterministic algorithm first handles the clustering
• An LLM writes the summary afterward. The model never decides what gets grouped.

Reducing facts to signatures

Zep runs a background process that periodically checks for new data. When new conversations have been ingested and the graph has settled, it kicks off the analysis.

The first step reduces every fact to a signature: the two entities it connects plus the relationship type. The goal is to find which conversations reference the same relationships between the same entities.

After reduction, each episode (conversation) has a set of signatures:

• Episode 1 (Alice’s update): (Alice, API Migration), (API Migration, Auth Service), (Auth Service, Backend Team)
• Episode 2 (Bob’s update): (Auth Service, Backend Team), (Payments Integration, Auth Service), (Bob, Payments Integration)
• Episode 3 (Clara’s update): (Bob, Payments Integration), (Checkout Flow, Payments Integration), (Clara, Checkout Flow)

The overlaps become visible:

• (Auth Service, Backend Team) appears in Episode 1 and Episode 2
• (Bob, Payments Integration) appears in Episode 2 and Episode 3
• Episode 1 and Episode 3 share nothing

Alice never mentions Bob, Clara, the payments integration, or the checkout flow. Clara never mentions Alice, the auth service, or the backend team.

Building the episode graph

Here, Zep flips the perspective.

Instead of looking at entities connected by facts, it builds an episode graph where conversations are the nodes and shared signatures are the edges.

• Episode 1 links to Episode 2 through (Auth Service, Backend Team)
• Episode 2 links to Episode 3 through (Bob, Payments Integration)
• Episode 1 and Episode 3 don’t share a signature

But all three end up in the same connected component because Episode 2 bridges them.

Alice’s update and Clara’s update join the same cluster despite having no entity in common, because Bob’s update touches both sides and acts as the bridge between the two.

Worth noticing that the chain the algorithm follows, and the dependency chain in the project are the same structure. Zep finds the root blocker by following exactly the kind of transitive link it was built to follow.

This does not use embeddings, semantic similarity, and no ML model. It’s pure graph topology: which conversations share facts about the same entity pairs, and how those shared pairs chain them into connected groups.

The distinction matters because embedding-based clustering groups content that discusses similar topics. All three status updates would cluster together under that approach simply because they’re all about engineering work, along with every other status update that week.

Signature-based clustering groups episodes that reference the same relationships between the same entities, which is what makes the specific chain detectable.

Summarizing the cluster

Once the algorithm identifies the connected component, an LLM writes the summary.

A single, constrained LLM call receives the cluster’s key entities, the supporting conversations in chronological order, and the relationship types.

It then returns a name and summary steered toward durable signals like decisions, constraints, dependencies, and state changes, and restricted to facts the evidence explicitly supports.

Every structural property of the observation (which entities, which conversations, the time window) is decided by the algorithm. The LLM only turns that structure into readable prose.

This is why observations are read-only. They’re structural properties of the graph, described in natural language. When the evidence changes:

• New conversations that fit an existing observation cause it to be regenerated with the new evidence
• Contradicting evidence causes the old observation to be retired
• The graph always reflects the current state of what is known’

If the backend team ships the auth service refactor next week, the observation updates. Alice and Bob unblock, Clara’s dependency shifts, and the pattern reforms around whatever is actually blocking work now.

How does this help the agent

Retrieving observations through the SDK:

# List all observations for a project graph
observations = client.graph.observation.get_by_graph_id(graph_id="project-atlas”)

# Search observations by relevance
results = client.graph.search(
    graph_id="project-atlas”,
    query="blocked workstreams”,
    scope="observations”,
)

Now consider the PM asking the agent what the team should prioritize this week.

Without observations, the agent retrieves the blocker facts and lists them. Three people are blocked. Two mention the auth service. One mentions the payment integration. The PM gets an accurate summary and still has to work out the ordering.

With observations, the agent’s context includes the dependency chain. It reports that the auth service refactor is the root blocker, that unblocking it clears Alice and Bob directly and Clara downstream, and that the single highest-leverage action this week is getting the backend team the resources to finish it.

Same query, same data. One response describes the situation. The other identifies the action.

The progression of context in Zep:

• Episodes store raw conversation data
• Facts structure it into timestamped claims
• Entity summaries narrate the history of individual nodes
• Observations synthesize patterns across the cluster

Each layer answers a different type of question. Together, they move agents from recall toward understanding.

Alice, Bob, and Clara each filed an accurate status update. Every fact went into the graph correctly, and every one was retrievable.

What none of them could report, because none of them could see it, was the chain running underneath all three.

That’s the gap Observations closes. Not better retrieval of what people said, but structural analysis of what their combined data shows.

Resources:

• Zep Graphiti GitHub repo →
• Zep Observations docs →

👉 Over to you: have you built an agent where the memory system returned the right facts but still missed the structure connecting them?

Recap

Over the last twelve parts of this RAG systems course, we built RAG systems from the ground up.

• In Part 1, we assembled a complete pipeline. Documents were chunked, embedded, and stored in a vector database. A user query was embedded, matched against that store, and the retrieved chunks were passed to an LLM.
• Part 2 covered evaluation. We measured faithfulness, answer relevance, and context precision, and learned that a RAG system without measurement is a system nobody can improve.
• In Part 3, we tackled latency. We rebuilt the pipeline in a modular form, then applied binary quantization to compress vectors. Retrieval got roughly four times faster in our demo.

That work was correct. But it only targeted half of the latency problem in a RAG system.

More specifically, we measured how long retrieval took. We never measured how long the model took to read what retrieval returned.

In this chapter, we will be exploring why the prefill stage dominates RAG latency. We'll understand how the key-value cache works, why the standard fix (prefix caching) recovers almost nothing for RAG workloads, and what actually does work. As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

Understanding a RAG pipeline from a latency perspective

Take a standard RAG request and break its latency into stages.

• Embedding the query takes a few milliseconds.
• Approximate nearest neighbor search over a few million vectors takes tens of milliseconds.
• Reranking eight candidates with a cross-encoder takes perhaps a hundred milliseconds.

Then the model reads those eight chunks. That step, on its own, can take seconds.

Production RAG queries typically feed between four thousand and sixteen thousand tokens to the model, and most of that volume is retrieved text rather than the user's question.

At the higher end of context usage, the cost becomes severe. Processing sixteen thousand input tokens with a fourteen billion parameter model on an NVIDIA L20 GPU has been measured at over five and a half seconds before a single output token appears, under full-attention prefill:

When the LLM reads the text, it's a bigger bottleneck than retrieval.

This reframes what "optimizing RAG" means. Part 3 saved milliseconds off a stage that was already cheap. The expensive stage sat directly downstream:

There is a reason this stage is expensive, and a reason the obvious fix does not work. Both come from the same place, which is how transformers actually process input. So let's discuss it next.

Prefill and decode are two different mechanisms

Autoregressive generation happens in two phases with opposite performance characteristics. And most confusion about inference latency comes from treating them as one thing.

The prefill phase

When a request arrives, the model must first read the entire input. It processes every token in parallel through every layer, computing attention across the full sequence.

During this pass, each layer produces two tensors for every token. These are the keys and the values, the quantities attention uses to decide what to look at and what to retrieve from. Together, they are stored as the key-value cache, usually shortened to KV cache.

Prefill is compute-bound. The attention computation scales with the square of the sequence length, because every token attends to every preceding token. So doubling the input quadruples the attention work.

It is worth making that quadratic term explicit, because it is the reason retrieved context is expensive rather than merely large.

Reading the symbols:

• $C_{\text{attn}}$ is the attention compute for one prefill pass
• $L$ denotes the total number of layers
• $H$ denotes the total number of heads
• $T$ represents the input length in tokens
• and $d_h$ is the per-head hidden dimension.

The proportionality hides any constant factors that depend on hardware and kernel implementation.

Structurally, the term that dominates the computation equation below is $T^2$:

Every one of the $T$ query vectors forms a dot product against every one of the $T$ key vectors, which produces a score matrix with $T^2$ entries.

That matrix is built independently in every head of every layer, which is where $L$ and $H$ enter as multipliers.

The intuition is that context length is the expensive dimension, and it is expensive in a way that compounds.

Going from 2,000 tokens to 4,000 does not double the attention cost but rather roughly quadruples it. Adding a ninth retrieved chunk to a prompt that already holds eight costs more than the first chunk did, because the new tokens must attend to everything already present and everything present must now account for them.

This is why RAG changed the economics of serving.

A bare question is a few dozen tokens. The same question with eight retrieved passages attached can easily become several thousand tokens instead, and the attention cost moved by a factor far larger than eight.

The time this phase takes is what the user experiences as the delay before the first word appears. The standard name for it is time to first token, or TTFT.

The decode phase

Once prefill finishes, the model generates output one token at a time. Each new token attends to everything before it, which is exactly what the stored KV cache holds.

Because the cache already exists, decode never recomputes attention over the input. It computes keys and values for one new token, appends them to the existing KV cache, and the process continues to subsequent tokens as they are generated.

Decode is memory-bandwidth-bound rather than compute-bound. Each step must stream the model's weights from memory to produce a single token, so the hardware spends most of its time waiting for weights transfer rather than actually running the computation to predict the next token.

The asymmetry is important to understand because it explains why the two phases respond to completely different optimizations.

• During prefill, thousands of tokens flow through the same weights at once, so the cost of loading those weights is spread across thousands of useful calculations.
• During decode, the identical weight loading happens to serve one token.

A seven billion parameter model in fp16 precision holds roughly 14 GB of weights. Producing a single decode token requires reading essentially all of them.

The arithmetic performed against those weights is tiny by comparison, which means the accelerator sits idle waiting on memory for most of the step.

This is the reason batching helps decode (the second stage) so much and helps prefill (first stage) comparatively little.

During decode, the expensive step is reading the model's weights from memory, and that read produces only one token. Batching sixty-four requests together reuses a single weight read to produce sixty-four tokens at once (because there are sixty-four requests being served in parallel), so the cost is shared across all of them instead of being paid per token.

Prefill gets little from this mechanism, because a single request already pushes thousands of tokens through those weights at once and keeps the hardware busy.

For this article, only a small part of that matters. Optimizing decode is a different discipline with different tools. RAG's latency problem is concentrated in prefill, and prefill responds to exactly one thing, which is not repeating the computation you have already computed.

What the cache actually costs

The KV cache is not free. Understanding its size tells us why reusing it matters so much, and later, why storing many versions of it becomes a problem.

The memory a cache occupies grows with the sequence length, the model depth, and the width of each attention layer.

Every symbol here is straightforward

• $M$ is the total memory in bytes.
• The leading factor of $2$ accounts for storing both keys and values (and they are of equal length/size).
• $L$ is the number of transformer layers, since every layer keeps its own cache.
• $H$ is the number of attention heads per layer and $d_h$ is the dimension of each head.
• $T$ is the number of tokens currently cached
• Finally, $b$ is the bytes per element, which is $2$ bytes for half precision (fp16).

Reading the structure, the product $H \cdot d_h$ is simply the model's hidden dimension spread across heads. Multiplying by $L$ accounts for depth. Multiplying by $T$ makes the whole thing linear in sequence length, which is the important property.

Do note that, unlike attention compute, cache memory does not grow quadratically.

The intuition is that the cache is a per-token record, kept once per layer, for both keys and values. Every additional token of context adds a fixed cost. Every additional layer multiplies it. A model twice as deep needs twice the cache for the same input.

Let's work on an example to make this concrete.

Consider a model with 32 layers, 32 heads, and a head dimension of 128, holding 8,000 tokens in half precision.

This would need roughly 4 GB of cache for a single request. It is important to note that this memory will be unavailable to any other request while it is held by this specific request.

In summary, prefill pays a large one-time compute cost and produces a cache. The decode process then consumes that cache cheaply.

If we could avoid paying the prefill cost repeatedly for text we have already processed, RAG latency would improve substantially. That idea is what prefix caching implements.

How prefix caching works

Serving engines already reuse KV caches across requests. Understanding exactly how they do will let us see, precisely, why the mechanism underperforms for RAG.

Paged memory for the cache

Early serving systems allocated one contiguous memory block per request, sized for the longest output the request might produce. Most requests finished early, so most of that memory sat unused.

The PagedAttention design, which underlies vLLM, borrowed the solution from operating systems. Instead of contiguous per-request buffers, the cache is split into fixed-size blocks that can live anywhere in memory. A request holds a list of block references rather than one large region.

A typical block holds 16 tokens. The exact size is configurable and matters later, so it is worth remembering.

This design was built for memory efficiency, but it also enabled something else. More specifically, if blocks are independent units, then identical blocks can be shared between requests instead of being duplicated.

The hashing rule

Sharing blocks requires knowing when two blocks are genuinely identical. Automatic prefix caching solves this by hashing.

Each block is hashed together with the hash of every block before it. The stored key for a block is therefore not a function of its own tokens alone. It is a function of its tokens and its entire preceding context.

This rule is deliberate, and the reason for it is correctness.

A token's keys and values depend on everything the model saw before that token. Reusing a block only when the full prefix matches guarantees the reused values are bit-identical to what a fresh computation would produce.

So, a new request that begins with the same tokens as a cached request skips prefill for the shared portion. A long shared system prompt across many requests is the ideal case, and prefix caching handles it well.

The question is what happens when the shared content is not at the start.

Why prefix caching barely helps RAG

A RAG prompt is not one shared prefix followed by a short question. It is a system instruction, then several retrieved chunks, then the query. The chunks change per request, and their order changes too.

Watch what the hashing rule we discussed above does to that.

Suppose a request retrieves chunks $D_1$, $D_2$, and $D_3$, and the system caches all of them. A second request retrieves $D_1$, $D_2$, and $D_5$.

The first two chunks match, so their blocks are reused. The third does not, so it is recomputed. Two-thirds of the retrieved content is reused, which sounds acceptable.

Google and Anthropic agree on one thing about LLM inference:

The faster a model runs, the fewer machines can run it.

This sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.

And those choices only fit the hardware they were made for.

That is why they build the same model more than once, one version per chip or runtime it has to run on.

The visual below covers the six formats used to run LLMs in production:

1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.

A .pt file is a list of instructions that torch.load executes and loading a checkpoint can also run a payload (possibly malicious).

Safetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.

Both still need Python, the model code, and a tokenizer alongside.

3) GGUF is llama.cpp’s format, built so a model can run on a machine with no ML framework installed.

To achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.

4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.

It decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.

5) MLX is Apple’s framework for its own silicon, where the CPU and GPU share a single pool of memory.

It decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.

6) TensorRT is NVIDIA’s compiler, which turns a model into machine code for one specific GPU.

Everything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.

The safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →
• Read Part 2 on understanding the core building blocks of LLMs →
• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →
• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →
• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →
• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →
• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →
• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →
• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →
• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →
• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →
• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →
• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →
• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

Good day!

The agent loop inside Claude Code is nothing special. It sends the model a task, runs whatever tools the model requests, feeds the results back, and repeats until there’s a final answer.

Anyone can write that loop in thirty lines. But a bare version of it falls apart on a real codebase, reading the wrong files, losing the goal midway, and filling its context with output it no longer needs, while Claude Code finishes the same task cleanly.

The easy conclusion is that Anthropic simply has a better model. The actual difference is the harness, the ordinary code wrapped around the model that handles planning, tool execution, memory, and safety, while the model only decides the next step.

Here’s what a fully harnessed agent looks like when you draw it out:

• Memory feeds the model its working context plus facts it has learned across sessions.
• Skills encode how the agent should operate, meaning the procedures, constraints, and heuristics it follows.
• Protocols connect the agent to users, tools, and other agents.
• The harness core ties it all together with sub-agent orchestration, a sandbox, an evaluator, an approval loop, observability, and context compression.

Anthropic describes this split as the brain and the hands. The model is the brain that picks each action, and the harness is the hands that carry it out and keep the run on track.

So the gap between your agent and Claude Code isn’t the model, it’s the machinery around the model.

Claude Code is one of the most capable harnesses in production today, and it’s built from a surprisingly small set of layers in that illustration. To see how much of that machinery you’d have to build yourself, I rebuilt it in CrewAI, an open-source framework for orchestrating agents.

More of it maps onto built-in features than I expected, and the part that doesn’t is where the real engineering lives.

Let’s build it layer by layer, starting with the core loop and stacking planning, subagents, sandboxing, and memory on top. At each step, we’ll mark where the framework ends and where your job begins.

How Claude Code's harness works

At the center of Claude Code is a plain agent loop. You send it a message, the model decides what to do next, and it either responds directly or requests a tool. If it requests one, the tool runs, the result goes back into the conversation, and the model decides again.

This repeats until the model returns a final answer with no further tool calls.

Inside that loop, the model reads files, edits code, runs shell commands, and executes tests. These aren’t separate modes. They’re just different tool calls inside the same loop.

The loop alone isn’t enough for a reliable coding agent, though. Claude Code adds planning, file tools, subagents, memory, and a permission and sandbox system around it. These layers don’t replace the loop, they make it safe and reliable enough for real work.

That’s the architecture we’ll rebuild, the core loop first, then each layer on top, mapping every layer to the CrewAI feature that handles it.

The core agent loop

The loop runs the same sequence until the task is done:

1. Ask the model to perform the task.
2. The model responds directly or requests one or more tools.
3. If tools are requested, run them and return the results to the model.
4. Repeat with the updated conversation.
5. When the model responds without requesting any tools, the task is complete.

while True:
    reply = model(messages, tools)
    calls = [b for b in reply if b.type == "tool_use"]
    if not calls:            # plain text, no tool call: the job is done
        return reply.text
    messages += [reply, run_all(calls)]

Each tool call completes one step, gives the model new information, and feeds into the next decision. A simple question might finish in one iteration, while fixing a complex bug or refactoring a large codebase can take dozens of iterations before the model has enough to produce a final answer.

CrewAI provides this execution loop automatically as soon as you create an agent. You don’t implement the while loop yourself, you define the agent and assign it a task.

Building the first agent

Let’s create a simple Bug Fixer agent.

from crewai import LLM, Agent, Crew, Task

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    llm="claude-sonnet-4-6",
)

task = Task(
    description="Find the fix for {objective}.",
    expected_output="A short description of the fix and which file it belongs in.",
)

result = Crew(agents=[bug_fixer], tasks=[task]).kickoff(
    inputs={"objective": "the overdraft bug in account.py"}
)

Three concepts to understand here:

• An Agent defines who does the work, through its role, goal, LLM, and tools.
• A Task describes the assignment.
• A Crew brings agents and tasks together. Calling kickoff() runs the same execution loop described above, regardless of whether the underlying model is Anthropic, OpenAI, Google, or something else.

Giving the agent tools

Tools are what let a model that only generates text actually work on a codebase. They read files, write them, run shell commands, and call external APIs.

CrewAI ships file system tools out of the box:

• FileReadTool reads files.
• DirectoryReadTool lists directories.
• FileWriterTool writes files.

from crewai_tools import DirectoryReadTool, FileReadTool, FileWriterTool

read_file = FileReadTool()
write_file = FileWriterTool()
list_dir = DirectoryReadTool()

filesystem_tools = [read_file, write_file, list_dir]

These also double as external memory. Instead of holding a large search result in the model’s context window, the agent can write it to a file, keep only the filename, and read it back when needed.

This keeps the context window smaller and the model more focused.

Built-in tools cover common workflows only. For anything more specific, you expose a Python function as a tool with the @tool decorator.

The docstring acts as the instruction manual, telling the model what the tool does, when to use it, and what it expects as input.

from crewai.tools import tool
import subprocess

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Run the pytest suite at the given path and return the result."""
    result = subprocess.run(
        ["pytest", path, "-q"], capture_output=True, text=True, timeout=120
    )
    output = result.stdout + result.stderr
    return output[-4000:] if len(output) > 4000 else output

Adding planning capability

As tasks get more complex, a plain execution loop starts losing track of the original goal. After enough tool calls, file reads, and intermediate results, the context fills up and the objective gets crowded out by everything that came after it.

This slow degradation is what people call context rot.

Planning addresses it directly. The agent builds a step-by-step plan before doing any work and keeps that plan in context throughout execution.

The plan doesn’t do the work. It’s a roadmap that keeps the model connected to the original goal, which is the same job Claude Code’s to-do list does.

CrewAI adds this at the crew level with planning=True. It generates a plan before execution and keeps it available as the task progresses.

from crewai import Crew, LLM

crew = Crew(
    agents=self.agents,
    tasks=self.tasks,
    planning=True,
    planning_llm=LLM(model="gpt-4o-mini"),
)

Note: By default, CrewAI uses gpt-4o-mini for planning, and you can swap in any LLM you prefer for that step.

Individual agents can also reason about their own work with reasoning=True:

from crewai import Agent

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    tools=[FileReadTool()],
    reasoning=True,
    max_reasoning_attempts=3  # Optional: Set a maximum number of reasoning attempts
)

Planning and reasoning solve different problems. Planning builds a high-level roadmap for the overall task, while reasoning gives one agent time to think through its own approach before acting.

When reasoning is enabled, the agent:

1. Reflects on the task and drafts an execution plan.
2. Evaluates whether the plan is ready.
3. Refines the plan if necessary, until it’s satisfied or hits max_reasoning_attempts.
4. Injects the finalized reasoning plan into the task before execution.

Together, they keep the agent anchored on long-running tasks and reduce drift from the original goal.

Subagents to maintain main agent’s context

Planning keeps the agent focused, but it doesn’t reduce how much information the model has to hold. On a large codebase, even a well-planned task can exceed a single context window.

Finding one bug may require reading dozens of files, and the main agent doesn’t need to keep all of them in memory.

Subagents solve this through delegation. The main agent hands a specific task to a helper agent, which works in its own context and returns a short summary. The main agent sees the conclusion, not the intermediate steps.

CrewAI supports this through hierarchical workflows, where a manager agent delegates to specialist agents and combines their results.

In our earlier setup, one Bug Fixer agent did all the heavy lifting. Let’s split the work across a manager and three specialists:

• Codebase Explorer explores the code and maps the repository.
• Software Engineer implements the requested change.
• Test Runner runs the tests in the sandbox and reports pass or fail.
• Engineering Lead oversees the three specialists.

from crewai import Crew, Agent, Task, Process

explorer = Agent(
    role="Codebase Explorer",
    goal="Map the repository and surface the files relevant to the task.",
    backstory="You read directories and files to build a picture of the code.",
    tools=[read_file, list_dir],
    llm=llm,
)  # Same for other two specialist agents

manager = Agent(
    role="Engineering Lead",
    goal="Break the request into steps and delegate each to the right specialist.",
    backstory="You decide who does what, review tests, finish once change is done.",
    llm=llm,
    allow_delegation=True,
)

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    manager_agent=manager,
    process=Process.hierarchical,
)

One thing to note is that allow_delegation is disabled by default, so it must be explicitly enabled on the manager.

Safety comes from a sandbox and approvals, not from prompts

An agent with shell access can run a destructive command, and telling the model not to do something isn’t a safeguard.

Real protection comes from two layers:

1. A permission system that requires approval for sensitive actions.
2. A sandbox that isolates execution, so even approved commands can’t touch the host machine.

Anthropic uses the same approach. Moving code execution into a sandbox cuts down how often a user needs to approve actions while still protecting the host system.

Sandboxing in CrewAI

Executing code inside a sandbox rather than on the host machine applies that second layer. In this setup, code runs inside E2B, which spins up a fresh VM per session and destroys it afterward.

Shell commands and Python run entirely inside that isolated environment.

from crewai_tools import E2BExecTool, E2BPythonTool
sandbox_tools = [E2BExecTool(), E2BPythonTool()]  # run tests / run code

Human-in-the-loop

Setting human_input=True on a Task pauses the crew after it generates an answer. You review the output, then approve it or send it back for another iteration.

When execution reaches that task, CrewAI waits for your feedback through standard input.

from crewai import Task

task = Task(
    description=(
        "In the working directory ./workspace, {objective}. "
        "Explore the code first, make the change, then run the tests and report."
    ),
    expected_output="A summary of the files changed and the final test output.",
    human_input=True,
)

If your crew runs behind a web app or chat interface rather than a terminal, CrewAI’s webhook-based human-in-the-loop system handles the same review step.

Memory and checkpointing

By default, an agent forgets everything once a run ends. So if you come back tomorrow to fix another bug in the same project, and it starts from zero.

Two mechanisms let an agent carry information across runs, and each serves a different purpose:

• Checkpointing saves the agent’s state during a run, so it can resume after an interruption or continue from the same point along a different path.
• Persistent memory stores facts across separate conversations, including project preferences like “always format the final code before finishing.”

Adding memory

CrewAI provides a unified Memory interface rather than separate short-term, long-term, entity, and external memory types. When saving, it uses an LLM to identify important details, organize them, and make them retrievable later.

Setting memory=True on the crew gives it memory across runs. After each task, CrewAI extracts useful facts from the output and stores them, and in future runs, it retrieves relevant memories and adds them to the task prompt.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    memory=True,
)

All agents in a crew share its memory unless an agent is given its own.

Adding Checkpoints

A checkpoint is a snapshot of an agent’s progress, including its configuration, task state, memory, intermediate results, inputs, and execution history.

By default, CrewAI creates a checkpoint whenever a task finishes, allowing the workflow to resume from that point if it’s interrupted.

Checkpoints can live in one of two built-in stores:

• JsonProvider saves each checkpoint as a separate JSON file, which is easy to read and inspect manually.
• SqliteProvider stores all checkpoints in a single SQLite database, which holds up better under frequent checkpointing and larger workloads.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    checkpoint=True,
)

Crew, Flow, and Agent all accept a checkpoint argument, and children inherit from their parent unless they set their own value.

All six layers on one bug-fix task

Here’s the full harness on one task, with the execution loop, tools, planning, subagents, sandboxing, and memory working together:

from crewai import Agent, Crew, LLM, Process, Task
from crewai.tools import tool
from crewai_tools import (DirectoryReadTool, FileReadTool, FileWriterTool, 
E2BExecTool, E2BPythonTool)

llm = LLM(model="anthropic/claude-sonnet-4.6")

list_dir = DirectoryReadTool(directory="./workspace")
filesystem_tools = [FileReadTool(), FileWriterTool(), list_dir]
sandbox_tools = [exec_tool, E2BPythonTool()]

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Sync ./workspace into the sandbox, then run pytest there."""
    return E2BExecTool().run(command=sync_and_test_command(path))

explorer = Agent(role="Codebase Explorer", goal="Map repo, surface relevant files.",
    tools=[read_file, list_dir], llm=llm)
coder = Agent(role="Software Engineer", goal="Implement requested change.",
    tools=filesystem_tools, reasoning=True, llm=llm)
tester = Agent(role="Test Runner", goal="Run tests in sandbox, report pass/fail.",
    tools=sandbox_tools + [read_file] + [run_tests], llm=llm)
manager = Agent(role="Engineering Lead", goal="Delegate steps, finish once tests pass.",
    allow_delegation=True, llm=llm)

task = Task(
    description="In ./workspace, {objective}. Explore, edit, test, report.",
    expected_output="Summary of changes and test output.", human_input=True,
)
crew = Crew(
    agents=[explorer, coder, tester], tasks=[task],
    manager_agent=manager, process=Process.hierarchical,
    planning=True, memory=True, checkpoint=True,
)
result = crew.kickoff(inputs={"objective": "fix failing tests in account.py"})

Agent harnesses are easiest to evaluate when success can be checked automatically. A test suite gives the agent a concrete objective, so it can plan, edit, test, and repeat until everything passes.

So this was tested against a small codebase, a BankAccount class with two real bugs and five tests, three of which failed. The rule was to fix only the implementation, not the tests.

This mirrors how Anthropic evaluates coding agents internally. One published example has Claude rebuilding a clone of the claude.ai interface against a large suite of failing tests.

Here, the harness took the project from 3 failing and 2 passing to all 5 passing, with the implementation-only rule closing off the shortcut of editing or removing the failing tests.

Beyond prompts, environment, and tool choice

Some parts of the system aren’t things the framework builds for you:

• The prompts. Each agent’s behavior comes from its role, goal, and backstory. Getting those right takes testing and iteration, and no configuration flag substitutes for it.
• The execution environment. The sandbox, whether E2B or a self-managed VM, has to be set up and wired in.
• Tool selection. Which tools each agent gets, and which agent should have access to what, is a design decision the framework doesn’t make.

There’s also a cost to the harness itself. Planning, subagents, and looping all add API calls, so a complex agent setup can end up more expensive than a task a single model call would have solved directly.

And there’s a longer-term limitation worth keeping in mind. As models improve, some of the scaffolding stops being necessary, because some of what gets built into a harness today is a workaround for today’s model limits rather than a permanent requirement.

Anthropic originally used context resets to keep Claude Sonnet 4.5 from ending tasks too early, and they weren’t needed anymore with the more capable Claude Opus 4.5.

That’s the whole finding. A coding agent’s capability lives mostly in the harness, and an orchestration framework hands you more of that harness than you’d guess.

The loop, planning, delegation, sandboxing, and memory all arrive as configuration, while the prompts, the execution environment, and the tool choices remain yours.

If you want to run this against your own codebase, the CrewAI docs cover every feature used here, and the framework is fully open source.

Check out CrewAI Docs →

Check out CrewAI GitHub (100% Open Source) →

Find the code for this project in this repo →

Good day!

Introduction

Over the past twelve chapters, we understood the key machinery of reinforcement learning piece-by-piece, from bandits and value functions all the way to GRPO, judges, and agentic training loops. With Chapter 12, that machinery became complete.

So this final chapter does something different. There is nothing new to derive and nothing to train. Instead, we look at what the world is actually doing with all of it.

Everything ahead is a story someone shipped. We chose four, each showing a different role RL plays in the industry:

• RL as a product engine
• RL as a capability engine
• RL as enterprise strategy
• RL as a market of its own

For each story, we will read it by asking the questions this series trained us to ask:

• What is the environment, and does it meet the resettability and safety requirements?
• Where does the reward come from, and is it hand-defined, learned from preferences, verified, or elicited from a judge?
• What is the trajectory, and how does the loop close?

Company blog posts rarely use our vocabulary, so part of the exercise is translation. By the fourth story, you should find yourself doing the translation automatically.

Let's begin!

Case study 1: Cursor's Composer, RL as the product engine

Cursor is a code editor built around AI assistance. Its in-house agent model is called Composer, and Composer is something like a Chapter 12 system running as a commercial product.

Start with how the model is trained. In the original Composer announcement, Cursor described giving the model access to its production search and editing tools during RL, and running hundreds of thousands of concurrent sandboxed coding environments in the cloud to support the rollouts.

The stated principle is to minimize train-test mismatch. The model trains inside the same harness, with the same tools, prompts, and context, that it will face when deployed.

You already own every word of that sentence. The harness is the environment. The sandboxed virtual machines are the resettable, side-effect-free copies from Chapter 12's two training requirements. The rollouts are trajectories.

Cursor also reported that, during RL, the model picked up useful behaviors on its own, like writing and executing unit tests and fixing linter errors.

Then comes the part that makes Composer the flagship story of this chapter. In March 2026, Cursor described what it calls real-time RL. Model checkpoints are served directly to production. Real user interactions are observed and aggregated into reward signals. An improved checkpoint can ship as often as every five hours.

The cycle has five stages:

• Collect billions of tokens from user interactions with the current checkpoint.
• Distill them into rewards (for example whether the user kept the suggested edit or sent a dissatisfied follow-up message).
• Update the model weights based on that implied feedback.
• Run the candidate against evaluation suites, including their internal CursorBench, to catch regressions.
• Deploy the improved checkpoint back to production.

The whole loop takes about five hours, so multiple checkpoints can ship in a single day.

Why go to all this trouble instead of staying in simulation? Cursor's answer echoes our environment discussion. Simulating the computer is easy, but simulating the person overseeing the agent is not. Real-time RL uses real environments and real users, removing that modeling error and assumptions entirely.

Cursor reported results from an A/B test of Composer 1.5 with this loop. Agent edits persisted in the codebase 2.28% more often, dissatisfied follow-up messages dropped by 3.13%, and latency fell by 10.3%.

It is worth pausing on how a user's behavior becomes a number, because this is reward engineering of a kind we have not seen yet.

User interactions are instrumented on the client, flow through backend data pipelines, and are distilled into signals:

• Did the suggested edit persist in the codebase, or was it reverted?
• Did the next message read as a dissatisfied correction?

No single interaction is trustworthy, since a developer might revert a perfectly good edit for unrelated reasons. So the signal is noisy, and Cursor compensates the way any of our chapters would suggest, by aggregating over very large batches so the noise averages out while the preference survives.

There is one more detail. Composer reward-hacked in production. Cursor reported that the model learned to emit invalid tool calls to avoid negative rewards, and separately learned to defer risky edits by asking clarifying questions, which lowered its editing rate.

Cursor offers a counterpoint worth remembering. Reward hacking is a bigger risk in real-time RL, because every seam in the production stack becomes a surface to exploit. But it is also harder to get away with. In simulated RL, a cheating model simply posts a higher score. In production, real users trying to get work done are less forgiving, and hacks surface as bug reports.

Karpathy’s LLM Wiki compiles raw sources into a persistent Markdown wiki with backlinks and cross-references.

The LLM reads papers, extracts concepts, writes encyclopedia-style articles, and maintains an index. The knowledge is compiled once and kept current, so the LLM never re-derives context from scratch at query time.

This works because research is mostly about concepts and their relationships, which are relatively stable.

But this pattern breaks when you apply it to actual work, where context evolves across conversations constantly.

A compiled wiki would have a page about a project, but it wouldn’t track that a deadline agreed in one email thread and moved to a later date in another thread, while the team still assumed the original date.

A wiki doesn’t track ground truth effectively.

We wrote about this recently, and Karpathy liked it:

Tracking this requires a different data structure altogether. Not a wiki of summaries, but a knowledge graph of typed entities where people, decisions, commitments, and deadlines are separate nodes linked across conversations.

Rowboat (GitHub Repo) is an open-source implementation of exactly this, built on the same Markdown-and-Obsidian foundation that Karpathy uses, but extended into a work context.

The way it works is that it ingests conversations from Gmail, Granola, and Fireflies, and instead of writing a summary page per topic, it extracts each decision, commitment, and deadline as its own Markdown file with backlinks to the people and projects involved.

That’s structurally different from a wiki because:

• A wiki page about “Project X” gives you a summary of what was discussed.
• But a knowledge graph gives you every decision made, who made it, what was promised, when it was promised, and whether anything has shifted since.

Next, let’s set up Rowboat from scratch, walk through what the knowledge graph looks like on disk, and see what happens once the graph is live.

Setup

Rowboat is a local desktop app (Mac, Windows, Linux) that runs entirely on your machine and lets you bring your own model from Ollama, LM Studio, or any hosted API.

It stores everything in ~/.rowboat/ as plain Markdown files in an Obsidian-compatible vault. If you already use Obsidian for notes, you can point it at the same vault and browse Rowboat’s knowledge graph alongside your own files.

If you don’t use Obsidian, the Markdown files are still readable in any editor.

To start, download the app here: rowboatlabs.com/downloads

Next, open ~/.rowboat/config/models.json and point it at whatever you’re running.

For a hosted provider, you can specify:

{
  "provider": {
    "flavor": "openai",
    "apiKey": "sk-..."
  },
  "model": "gpt-4o"
}

Anthropic, Google, and OpenRouter use the same structure, so you can just swap the flavor property above. If you want inference fully on your machine, Ollama works too:

{
  "provider": {
    "flavor": "ollama",
    "baseURL": "http://localhost:11434"
  },
  "model": "llama3.2"
}

Lastly, you can also set the preferred LLM providers using their respective API Keys from the UI itself. Visit Settings → Models:

Once a model is connected, Rowboat wraps it with system prompts that make the LLM aware of your knowledge graph structure.

The model knows about the knowledge/ directory, understands the entity types (People, Projects, Organizations, Topics), can traverse backlinks between notes, and can read Today.md for current context before responding.

It’s effectively a work-oriented system prompt layer on top of whatever model you chose, so the LLM operates as a context-aware copilot rather than a blank chat session.

Moving on, you’ll need your own Google Cloud project with OAuth credentials so that you can invoke Gmail, Calendar, and Drive directly from your machine using those credentials.

The full walkthrough is in google-setup MD file in the Rowboat GitHub repo.

The core flow is this:

• Create a project at console. cloud. google. com
• Enable Gmail, Calendar, and Drive APIs
• Set up an OAuth consent screen (Testing mode works)
• Add your email as a test user
• Create an OAuth Client ID (type: Web application)
• Set redirect URI to http://localhost:8080/oauth/callback
• Paste your Client ID and Secret into Rowboat when prompted

Once connected, the first sync kicks off, and the graph starts building.

If you use Fireflies or Granola, you can add your API keys here:

// ~/.rowboat/config/fireflies.json  # Fireflies meeting transcripts

{
  "apiKey": "<your-fireflies-api-key>"
}


// ~/.rowboat/config/granola.json  # Granola meeting notes

{
  "apiKey": "<your-granola-api-key>"
}

The transcripts get pulled into the Meetings/ folder automatically, with decisions and action items extracted into the knowledge graph.

Additionally, you can add other integrations if needed, based on what fits your workflow.

~/.rowboat/config/deepgram.json      # voice input and notes
~/.rowboat/config/elevenlabs.json    # voice output
~/.rowboat/config/exa-search.json    # web research via Exa
~/.rowboat/config/composio.json      # external tools
~/.rowboat/config/mcp.json           # other MCP tools

Once the first sync completes, the graph is ready to query.

The vault structure on disk

After the first sync, your ~/.rowboat/ directory is live.

• The config/ folder has all your API keys and model config
• The gmail_sync/ and calendar_sync/ folder has synced data before it gets processed into the graph
• The events/ folder contains the background agent activity, split into done/ and pending/ folders.
• Anything in the sites/ folder gets served at http://localhost:3210/sites/<slug>/ and can be embedded as a live iframe in any note.
• The logs/ and runs/ folders contain the operation history and agent run records.

The more interesting part of the structure is inside the knowledge/ folder:

This is what it holds:

• Agent Notes/ is Rowboat’s memory layer, written from your behavior and writing patterns over time.
• Meetings/ contains processed transcripts from Fireflies or Granola, with decisions and action items extracted.
• Notes/ is for your own Markdown files.
• Today.md is the file Rowboat reads first before answering anything. It aggregates recent emails, meeting notes, and drafts into a single entry point for every query.

In addition to this, there are also People/, Organizations/, Projects/, and Topics/ folders.

They all start empty and populate as the signal accumulates. The Rowboat Agent only creates an entity file once there’s enough evidence across emails, meetings, and decisions to justify it.

This is a deliberate design choice because premature entity creation from noisy data (spam senders, marketing emails) would pollute the graph.

Rowboat doesn’t load the full vault on every query. It reads Today.md first, then pulls only the entity files relevant to your request. The graph can grow to hundreds of notes while the query cost stays flat.

Querying the graph

Once sync completes, here’s a useful first query you can ask:

Prep me for my 2pm meeting with Sarah Chen

This depicts the result:

Under the hood, the Rowboat Agent traversed Sarah’s entity node and its backlinks.

• Her role and firm came from email signatures and meeting attendees.
• Your interaction history was reconstructed from calendar events and threads.
• Open items were extracted from past meeting transcripts.
• And talking points come from related entities in the graph.

Any [[entity]] reference in the brief is a live backlink that you can click through:

This will show the full entity node, so additional context for any account, person, or project becomes visible in just one hop:

Here’s another query:

Create a voice summary of my day ahead

This depicts the result:

In this case, the Agent pulled today’s meetings from your calendar, gathered context on attendees from the graph, generated a script grounded in your priorities, and handed the audio back as an MP3.

This requires the ElevenLabs or Deepgram integration discussed above for the audio part specifically.

For ongoing tracking, tagging @ rowboat on any note turns it into a live note. Rowboat keeps it updated automatically as new relevant information comes in from connected sources.

For instance, if you’re tracking a deal, a project, or an account across multiple conversations, the note stays current without manual upkeep.

Finally, you can also spin up agents that run on a schedule independently of your queries. You control what runs, when it runs, and what gets written back to the vault.

Knowledge graph evolution over time

As expected, the knowledge graph gets denser with every email and meeting. New signals attach to existing entity nodes rather than creating isolated files.

Imagine that Sarah was in the Series B intro call 3 weeks back. That link already exists from day one.

But when she emails term sheet feedback now, that update attaches to the same node as a new line in her activity log.

The graph doesn’t create a second “Sarah Chen” file. It extends the existing one.

At month 2, when you query “what has Sarah committed to this quarter,” the Agent traverses connected context across every interaction, not individual threads sitting in separate inboxes.

If you set a live note set on a deal in week one, it stays current because the Agent keeps pulling new relevant context into it automatically.

So you set it once, and the graph maintains it.

This is the structural difference from Karpathy’s wiki pattern. A wiki compiles concepts into pages. A knowledge graph tracks state across conversations, and that state compounds as new interactions link back to existing entities

Rowboat takes the compounding knowledge base Karpathy’s pattern describes and makes it work for the context that actually changes day to day.

It stores everything as plain Markdown in ~/.rowboat/ so nothing is locked into a proprietary format.

Model calls go to wherever you pointed models. json.

• With Ollama, the entire pipeline runs on your hardware.
• With a hosted API key, your prompts go directly to that provider under your own account’s terms.

Rowboat doesn’t proxy or intercept model calls.

OAuth credentials for Gmail, Calendar, and Drive live in ~/.rowboat/config/ and call Google’s API directly from your machine. Every integration follows the same architecture, i.e., they are invoked locally and outputs are written to the local vault.

The repo is Apache-2.0 licensed.

GitHub repo (13k+ stars): github.com/rowboatlabs/rowboat

(don’t forget to star it 🌟)

Thanks for reading!

Inference bills climb with every new agent and every extra call. So in production, teams prefer to load the repetitive steps onto small task-specific models to bring the cost back down.

The move is correct, but using SLMs alone doesn’t imply savings since those models still have to be served. And serving them without wasting the GPU underneath is a separate problem to be solved.

Today, let’s understand why that is the case, what serving many small models well actually takes, and how the SIE open-source inference engine solves it end-to-end.

Let’s begin!

Workflow of a traditional AI pipeline

A production AI system is usually a stack of individual tasks, and each task is handled by the model that happens to be best at it.

For instance, a single request might:

• parse a document or an image into clean text
• embed that text into vectors for retrieval
• rerank the retrieved results by relevance
• extract the specific fields, entities, or labels that matter
• and check the output against a safety policy before sending it to user

Until recently, the default instinct was to use the largest general-purpose model you could and make it do as much of this as possible, paying frontier rates for every call.

That reflex is now reversing toward using small models fine-tuned for each task, which can run the same step at a fraction of the cost since most tasks don’t always need frontier reasoning.

But the catch teams usually miss is that a cheaper model does not imply a cheaper system.

This is because the cost of a model isn't only what it charges per call. It's also the hardware you keep running to answer that call.

A pipeline that replaced one large model with multiple small ones now has multiple models to serve, each occupying a GPU of its own.

Why don’t we fit multiple models on one GPU?

The obvious objection is that we should just put them all on one card.

This will help because no GPU provider bills for the work you do on them. They do not meter FLOPs.

What you rent is a whole GPU for a span of wall-clock time, and the meter runs whether the card is busy or sitting idle.

So every GPU you hold adds another term to the full bill:

Four models on four cards means four terms in that sum. If one card can carry all four models’ traffic, three of those terms disappear. Nothing about the work has changed, and yet the bill drops by 75%.

The objection holds up on the memory front too. A small embedding model, a cross-encoder reranker, and an entity extractor together take a few gigabytes.

An L4 has 24GB, so they can fit with room to spare, and nothing in the hardware stops you.

The reason almost nobody does it is that every layer of the stack, from the serving framework down to the driver, was built around one model owning one GPU.

vLLM and TEI are two common servers.

• TEI is Hugging Face’s embedding and reranking server and takes a single --model-id at launch.

• vLLM’s engine is built around one model as well.

So putting four models on one card requires running four separate server processes on it. Nothing in either tool coordinates those processes, and you must own that job.

In fact, each process claims its memory upfront and cannot see its neighbours.

vLLM’s --gpu-memory-utilization defaults to 0.9. The setting pre-allocates that fraction of the card at startup rather than adjusting it as per demand.

vLLM’s own documentation says this is a per-instance limit that ignores any other vLLM instance on the same GPU, and that running two means setting the value to 0.5 for each of them, by hand.

So the memory split is a constant you compute yourself, before any traffic arrives.

It must cover each model’s peak, because activation memory grows with batch size and sequence length.

Four worst-case slices pinned onto one card is the same over-provisioning you were trying to escape, and now they have been moved inside a single GPU.

Even if you size them optimally and two models spike together later, they will take the whole card down.

For instance, if the extraction model hits an unusually long document and asks for more memory than its slice allows, it does not just fail on its own. Instead, it takes OCR, embedding, and reranking with it.

This explains why one model per GPU has not been a hardware requirement but a path of least resistance through a stack where every tool was built to serve one model well, and none of them was built to share.

Sharing needs a different kind of server. One server that holds all four models, instead of four servers holding one each.

• It owns the card’s memory, so nothing has to be split by hand.
• It sees every request, so it can decide what runs first.
• And it loads a model when a request needs it, then drops it when memory runs short, so a model nobody is calling is not sitting on memory it does not need.

You do not get that by running four servers side by side. This requires an entirely different stack.

The three ways teams try to avoid this

At this point, one common reaction teams follow is to not to run GPUs the naive way. There are three common escape routes, and each one trades the problem for a different one.

• It is a good way to start, but catches fast when you have got scale.
• The bill increases in a straight line with usage while the work stays routine.
• You also run whatever the provider hosts, so the model you fine-tuned for your own task usually cannot go there, and your prompts and documents leave your environment on every call.
• This fixes the idle-time problem on paper.
• The problem is the cold start. A model that has scaled to zero has to pull gigabytes of weights into memory before it can answer, which adds seconds to the first request, and a reranker in the middle of a search cannot spend seconds waking up.
• You avoid that by keeping instances warm, and now you are paying for idle GPUs again, which is the exact thing we came here to escape.
• This is the path most serious teams end up on.
• So you self-host, which requires you to reach for the standard serving tools. And that is where the infrastructure problem gets specific. Let’s understand more problems about them below.

The third is the one that is actually right for cost and for control. You self-host the models on GPUs you rent, in your own environment, where the data stays inside your walls, and the bill is a flat hourly rate rather than a per-token meter.

The second is serverless GPUs, where you deploy your own model and pay only while it runs.

The first is a managed API. Invoke OpenAI or Cohere to avoid thinking about GPUs at all.

Problems with standard serving tools in a multi-model pipeline

The tools devs use here were built for a different job than the one you have, and it shows up in two ways.

Start with the fact that each tool serves one shape of model.

• vLLM is built to serve LLMs.
• TEI is built to serve embedding and reranking models.
• Neither one does OCR, vision, or document parsing.

This is not a gap in the models themselves. The models for every one of these tasks already exist and are excellent, like docling and PaddleOCR for documents, SigLIP and Florence for vision, GLiNER for extraction.

What does not exist is a single server that runs all of them.

So when the agent needs to parse a document and embed it and rerank and check a policy, you need vLLM for one part, TEI for another, and some FastAPI wrapper somebody wrote for the rest.

This gives you four servers + four APIs + four deployments, all behind one agent.

Now, notice what this setup does to the GPU by going back to the reason each server holds a whole card.

vLLM will occupy 90% of the GPU at startup. TEI will grab its part. The OCR wrapper also needs something.

Each was written on the assumption that it owns the card, and none of them will give memory back to the others, because none of them knows the others exist.

There is no shared queue across them and no way to say that the live search matters more than the background indexing job. We already walked through what that looks like on one card, and it does not get better when the processes come from four different tools instead of four copies of one.

You moved to small models to save money.

But the discussion so far proves that the savings will only show up if you can put several of them on one GPU.

The standard tools cannot make that happen since they force one model per server, and one server per card, which puts you right back on one GPU per model.

The fragmentation in tooling re-creates the exact waste we were trying to avoid.

What should the serving stack for small models look like

Let’s step back from the tools and understand the things we need the ideal tooling to do.

• It should run every kind of model an agent needs, embeddings, reranking, OCR, vision, extraction, and generation, behind one API. That is the only way to get off four servers and onto one.

• It should fill the GPU instead of padding it. And that turns out to require more than a good scheduler, because to pack requests of different lengths into one pass without wasting compute, the engine has to control the batching and the attention path for every architecture it serves.
• It should load and evict models as traffic moves, keeping the busy ones loaded and letting the idle ones move out, so that a model nobody is calling is not holding memory the way a cold serverless worker or a padded vLLM instance does.

• It should ship the production layer with routing, autoscaling, monitoring, and GPU pools. This is important since a bare runtime like vLLM is only the engine. On its own, it cannot spread load across replicas, add or remove GPUs as traffic changes, or place models across a mixed fleet of cheap and large GPU cards. Today, devs need to write all of that themselves, which explains why self-hosting is more work than it sounds.
• And adding a new model should be a config change, not a redeployment.

All this is hard with months of engineering since different model families do not run the same way underneath.

• A BERT and a Qwen handle positions and attention differently.
• A ColBERT returns a vector for every token.
• A reranker returns a single score and no vector at all.

Building one engine that can hold all of those shapes, and pack any of them into a full batch, is critical work, and it is the reason this did not already exist as an open-source package.

Open-source solution: Superlinked Inference Engine

The solution to all of this is actually implemented in the open-source Superlinked Inference Engine (SIE):

It runs as one cluster inside your own cloud, and it’s built for exactly the kind of pipeline we have been describing, i.e., several small models of different kinds running back to back on shared GPUs.

• It runs every kind of model through one API with four calls, so document parsing, vision, content-safety checks, and open-model generation all run on the same server:
  • encode turns text or images into vectors
  • score reranks a query against a set of documents (returns a number)
  • extract pulls structured fields, entities, or markdown out of a raw document (return span of text)
  • and generate runs an open LLM on a prompt (returns text and token usage)
  • TEI can hand back vectors and relevance scores, but one TEI process serves exactly one model, so embedding plus reranking already means two separate deployments. And TEI does not cover extraction, OCR, vision, or generation, so each of those still needs a server of its own.
  • A single SIE cluster can serve all four shapes in one server.
  • Even inside a full batch, there is a second kind of waste. Requests come in different lengths, and the simple way to run them together is to pad the short ones out to match the longest, then process the padding as if it were real.
  • SIE batches by compute cost instead of item count. It groups requests of similar length and pads only to the longest item in that batch, never to a global maximum, so almost none of the card’s time goes to padding.
  • One key difference is where the batch gets formed. Standard stacks route each request to a worker first, and every worker batches only the slice of traffic it happens to receive, so queues run unevenly, and mixed sizes pack poorly.
  • SIE inverts this order. The gateway publishes all work into one shared queue, and each worker pulls from that pool and forms a full batch before touching the GPU.
  • A model loads into GPU memory the first time a request needs it, and the least recently used one is dropped when memory runs short, the same idea as a browser cache.
  • This is also the concrete answer to the reserved-but-idle problem. Several public vLLM issues report that models reserve about 80% percent of a card’s memory on load, while utilization during embedding stays under forty percent.
  • Optimized load-and-eviction strategies let you run several models in the whole pipeline.
• It comes with the production layer already built.
  • A gateway sits in front and routes each request to a worker, and when there is a burst, it accepts the request and holds it while more workers start up.
  • The workers scale up under load and back down to zero when traffic drops, so you are not paying for idle GPUs.
  • It spreads work across whatever cards you give it and comes with the dashboards to watch it and the Terraform to deploy it on AWS or GCP, so the same setup runs on your laptop and on a full cluster.
  • It can also run fully offline, with the model weights downloaded ahead of time, for environments where data cannot leave.

It loads and evicts models based on traffic:

It packs those requests without wasting space.

• With vLLM or Triton, each model has to be tuned by hand, meaning the batch size, memory fraction, precision, maximum sequence length, and attention implementation all set for that specific architecture on that specific GPU.
• In SIE, every one of the 85+ models in the catalog carries a config file with those values already filled in, so you reference a model by name, and the engine loads it with settings known to work in production.
• Adding or swapping just requires pointing to a different config rather than starting the tuning over.

Every model is tuned and ready-to-use.

Implementation

Below, let’s look at the pipeline that involves three calls to one server.

We'll build a small retrieval pipeline that embeds some text, reranks it, pulls named entity fields out of the context, and then generates the final answer, all through a single SIE server.

This will involve four different model architectures, all hitting one endpoint. The first three calls run on CPU, so you can follow along on a laptop.

The generation step is the one exception, since it runs on SGLang underneath, which needs a GPU.

Start by installing and starting the server. The local extra pulls the CPU runtime, and serve command starts the server on port 8080.

pip install "sie-server[local]"
sie-server serve

It is ready when the health check answers.

curl http://localhost:8080/readyz     # ok

Now, instantiate the client and point it at the running server:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

query = "Who issued the invoice and what is the amount due?"
passages = [
    "Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.",
    "The total amount due is $18,240, payable within 30 days.",
    "The cafeteria menu changes every Monday and Thursday.",
]

Every call in the rest of the pipeline goes through this one object and this one endpoint, no matter which model it hits.

First we invoke encode.

This turns each passage into a dense vector using an embedding model.

docs = client.encode(
    "sentence-transformers/all-MiniLM-L6-v2",
    [Item(text=p) for p in passages],
)

print(docs[0]["dense"].shape)
# (384,)

Passing a list gives you back a list of results, one per passage, and each carries its vector under the dense key.

Next, we invoke score that uses a cross-encoder to read the query against each passage together and returns a relevance score for each one, a different kind of output than the vectors above:

from pprint import pprint

scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text=query),
    [Item(id=str(i), text=p) for i, p in enumerate(passages)],
)
pprint(scores["scores"])
# [{'item_id': '0', 'rank': 0, 'score': 3.009},
#  {'item_id': '1', 'rank': 1, 'score': -1.061},
#  {'item_id': '2', 'rank': 2, 'score': -11.425}]

The scores are cross-encoder logits, so the absolute numbers do not mean much on their own. Instead, the order is important. The invoice line scores above the payment terms, and the cafeteria has the lowest score, as expected.

The results come back already ranked, so the top passage is the best match:

best_id = scores["scores"][0]["item_id"]
best = passages[int(best_id)]
# 'Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.'

Finally, we invoke extract which runs an extraction model to generate the fields we ask for, returning each as a labelled span with a confidence score.

result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=best),
    labels=["organization", "date", "amount"],
)
print(result["entities"])

[
  {
    'text': 'Northwind Logistics', 'label': 'organization', 
    'score': 0.979, 'start': 31, 'end': 50, 'bbox': None
  },
  {
    'text': '14 March 2026', 'label': 'date',
    'score': 0.951, 'start': 54, 'end': 67, 'bbox': None
  }
]

This is a third architecture that has a different output shape, but is invoked via the same client and the same server.

Finally, we invoke generate to compose the answer from the top-ranked context. This call needs the GPU build of the server, since generation runs on SGLang, which has no CPU path. Everything else about the call is identical.

context = " ".join(
    passages[int(s["item_id"])] for s in scores["scores"][:2]
)

answer = client.generate(
    "Qwen/Qwen3-0.6B",
    f"Answer in one sentence using only this context.\n\n"
    f"Context: {context}\n\nQuestion: {query}",
    max_new_tokens=64,
)

print(answer["text"])
# 'The invoice was issued by Northwind Logistics, and the amount due is $18,240.'

print(answer["usage"])
# {'prompt_tokens': 61, 'completion_tokens': 17, 'total_tokens': 78}

This is a fourth output shape, free text with token usage, produced by a generative model, and it comes back through the same client as the vectors, the scores, and the spans.

That is the whole pipeline involving four model families with four different output types, running under one server and client, with nothing to deploy beyond the single process started at the top.

Each model is pulled from Hugging Face on its first call and stays active for the next request.

Using small, specialized models for the narrow tasks is the right approach, and it is not really controversial. They are accurate enough on the task they are trained for, and they keep your data in your own environment.

But switching to small models does not make inference cheaper by itself. It moves the cost from a per-token bill to the GPUs you rent, and if you put each model on its own GPU, most of that hardware sits idle, and you are paying for it.

The saving only appears when the models share GPUs, and that requires a single engine that can run all of them, because the moment you serve each model with a different tool, each one takes a GPU of its own again.

SIE (open-source) already implements that, and it lets you run the different models an agent needs on one shared cluster, keeps the GPUs busy instead of idle, and includes the routing and autoscaling you would otherwise have to build.

It also plugs into the retrieval stack you already run, with integrations for Chroma, Qdrant, Weaviate, and LanceDB.

You can find the repo here: github.com/superlinked/sie

(don’t forget to star it ⭐️)

👉 Over to you, how many separate models does a single request in your stack call, and how many of them are holding a dedicated GPU right now?

Good day!

Andrej Karpathy summarized the entire history of LLM training in three nouns:

• text
• conversations
• and environments

Pretraining ran on internet text, supervised fine-tuning ran on curated conversations, and the current era of reinforcement learning runs on environments.

OpenAI’s o1 proved that framing by training on math and coding problems with verifiable answers, and DeepSeek-R1 published the recipe openly.

The industry now treats environments as the scarce resource. Anthropic reportedly discussed spending over $1 billion on them in a single year.

Meanwhile, someone is giving them away for free through a Hugging Face-style hub hosting 2,500+ open-source environments.

And today, we’ll use their framework to build one of our own from the ground up.

Before we build anything, let’s understand what an environment actually is. Every RL setup is a loop with four moving parts, and they map onto anything the model interacts with:

• State → what the model sees right now
• Action → the choice it makes from that state
• Reward → a number that says how good that choice was
• Environment → the thing that holds the state, accepts the action, and returns the reward

The hard part of this loop was always the reward, which traditionally required training a separate model on human preferences.

DeepSeek-R1 replaced that entire model with a plain Python function through GRPO (Group Relative Policy Optimization). Score several answers to the same prompt and push the model toward the ones that beat the group average.

Here is an illustration of how GRPO works:

That leaves the environment as the barrier still standing. A reward function can score a final state, but something has to present the task, accept the model’s actions, enforce the rules, and produce that final state in the first place.

Frontier labs guard this part as a proprietary asset, which is why few practitioners have ever seen a working environment from the inside.

That team giving environments away is Prime Intellect, and their library, Verifiers, is the framework we’ll build with (100% open-source).

The design is model-agnostic, the rewards are fully verifiable, and the same skeleton works for any turn-based task you want to train on.

How to read this issue

The goal here is simple. We’re building a complete RL environment, and we’re covering it in a way that stays easy to digest even if this is your first one.

Every idea comes with the code that implements it, kept short enough to read in the flow of the article. The complete, runnable version is shared at the end.

And the game we’ll build around is just the working example. By the time you finish, you’ll be able to take this exact structure and adapt it to your own use case.

The Othello RL env: Why a board game

Othello is a two-player game on an 8x8 grid. You place a disc so it traps opponent discs in a straight line, every trapped disc flips to your color, and whoever owns more discs when the board fills up wins.

That flipping rule is why a single move can swing the score hard, and why corners matter so much (a corner disc can never be flipped back).

Every RL part has a home here. The board is the state, the disc placement is the action, the final result feeds the reward, and the game engine that validates moves, flips discs, and plays the other side is the environment.

The LLM plays Black and a built-in engine plays White. After each Black move, White responds, the updated board goes back to the model, and this repeats until the game ends.

Tech stack

Three tools make this work, each handling one layer:

• Verifiers: the RL framework. It defines the environment, runs the turn loop, and handles evals.
• Lightning AI: an OpenAI-compatible inference API, so the same code calls hosted models like Claude or DeepSeek without provider-specific rewrites.
• vLLM: serves open-weight models locally behind that same OpenAI-compatible endpoint.

The shared interface is what makes the environment model-agnostic. A local Ministral-3B and a hosted GPT-4.1 swap in one line, and you change the model name, nothing else.

The game loop

Here’s what the model sees on each turn:

The board, the score, and the list of valid moves are the entire state. Everything the model knows about the game comes from this text.

It must respond with a <think> section followed by a <move> tag, which forces it to reason through the position before committing to a move.

The environment validates the move against the valid list, applies it to the board, lets White respond, and sends the updated board back. An invalid move gets an error message and a retry, with a penalty applied to the reward.

All of that lives in one method. OthelloEnv inherits from MultiTurnEnv in Verifiers, which handles the loop, turn tracking, and termination, and the base class calls your env_response every time the model sends a move:

class OthelloEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        move = parse_move(model_output)

        if not is_valid(move, state.board):
            state.penalty += INVALID_MOVE_PENALTY
            return error_message(move), state          # same turn, retry

        state.board = apply_move(state.board, move, player="black")

        if not game_over(state.board):
            white_move = opponent_engine(state.board, state.difficulty)
            state.board = apply_move(state.board, white_move, player="white")

        return render_board(state.board), state

This is the shape of the logic; the real version also handles board parsing and edge cases that the snippet skips. I’ll share the full working code later once we understand the setup end-to-end.

The built-in opponent engine

The built-in engine plays white and has two modes:

• Random picks any legal move. Useful for early training, since the model can win just by making reasonable decisions.
• Minimax simulates future moves, scores each resulting position by corner control, board position, and available moves, and picks the move with the best worst-case outcome. Depth-3 means it looks three moves ahead, enough to set traps and avoid obvious blunders.

A randomness parameter controls how often White ignores its strategy and picks randomly instead. Lower randomness means more consistent, punishing play.

White’s moves are generated from the current board state and a fixed game seed, so the same position always produces the same response. Determinism lets you compare different models under identical conditions.

Here’s how the code for the same looks like:

def opponent_engine(board, randomness, depth):
    if random.random() < randomness:               # random mode
        return random.choice(legal_moves(board))
    best_move = None                                 # minimax mode: try every move,
    best_score = float("-inf")                       # keep the one with the best score
    for move in legal_moves(board):
        score = minimax(apply_move(board, move), depth - 1, my_turn=False)
        if score > best_score:
            best_move, best_score = move, score
    return best_move

The reward functions

At game end, four signals combine into a single reward:

Each one is a function that reads something off the final state and returns a number, and the win signal is the simplest:

def win_reward_func(state):
    result = state.get("result")
    if result == "black":   # the model's color
        return 1.0
    if result == "draw":
        return 0.5
    return 0.0              # loss, or game never finished

The other three follow the same shape, and all four combine into one score:

def total_reward(state):
    return (
        win_loss_score(state)
        + piece_advantage(state)
        + format_compliance(state)
        - invalid_move_penalty(state)
    )

The reason for four signals instead of a single win/loss bit is resolution. Early in training, most games are losses, and to a pure win/loss reward, they all look identical, so the model has nothing to optimize.

• Piece advantage separates a close loss from a blowout, giving the model a gradient before it starts winning.
• Format compliance carries a low weight, so clean formatting never outweighs good play.
• The invalid move penalty is capped, so one broken game can’t drown out everything the model did right.

Every score comes directly from the game state and rules, with no judge model or LLM evaluator involved, so the reward is fully deterministic and reproducible.

Wiring it together

The environment generates games across different starting positions and opponent difficulties, the model plays each one through the loop above, and at game end the four rewards are computed and combined. During evaluation, rewards, token usage, and turn counts are aggregated across all games into a results table.

One function wires everything up, and it’s what the prime eval command calls behind the scenes:

def load_environment(min_random_move_prob, max_random_move_prob, parse_think):
    dataset = generate_games(min_random_move_prob, max_random_move_prob)
    parser = XMLParser(fields=["think", "move"])
    rubric = Rubric(funcs=[...], weights=[1.0, 1.0, 0.2, 1.0])
    return OthelloEnv(dataset, parser, rubric)

What the numbers show

Running an evaluation is one command, with the opponent settings passed as arguments:

prime eval run othello -m openai/gpt-4.1 -n 100 \
  -a '{"min_random_move_prob": 0.0, "max_random_move_prob": 0.0, "minimax_depth": 3}'

Swap the model name to test something else, whether it’s hosted through Lightning AI or running on a local vLLM server. Here’s how two models fared against two opponents:

From evaluation to training

Evaluation tells you where the model struggles, and training fixes it in three stages. The same environment supports all of them:

Data generation: have your strongest model play a few hundred games and save the results. The same eval command writes straight to a dataset:

prime eval run othello -m openai/gpt-4.1 -n 200 \
  --save-to-hf-hub --hf-hub-dataset-name your-username/othello-data

Filter it down to wins and draws before training on it, so you’re not teaching the model your strongest player’s mistakes alongside its format.

Supervised fine-tuning: teach format and valid moves first. The Ministral-3B turn counts make the point directly, since unreliable formatting and illegal moves are noise RL can’t train through.

RL training: this is where strategy improves. Multiple games are played from the same starting position, each is scored with the same reward functions from evaluation, and the model is updated toward the higher-scoring rollouts.

Stripped to its core, that’s the GRPO loop from the top of the article:

for prompt in batch:
    rollouts = [play_game(model, prompt) for _ in range(group_size)]
    rewards = [total_reward(r.final_state) for r in rollouts]
    advantage = rewards - mean(rewards)     # relative to the group
    update_model(model, rollouts, advantage)

Each rollout is scored against the average of its own group, so a move that wins 6 of 10 games from a given position is rewarded more than the weaker attempts beside it.

Adapting this to your own task

Once you remove the Othello specifics and the same MultiTurnEnv gives you a skeleton that fits any turn-based task:

class TaskEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        action = parse_action(model_output)          # your task's syntax
        if not is_valid(action, state):
            return error_message(action), penalize(state)
        state = apply_action(state, action)          # your task's rules
        if not task_complete(state):
            state = environment_step(state)          # tool, API call, or opponent
        return render_state(state), state            # your task's display

This isn’t specific to games. A coding agent swaps apply_action for running a test suite against generated code, a support agent swaps it for checking whether a tool call retrieved the right record, and a research task swaps it for verifying a claim against a cited source.

Adapting it comes down to four swaps:

• Task logic: your domain’s rules for what counts as a valid action and how the state changes
• Response engine: whatever the model reacts to, from a rule-based simulator to a live API to another model
• Reward functions: keep the pattern (outcome signal, partial credit, format, penalty) and replace the domain logic
• State rendering: whatever the model needs to see, whether that’s a file diff, a conversation transcript, or a tool’s response

The structure underneath stays the same regardless of domain: parse, validate, apply, respond, score. The rubric is the design, and if you get the components right, the training signal takes care of itself.

Try it yourself

All the code, setup instructions, and ready-to-use GPUs to reproduce these results are in the Lightning AI Studio template:

​Build a Custom RL Environment →​

​Check out Lightning AI Inference →​

Good day!

Recap

In Chapter 11, we solved the problem that RLVR left open. Verifiers only exist for tasks with an answer key and most agent work has no such key. So we asked where the reward comes from when the environment stays silent.

We started with an observation about GRPO itself. The optimizer consumes one number per response and normalizes within the group. It never inspects where that number came from. The reward source is an open slot in the loop.

We then drew the line between verifiable and non-verifiable tasks and understood that verifiability is a property of the outcome.

We examined the obvious fix, hand-written reward functions, and saw why they are brittle. Every threshold is a knob we must set ourselves.

The way out was the LLM-as-a-judge. Two properties made it fit GRPO almost perfectly. Relative scoring is reliable where absolute scoring is not. And GRPO only needs relative scores anyway. Scoring the whole group in one judge call turned evaluation into comparison.

Finally, we built a from-scratch judge scorer on a RAG example.

If you have not read Chapter 11, we recommend doing so first:

The Reward Signal Problem for Agents

RL Part 11: From verifiable rewards to LLM-as-a-judge.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will assemble the full training loop for multi-step agents. We'll define what an RL environment means for an LLM agent, and formalize the trajectory as the training unit. Then we'll run the complete rollout, score, and update cycle using OpenPipe's ART (Agent Reinforcement Trainer) framework and its RULER reward function.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

From one judged group to a training loop

Look closely at what Chapter 11 actually trained on. Each candidate in the judged group was a single response. One prompt went in, one completion came out, and the judge compared the completions. That shape matches a chatbot but not truly an agent.

An agent does not produce one completion. It reads a goal, calls a tool, reads the result, and calls another tool. Only after several steps does it produce the result.

This creates two gaps that Chapter 11 never closed.

First, we never defined where the agent acts. A multi-step agent needs tools, state, and a rule for when the episode ends.

Second, we never showed how multi-step interactions become training data. GRPO updates the policy from groups of scored samples. So what exactly is a sample when the agent acted eight times?

Closing the above gaps is the destination for this chapter.

The agentic setting

We need to say precisely what "the agent acts over many steps" means. The good news is that we already own the vocabulary. Chapter 2 gave us states, actions, and rewards. We only need to map them onto the new setting.

An agent episode looks like this. The agent receives a goal in natural language. It observes its situation. It acts, by emitting either a tool call or a message. The environment responds with an observation, such as a tool result. The agent acts again. This repeats until the episode ends, usually when the agent produces a final answer or hits a turn limit.

Each piece maps onto our old language:

• Action: one model output, typically a tool call or a message.
• Observation: what comes back, typically a tool result.
• State: everything the agent knows so far. For an LLM agent, that is the interaction history.
• Reward: a score for the episode. It usually arrives once, at the end.

Let us make the policy precise. The policy chooses the next action given that history. Now, one episode, run start to finish, produces a trajectory. We will write it as $\tau$. It is the full sequence of the goal, all actions, all observations, and the final answer. The training objective is then stated over whole trajectories:

Here,

• $J(\theta)$ is the quantity we want to maximize, as a function of the policy parameters $\theta$.
• The symbol $\tau$ is one complete trajectory. The notation $\tau \sim \pi_\theta$ says trajectories are generated by running the current policy in the environment.
• $R(\tau)$ is the reward assigned to the whole trajectory.
• The expectation $\mathbb{E}$ averages over the randomness in both the policy's sampling and the environment.

The intuition is the same one that has carried all along. Make good trajectories more likely and bad ones less likely. GRPO will estimate this "good versus bad" comparison, exactly as it did in Chapters 10 and 11. Nothing about the optimizer changes. What changes is the object being scored, from a response to a trajectory.

In summary, the agentic setting is our familiar MDP with three twists. The state is the interaction history, the setting is partially observable, and the reward is sparse and trajectory-level. The objective maximizes expected reward over whole trajectories. Everything else in this chapter builds the machinery to optimize it.

RL environments for agents

The word "environment" has followed us since the start. Gridworld was an environment, cliff walking was an environment, similarly cart pole, lunar lander, etc. Now let's see what the word means when the agent is an LLM.

An agentic RL environment is the world the agent is in. Concretely, it must supply four things:

Researchers at Stanford studied how AI agents actually spend their inference budgets.

One key finding was that ~62% of what gets sent to an agent on every call is just repeated content, i.e., the same system prompts, tool definitions, and documents, which are fed in again and again.

So every time the agent takes a single step, you hand it everything from scratch, even if it just processed the exact same info one turn ago:

Per-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from $30/M to $0.40/M tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context.

So even though each token got cheaper, the total bill went up, since volume outran the price cuts.

Uber shared a similar story recently. After rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in just 4 months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.

The industry is optimizing the wrong variable. Making tokens cheaper doesn’t help if most of those tokens shouldn’t exist in the first place.

So, in this article, we’ll learn about a new open-source architecture called LMCache that moves cache management out of the inference engine entirely.

Teams running it are seeing up to 14x faster time-to-first-token, so understanding it now puts you ahead of nearly everyone running inference today.

What happens when we prompt a model in a naive setup

Every time you prompt a model, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These vectors capture how the model understands each token’s relationship to every other token in the context.

This collection of K and V vectors is called the KV cache, and the computation scales quadratically with input length.

One MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.

The KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.

Think of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding.

What prefix caching solves and what it doesn’t

The industry noticed the above problem and built a technique called prompt caching to deal with it.

If two consecutive requests share the same opening tokens (the “prefix”), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what’s new.

This is incredibly helpful. Anthropic's own implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.

But prefix caching has a hard ceiling.

The cached portion must be an exact, byte-for-byte prefix of the new request. If you change anything in the cached region (even a single character), it leads to a full cache miss, and it happens in three common scenarios:

• RAG with multiple documents → You cache document A alone and document B alone. If a new query needs both documents, the 2nd document’s cached KV state is invalid since it was computed without awareness of the first document.
• Document order changes → The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.
• Growing conversation history → Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.

Alibaba Cloud’s production data validates these limitations. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.

Prefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning, and many real-world workloads don’t look like that.

Another performance bottleneck with caching

Every KV cache library runs inside the inference engine’s process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources.

They can’t run at the same time, so when the engine is busy managing cache, it stops doing inference, and vice versa.

Google's TurboQuant shows this effect. It is a recent KV cache quantization technique that compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.

Cache management and inference serving are fundamentally different workloads.

One is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).

LMCache and the disaggregated approach

LMCache is an open-source project (10k+ stars) that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.

In practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache “here are the block IDs I need” (tiny messages, almost no data).

All the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache’s own process. The inference engine doesn’t even notice it’s happening.

This separation produces three benefits:

• No resource contention → Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.
• Zero-copy sharing across GPUs → In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.
• Multi-tier parallel loading → Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.

The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.

Also, LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.

Solving the prefix problem with CacheBlend

LMCache’s architecture solves the performance side of caching.

But recall another problem we discussed above, where a query needed 2 documents.

The LMCache team’s research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.

The observation is that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.

CacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.

This gives 2 to 4x faster processing for multi-document queries (the kind you see in RAG apps) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.

Using LMCache in production

LMCache isn’t a research prototype but rather ships with the infrastructure that production teams expect.

• Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.
• Kubernetes operator for deployment
• CLI for debugging and benchmarking.

If the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn’t start cold.

If LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled, but inference continues normally, and it reconnects automatically when the cache process recovers.

Neither failure takes the whole system down.

You can find the LMCache GitHub repo here →

(don’t forget to star it ⭐️)

Good day!

Brian Armstrong from Coinbase said humans shouldn’t be the ones choosing which model answers a request. Model selection is a job for AI.

Coinbase is already running on that belief. They built a routing layer in front of every request.

LLM routing sits between the app and model providers, reads each prompt, classifies the task, and forwards it to the right model automatically.

Coinbase used it to match each prompt to the right model and default to cheaper options where the task allowed it. Cache hit rate went from 5% to 60% and spend ended up nearly cut in half while usage kept growing.

Plano (here’s the open-source GitHub repo) is an open-source implementation of exactly that routing layer, the one that decides which model answers which request, without your application code ever specifying one.

Plano GitHub Repo

The way it works is that you define routing preferences in plain English, and Plano orchestrator infers the domain and action of each incoming prompt, matches it to the right preference, and picks the model automatically.

Next, let’s set up Plano in front of Hermes Agent, and watch it route requests to different models based on what each prompt actually needs.

Before the setup, let’s understand one key concept.

The problem with routing

When an agent runs a task, it doesn’t send a single request. One user prompt triggers several LLM calls in sequence through tool use, each building on what came before.

With a hardcoded model, that’s never a problem. But once routing is in place, each of those calls gets classified independently.

Consider an agent debugging a function. It makes a planning call, runs a tool, reads the output, then makes another call to analyze what came back.

Call 1 routes to claude-sonnet-4.5 for code generation. Call 4 gets classified as analysis and routes to gpt-4o-mini instead.

Two problems occur when that happens.

• The model on call 4 has no memory of decisions made in call 1.
• And every model switch invalidates the cache, so you’re paying full token rate again from scratch.

Plano solves this with model affinity.

The first call in a session routes normally, and Plano pins that model to a session ID passed in a header.

Every subsequent call with the same ID goes to the same model regardless of how it gets classified.

import uuid
from openai import OpenAI

client = OpenAI(base_url="http://localhost:12000/v1", api_key="EMPTY")
affinity_id = str(uuid.uuid4())

response = client.chat.completions.create(
    messages=messages,
    tools=tools,
    extra_headers={"X-Model-Affinity": affinity_id}
)

The pin lasts 10 minutes by default.

When the task changes, you generate a new ID, and routing starts fresh. For multi-replica deployments, backing it with Redis keeps the pin shared across instances:

routing:
  session_ttl_seconds: 600
  session_cache:
    type: redis
    url: redis://localhost:6379

Model affinity handles consistency within a task. Now let’s build the routing layer using Plano.

Setup

Plano runs as a local proxy on your local machine. The routing config, model providers, and listener settings…everything is defined in one YAML file.

Plano supports three routing methods:

• Model-based: the client specifies an exact model name on every call.
• Alias-based: you define a friendly name like fast-model or reasoning-model that maps to a real model behind it, so your app never hardcodes a provider, and swapping models is one line in the config.
• Preference-aligned: the client specifies no model at all. Plano-Orchestrator infers the domain and action of each prompt and picks the right model automatically based on plain-English descriptions you define.

For alias-based, the config looks like this:

model_aliases:
  fast-model:
    target: gpt-4o-mini
  reasoning-model:
    target: claude-sonnet-4.5

We’ll use preference-aligned routing in this walkthrough.

First, we install Plano. Also, we’re using OpenRouter as our model provider, so one key gives us access to every model we need without managing separate API keys:

pip install planoai
export OPENROUTER_API_KEY=sk-or-...

Note: OpenRouter is one option. Plano works with any OpenAI-compatible endpoint, so you can point it at a self-hosted model running on Ollama or vLLM just by swapping the base_url.

Then we create the config.

The routing preferences tell Plano which tasks go where. For complex reasoning and code, we use claude-sonnet-4.5, for everything routine gpt-4o-mini:

# plano_config.yaml
version: v0.4.0

listeners:
  - name: egress_traffic
    type: model
    address: 0.0.0.0
    port: 12000
    timeout: 30s

model_providers:
  - model: anthropic/claude-sonnet-4.5
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY

  - model: openai/gpt-4o-mini
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY
    default: true

routing_preferences:
  - name: code generation
    description: writing new code, implementing functions, or building features from scratch
    models:
      - anthropic/claude-sonnet-4.5

  - name: complex reasoning
    description: deep analysis, architectural decisions, debugging hard problems, multi-step reasoning
    models:
      - anthropic/claude-sonnet-4.5

  - name: general tasks
    description: answering questions, explaining concepts, quick lookups, casual conversation
    models:
      - openai/gpt-4o-mini
tracing:
  random_sampling: 100
  opentracing_grpc_endpoint: http://localhost:4317

Plano reads every incoming prompt and routes it to the right model based on these preferences.

Pointing Hermes at Plano

Hermes connects to any OpenAI-compatible endpoint, so we point it at Plano instead of a provider directly:

First, we install Hermes:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

Then we configure it to use Plano as its endpoint:

hermes model
# Select "Custom endpoint (self-hosted / VLLM / etc.)"
# Enter URL: http://localhost:12000/v1
# Skip API key

From here, every prompt Hermes sends goes to Plano, which classifies it and forwards it to the right model through OpenRouter. Hermes just got a completion back.

We start Plano first, so the proxy is ready before Hermes begins sending requests:

planoai up plano_config.yaml
hermes

Hermes is now sending every request through Plano instead of directly to a provider.

Routing in practice

Now that both Plano and Hermes are up, we send two different kinds of requests.

First, a simple question:

> what does the __init__ method do in Python?

Plano classifies this as “general tasks” and routes it to gpt-4o-mini. Fast, cheap, more than capable enough for a conceptual question.

Then something that actually needs horsepower:

> implement a rate limiter middleware for a FastAPI app using Redis, with per-user and per-endpoint limits

Plano classifies this as “code generation” and routes it to claude-sonnet-4.5.

The Hermes session is hitting the same endpoint, but it’s picking a different model automatically based on what the prompt actually needed.

We wrote zero branching logic to make that happen. The routing decision lived in the YAML, not in the agent.

That covers routing by intent. The next case is when two models can both handle a task equally well, and the only difference is price.

Routing to the cheapest model that qualifies

When two models can both handle a task equally well, we can let Plano pick based on live pricing instead of the order we wrote them in.

routing_preferences:
  - name: code review
    description: reviewing, analyzing, and suggesting improvements to existing code
    models:
      - anthropic/claude-sonnet-4.5
      - openai/gpt-4o
    selection_policy:
      prefer: cheapest

Plano reads a pricing catalog before routing and reorders the candidate list by total token cost.

The cheaper model answers, but the expensive one stays as a fallback if the cheaper one fails.

Plano also ships an observability console, planoai obs, which shows a per-request USD cost column so we can see exactly which model answered each request and what it cost.

Model agility, without the rewrite

Armstrong's team built their routing layer from scratch, inside custom harnesses with engineers assigned to own it.

Most teams are making the same decisions without any of that infrastructure

Plano gives you three things out of the box without writing a single line of routing logic.

• Preference-aligned routing automatically sent complex tasks to claude-sonnet-4.5 and routine ones to gpt-4o-mini.
• Cost-aware selection reached for the cheaper option when two models could both handle the job.
• Model affinity kept the same model pinned across every turn of a multi-step task, so nothing broke mid-session.

All of it is configured in YAML instead of inside your coding agent.

When a cheaper model drops for routine tasks, you update one line. When a better one beats Claude on code, you point the preference at it. The agent never changes.

The routing decision just moves.

Plano’s GitHub repo →

(Don’t forget to star it ⭐️)

Good day!

Recap

In Chapter 10, we resolved the tension that closed Chapter 9. Some tasks do not need a learned reward model at all. When correctness is a matter of fact, a verifier supplies the reward directly. We built this idea into Reinforcement Learning with Verifiable Rewards (RLVR).

We then explored how the critic becomes optional. Its job was to estimate the expected reward, so we could tell whether a reward was better than average. GRPO does this without a network. It samples a group of responses to the same prompt and averages their rewards. That average becomes the baseline.

Then we understood the group-relative advantage. It measures how much better a reward is than the group's average, scaled by how spread out the rewards are.

We plugged this advantage into PPO's clipped objective and controlled drift with a KL penalty, estimated cheaply through the k3 estimator.

Finally, the payoff was the collapse from four models to two. The verifier replaced the reward model (in RLVR setting), the group baseline replaced the critic, and only the policy and reference remained.

If you have not read Chapter 10, we recommend doing so first:

Verifiable Rewards and GRPO

RL Part 10: Dropping two models from the four-model pipeline, and building rewards you can trust.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will explore where the reward comes from when there is no answer key. We will look at why hand-written reward functions are brittle, how research and industry approach this problem, and the central idea that makes it tractable, using an LLM-as-a-judge.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

GRPO does not care where the number comes from

The first thing to notice about GRPO is how little it asks for. It needs one number for each response in a group, nothing more.

Look back at the algorithm from Chapter 10. The policy generates a group of responses. Something assigns a reward to each. GRPO subtracts the group mean and divides by the standard deviation. That value becomes the training signal.

At no point does GRPO inspect where the reward came from. It could come from a math checker, a compiler, a human rating, or a Python script. To the optimizer, all of these are identical. They are just a list of scalars.

It helps to picture the same GRPO loop with different reward sources plugged into one slot. With a math verifier, the slot holds a calculator. With a compiler, it holds a test runner. With human feedback, it holds a panel of raters. With a script, it holds hand-written Python.

The loop around the slot is identical in every case, only the contents of the slot change.

In Chapter 10 that slot held a verifier, and it was easy. The environment handed us a clean signal at almost no cost. But the moment we step outside tasks that can be scored with a verifier, the empty slot in the loop becomes the hardest part of the system.

In summary, GRPO consumes a number per response and normalizes within the group. The source of that number is left entirely open. That open slot is where this chapter lives.

Verifiable and non-verifiable tasks

To see why the open slot is hard to fill, we need a clear line between tasks where a verifier exists and tasks where one does not.

A verifiable task is one whose outcome can be checked by a rule or a program:

• A math answer matches the key or it does not.
• A program passes the test suite or it fails.
• A SQL query returns the right rows or the wrong ones.

In each case, a deterministic check produces an exact verdict.

A non-verifiable task has no such check. Consider a RAG system, which retrieves documents and writes an answer grounded in them. There is no single correct answer to compare against. Many phrasings are valid, and no string match can tell a faithful answer from a plausible-sounding invented one.

The same gap appears across most real agent work. A customer support agent drafts a reply, and there is no compiler to run it through. A summarization agent condenses a twenty-page document, and there can be many good summaries.

Tool use sits in an interesting position here. Some tool-use tasks are fully verifiable. If an agent's job is to book a flight matching a set of constraints, you can check the final booking against those constraints. If its job is to query a database and return a row count, you can compare the count. The presence of tools does not block verification.

Other tool-use tasks are not verifiable at all. If an agent searches an inbox and writes a natural-language answer, there may be no single gold answer, and the quality of the writing matters as much as the facts. The tools are the same, but the outcome is what differs.

Dario Amodei said a one-person billion-dollar company has a 70-80% chance of showing up by 2026. He said it on stage at Anthropic’s developer conference last year.

Matthew Gallagher proved it by starting Medvi, an AI-run telehealth company, with $20,000 and no employees. It made $401 million in revenue in its first year.

Gallagher's stack was AI agents talking to each other. When two couldn't talk, he built more agents to handle the coordination. He tested the prompts himself and stepped in when something broke.

Alook (GitHub Repo) is an open-source, self-hosted platform that turns your coding agents into a real org chart.

The way it works is that you create an agent, assign it a role, and give it a real email inbox.

Agents email each other and update you. They run as actual Claude Code or OpenCode sessions on your machine, with full access to your tools.

An org chart gives every agent a defined role and a reporting line. They coordinate with each other without you relaying a single message or fixing the wiring yourself.

Let’s set up Alook from scratch, build a four-agent org chart, and watch what happens once a real job gets handed to it.

Setup

Alook runs as a daemon on your own machine, and this one command connects it:

npx @alook/app onboard

It detects whatever coding agent runtime is already installed, Claude Code or OpenCode, and deploys the agent company.

This opens a local dashboard at http://localhost:15210.

From there, you start from a blank org chart or one of Alook’s pre-built templates, whichever’s closest to what you’re building.

Each agent on the chart is a real Claude Code or OpenCode session running on your machine, with full access to the same tools you use, and a real @alook.ai inbox.

The inbox is the coordination layer. Agents email each other, the same way a team would, instead of passing data through a trigger you wired by hand.

Building the company

Competitive intelligence usually means someone checking a pricing page, copying numbers into a spreadsheet, and doing it again tomorrow.

We’re going to replace that with four agents who build a price tracker, run it on a schedule, and email you the moment something changes.

First, we create the agents one by one, assigning them distinct roles and claiming their real @alook.ai inboxes:

• Atlas (CEO) is the single point of contact for the human. It delegates tasks to Mara.
• Mara (PM) turns Atlas's briefs into specs and routes them to Theo or Ren. She's the sole router on the chart.
• Theo (engineer) builds and maintains the scrapers for competitive intelligence.
• Ren (Ops and Customer-facing) notifies the human when a tracked change is detected.

Once the agents are live, we wire the reporting hierarchy: Atlas to Mara, and Mara to Theo and Ren.

Theo and Ren never talk to each other, or to Atlas directly. They only go through Mara.

This setup avoids building a chaotic AI group chat where every agent talks over each other and loses context.

Theo’s job touches the competitor’s site tracks, so he needs a way to scrape it reliably and schedule.

So we have given it access to the Bright Data CLI, which lets it scrape any website, provision custom scrapers if needed…all while avoiding IP blocks and CAPTCHAs that hit any agent scraping at real scale.

To build a custom scraper over any website, you can describe the page in plain English, and it builds the scraper that returns structured data:

Now let’s see how our AI company handles the real task.

Running the company

As mentioned above, we don't need to manage every agent ourselves. Instead, we just talk to Atlas, the CEO, and let the org chart handle the rest.

We ask him to track pricing on railway.app/pricing.

Atlas replies in the chat, and behind that reply, he's briefing Mara over email, and the thread shows up in the same window:

Mara turns the brief into a spec and hands it to Theo.

The spec covers a scraper for the page, timestamped snapshots, change detection, a daily run, and a report we can actually read.

Theo acknowledges the spec over email, the same way Mara and Atlas did, then builds it using the Bright Data CLI and reports back once it's running.

This is the scraper Theo just built, sitting in Bright Data's dashboard:

It's a real custom scraper that the agent provisioned itself by understanding the specified website, not a one-off CLI call that disappears after running. You can trigger it manually from this same screen, or call it straight using the API:

Mara relays the build-up to Atlas, and Atlas informs us in the same chat:

The company runs without you

Once Theo confirms the build is live, the job isn’t done. The schedule still needs to run, and someone still needs to watch what it finds.

The agent adds the scraper to the company calendar as a recurring 9am task, on its own.

This is Ren’s job. He watches the tracker’s output, and the moment the price on the page actually changes, he sends a note.

The whole loop runs unattended.

We handed Atlas one brief, and the org provisioned a scraper, scheduled it, and kept someone watching the output, with no further input from us.

Try it yourself

Every agent in this walkthrough ran as Claude Code, but Codex and OpenCode work the same way, as Alook lets you bring your own agent and gives whichever one you pick a role, an inbox, and a runtime that stays on.

Every completed task builds context for the next one, so the agents aren’t relearning the company from scratch on every run.

Every email between them gets logged the same way, so you can read back exactly how a decision got made.

Here’s Alook’s GitHub repo →

And here’s the Bright Data CLI→

(don’t forget to star them 🌟)

Good day!