Intro

Every production LLM today generates text the same way, i.e., one token at a time, left to right. GPT-4, Claude, Gemini, DeepSeek, LLaMA. All of them use autoregressive (AR) generation, where each token depends on every token before it.

This creates two structural problems. First, it's slow in a way that better hardware can't fully fix. Second, it creates blind spots in reasoning that no amount of training data can patch.

Diffusion language models (dLLMs) take a fundamentally different approach. Instead of generating tokens sequentially, they start with a fully masked sequence and iteratively reveal all tokens in parallel, refining the output over multiple steps.

This gives a generation paradigm that's compute-efficient, bidirectional, and capable of things autoregressive models structurally can't do.

This article builds a complete understanding of how dLLMs work, from first principles to the math. By the end, you should be able to read any dLLM paper fluently, understand every component of the architecture, and reason about why this approach works.

Let's begin.

The sequential bottleneck in autoregressive generation

Autoregressive models factorize the probability of a text sequence as a product of conditional probabilities:

Each of these factors in the above mathematical formulation requires loading the full model weights through GPU memory to produce a single token.

To understand better, consider what happens during autoregressive decoding at the hardware level.

A 7B parameter model stored in FP16 precision occupies roughly 14GB of GPU memory.

To generate a single token, the GPU needs to read those 14GB of weights from its high-bandwidth memory (HBM) into its compute cores, perform the matrix multiplications and attention computations, and produce one output token. Then, for the next token, it reads all 14GB again.

The reading part is the bottleneck. An A100 GPU can read data from HBM at about 2 TB/s. Reading 14GB at that speed takes roughly 7 milliseconds. The actual math (all the matrix multiplies, the attention scores, the MLP layers) may be takes about 0.1 milliseconds.

So the GPU spends 7ms waiting for data that it then uses for a computation that takes 0.1ms. This means roughly 98% of the time spent on data transfer and only 2% on actual math.

A useful way to quantify this imbalance is arithmetic intensity, which is the ratio of compute operations (FLOPs) to bytes of data moved.

During Autoregressive decoding, this ratio is roughly 1 FLOP per byte. Modern GPUs like the A100 are designed for workloads with 100+ FLOPs per byte.

To put that in perspective, the GPU's compute cores are capable of performing 100 operations for every byte of data that arrives from memory.

But autoregressive decoding only needs 1 operation per byte. So the compute cores finish their work almost instantly and then sit waiting for the next chunk of data to arrive.

It's like having a team of 100 workers but only ever giving them 1 task at a time. They're not overloaded with work; they're waiting on the supply line.

The GPU is being used at roughly 1% of its computational capacity, not because the model is small, but because the memory system can't deliver data fast enough to keep the cores busy.

This is what it means for autoregressive decoding to be memory-bandwidth bound. The limiting factor isn't the GPU's ability to do math, it's how fast data can be shuttled from memory to the compute cores.

Buying a GPU with 2x more FLOPs doesn't help much if the memory bandwidth only improves by 1.3x. The gap between compute capability and memory bandwidth has been widening with every GPU generation, which means the Autoregressive bottleneck gets proportionally worse over time, not better.

The second structural problem is directional.

Autoregressive models only ever see left-to-right context during generation. This creates what's known as the reversal curse, where a model trained on documents containing "Tom Cruise's mother is Mary Lee Pfeiffer" will be significantly worse at answering "Who is Mary Lee Pfeiffer's son?" than "Who is Tom Cruise's mother?"

The original paper testing this on GPT-4 found 79% accuracy on the forward direction versus 33% on the reverse for 1,000 celebrity-parent pairs.

The asymmetry is caused by the autoregressive training objective. The model learns $p(\text{mother} | \text{Tom Cruise})$ but not $p(\text{Tom Cruise} | \text{mother's name})$ unless that ordering also appears in the training data.

These two bottlenecks (sequential generation that underutilizes hardware, and unidirectional context that creates reasoning blind spots) are structural properties of the autoregressive factorization itself. They motivate the search for a fundamentally different generation paradigm.

How diffusion works in images?

Before we look at diffusion for text, it helps to understand how diffusion works in the domain where it first succeeded, i.e., images.

The core idea is surprisingly simple, and understanding it will make the text case (and the problems it introduces) much clearer.

The forward process: systematically destroying data

A diffusion model starts with a clean image and progressively corrupts it by adding Gaussian noise over many timesteps. At each step, a small amount of random noise is added to every pixel value. After enough steps, the image becomes pure noise, indistinguishable from random static.

Formally, given clean data $x_0$ (the original image), the forward process at timestep $t$ produces:

There are several components here, so let's unpack each one.

A ReAct loop, a couple of tools, and a well-written system prompt can get surprisingly far in a demo.

But the moment the task requires 10+ steps, things fall apart like the model forgets what it did three steps ago, tool calls fail silently, and the context window fills up with garbage.

The problem isn't the model. It's everything around the model.

LangChain proved this when they changed only the infrastructure wrapping their LLM (same model, same weights) and jumped from outside the top 30 to rank 5 on TerminalBench 2.0.

A separate research project hit a 76.4% pass rate by having an LLM optimize the infrastructure itself, surpassing hand-designed systems.

That infrastructure has a name now: the agent harness.

What is Agent Harness?

The term was formalized in early 2026, but the concept existed long before.

The harness is the complete software infrastructure wrapping an LLM, including the orchestration loop, tools, memory, context management, state persistence, error handling, and guardrails.

Anthropic’s Claude Code documentation puts it simply: the SDK is “the agent harness that powers Claude Code.“

We really liked the canonical formula, from LangChain’s Vivek Trivedy: “If you’re not the model, you’re the harness.”

To put it another way, the “agent” is the emergent behavior: the goal-directed, tool-using, self-correcting entity the user interacts with. The harness is the machinery producing that behavior. When someone says “I built an agent,” they mean they built a harness and pointed it at a model.

Beren Millidge made this analogy precise in his 2023 essay:

• A raw LLM is a CPU with no RAM, no disk, and no I/O.
• The context window serves as RAM (fast but limited).
• External databases function as disk storage (large but slow).
• Tool integrations act as device drivers.

The harness is the operating system.

Three levels of engineering

Three concentric levels of engineering surround the model:

• Prompt engineering crafts the instructions the model receives.
• Context engineering manages what the model sees and when.
• Harness engineering encompasses both, plus the entire application infrastructure: tool orchestration, state persistence, error recovery, verification loops, safety enforcement, and lifecycle management.

The harness is not a wrapper around a prompt. It is the complete system that makes autonomous agent behavior possible.

The 11 components of a production Harness

Synthesizing across Anthropic, OpenAI, LangChain, and the broader practitioner community, a production agent harness has eleven distinct components. Let’s walk through each one.

1. The Orchestration Loop

This is the heartbeat. It implements the Thought-Action-Observation (TAO) cycle, also called the ReAct loop. The loop runs: assemble prompt, call LLM, parse output, execute any tool calls, feed results back, repeat until done.

Mechanically, it’s often just a while loop. The complexity lives in everything the loop manages, not the loop itself. Anthropic describes their runtime as a “dumb loop” where all intelligence lives in the model. The harness just manages turns.

Tools

Tools are the agent’s hands. They’re defined as schemas (name, description, parameter types) injected into the LLM’s context so the model knows what’s available. The tool layer handles registration, schema validation, argument extraction, sandboxed execution, result capture, and formatting results back into LLM-readable observations.

Claude Code provides tools across six categories: file operations, search, execution, web access, code intelligence, and subagent spawning. OpenAI’s Agents SDK supports function tools (via function_tool), hosted tools (WebSearch, CodeInterpreter, FileSearch), and MCP server tools.

3. Memory

Memory operates at multiple timescales. Short-term memory is the conversation history within a single session. Long-term memory persists across sessions: Anthropic uses CLAUDE.md project files and auto-generated MEMORY.md files; LangGraph uses namespace-organized JSON Stores; OpenAI supports Sessions backed by SQLite or Redis.

Claude Code implements a three-tier hierarchy: a lightweight index (~150 characters per entry, always loaded), detailed topic files pulled in on demand, and raw transcripts accessed via search only.

4. Context management

This is where many agents fail silently. The core problem is context rot: model performance degrades 30%+ when key content falls in mid-window positions.

Even million-token windows suffer from instruction-following degradation as context grows.

Production strategies include:

• Compaction: summarizing conversation history when approaching limits (Claude Code preserves architectural decisions and unresolved bugs while discarding redundant tool outputs)
• Observation masking: JetBrains’ Junie hides old tool outputs while keeping tool calls visible
• Just-in-time retrieval: maintaining lightweight identifiers and loading data dynamically (Claude Code uses grep, glob, head, tail rather than loading full files)
• Sub-agent delegation: each subagent explores extensively but returns only 1,000 to 2,000 token condensed summaries

Anthropic’s context engineering guide states the goal: find the smallest possible set of high-signal tokens that maximize likelihood of the desired outcome.

5. Prompt construction

This assembles what the model actually sees at each step. It’s hierarchical with system prompt, tool definitions, memory files, conversation history, and the current user message.

OpenAI’s Codex uses a strict priority stack: server-controlled system message (highest priority), tool definitions, developer instructions, user instructions (cascading AGENTS.md files, 32 KiB limit), then conversation history.

6. Output parsing

Modern harnesses rely on native tool calling, where the model returns structured tool_calls objects rather than free-text that must be parsed.

The harness checks if there are any tool calls? If yes, it executes them and loops. If not, it gives the final answer.

For structured outputs, both OpenAI and LangChain support schema-constrained responses via Pydantic models.

Legacy approaches like RetryWithErrorOutputParser (which feeds the original prompt, the failed completion, and the parsing error back to the model) remain available for edge cases.

7. State management

LangGraph models state as typed dictionaries flowing through graph nodes, with reducers merging updates.

Checkpointing happens at super-step boundaries, enabling resumption after interruptions and time-travel debugging.

OpenAI offers four mutually exclusive strategies: application memory, SDK sessions, server-side Conversations API, or lightweight previous_response_id chaining. Claude Code takes a different approach: git commits as checkpoints and progress files as structured scratchpads.

8. Error handling

Here’s why this matters: a 10-step process with 99% per-step success still has only ~90.4% end-to-end success due to compounding.

LangGraph distinguishes four error types: transient (retry with backoff), LLM-recoverable (return error as ToolMessage so the model can adjust), user-fixable (interrupt for human input), and unexpected (bubble up for debugging). Anthropic catches failures within tool handlers and returns them as error results to keep the loop running. Stripe’s production harness caps retry attempts at two.

9. Guardrails and safety

OpenAI’s SDK implements three levels: input guardrails (run on the first agent), output guardrails (run on the final output), and tool guardrails (run on every tool invocation).

A “tripwire” mechanism halts the agent immediately when triggered.

Anthropic separates permission enforcement from model reasoning architecturally. The model decides what to attempt; the tool system decides what’s allowed. Claude Code gates ~40 discrete tool capabilities independently, with three stages: trust establishment at project load, permission check before each tool call, and explicit user confirmation for high-risk operations.

10. Verification loops

This is what separates toy demos from production agents. Anthropic recommends three approaches: rules-based feedback (tests, linters, type checkers), visual feedback (screenshots via Playwright for UI tasks), and LLM-as-judge (a separate subagent evaluates output).

Boris Cherny, creator of Claude Code, noted that giving the model a way to verify its work improves quality by 2 to 3x.

11. Subagent orchestration

Claude Code supports three execution models: Fork (byte-identical copy of parent context), Teammate (separate terminal pane with file-based mailbox communication), and Worktree (own git worktree, isolated branch per agent).

OpenAI’s SDK supports agents-as-tools (specialist handles bounded subtask) and handoffs (specialist takes full control). LangGraph implements subagents as nested state graphs.

A step-by-step walkthrough

Now that you know the components, let’s trace how they work together in a single cycle.

• Step 1 (Prompt Assembly): The harness constructs the full input: system prompt + tool schemas + memory files + conversation history + current user message. Important context is positioned at the beginning and end of the prompt (the “Lost in the Middle” finding).
• Step 2 (LLM Inference): The assembled prompt goes to the model API. The model generates output tokens: text, tool call requests, or both.
• Step 3 (Output Classification): If the model produced text with no tool calls, the loop ends. If it requested tool calls, proceed to execution. If a handoff was requested, update the current agent and restart.
• Step 4 (Tool Execution): For each tool call, the harness validates arguments, checks permissions, executes in a sandboxed environment, and captures results. Read-only operations can run concurrently; mutating operations run serially.
• Step 5 (Result Packaging): Tool results are formatted as LLM-readable messages. Errors are caught and returned as error results so the model can self-correct.
• Step 6 (Context Update): Results are appended to the conversation history. If approaching the context window limit, the harness triggers compaction.
• Step 7 (Loop): Return to Step 1. Repeat until termination.

Termination conditions are layered: the model produces a response with no tool calls, the maximum turn limit is exceeded, the token budget is exhausted, a guardrail tripwire fires, the user interrupts, or a safety refusal is returned. A simple question might take 1 to 2 turns. A complex refactoring task can chain dozens of tool calls across many turns.

For long-running tasks spanning multiple context windows, Anthropic developed a two-phase “Ralph Loop” pattern.

It uses an Initializer Agent that sets up the environment (init script, progress file, feature list, initial git commit), then a Coding Agent in every subsequent session reads git logs and progress files to orient itself, picks the highest-priority incomplete feature, works on it, commits, and writes summaries.

The filesystem provides continuity across context windows.

How frameworks implement the pattern

Anthropic’s Claude Agent SDK exposes the harness through a single query() function that creates the agentic loop and returns an async iterator streaming messages.

The runtime is a “dumb loop.” All intelligence lives in the model. Claude Code uses a Gather-Act-Verify cycle: gather context (search files, read code), take action (edit files, run commands), verify results (run tests, check output), repeat.

OpenAI’s Agents SDK implements the harness through the Runner class with three modes: async, sync, and streamed.

The SDK is “code-first”: workflow logic is expressed in native Python rather than graph DSLs. The Codex harness extends this with a three-layer architecture: Codex Core (agent code + runtime), App Server (bidirectional JSON-RPC API), and client surfaces (CLI, VS Code, web app). All surfaces share the same harness, which is why “Codex models feel better on Codex surfaces than a generic chat window.”

LangGraph models the harness as an explicit state graph. Two nodes (llm_call and tool_node) connected by a conditional edge: if tool calls present, route to tool_node; if absent, route to END.

LangGraph evolved from LangChain’s AgentExecutor, which was deprecated in v0.2 because it was hard to extend and lacked multi-agent support. LangChain’s Deep Agents explicitly use the term “agent harness”: built-in tools, planning (write_todos tool), file systems for context management, subagent spawning, and persistent memory.

CrewAI implements a role-based multi-agent architecture: Agent (the harness around the LLM, defined by role, goal, backstory, and tools), Task (the unit of work), and Crew (the collection of agents). CrewAI’s Flows layer adds a “deterministic backbone with intelligence where it matters,” managing routing and validation while Crews handle autonomous collaboration.

The scaffolding metaphor

Construction scaffolding is a temporary infrastructure that enables workers to build a structure they couldn’t reach otherwise. It doesn’t do the construction. But without it, workers can’t reach the upper floors.

The key insight is that scaffolding is removed when the building is complete. As models improve, harness complexity should decrease. Manus was rebuilt five times in six months, each rewrite removing complexity. Complex tool definitions became general shell execution. “Management agents” became simple structured handoffs.

This points to the co-evolution principle where models are now post-trained with specific harnesses in the loop. Claude Code’s model learned to use the specific harness it was trained with. Changing tool implementations can degrade performance because of this tight coupling.

The future-proofing test for harness design states that if performance scales up with more powerful models without adding harness complexity, the design is sound.

Seven decisions for Harness definitions

Every harness architect faces seven choices:

1. Single-agent vs. multi-agent. Both Anthropic and OpenAI ask to maximize a single agent first. Multi-agent systems add overhead (extra LLM calls for routing, context loss during handoffs). Split only when tool overload exceeds ~10 overlapping tools or clearly separate task domains exist.
2. ReAct vs. plan-and-execute. ReAct interleaves reasoning and action at every step (flexible but higher per-step cost). Plan-and-execute separates planning from execution. LLMCompiler reports a 3.6x speedup over sequential ReAct.
3. Context window management strategy. Five production approaches include time-based clearing, conversation summarization, observation masking, structured note-taking, and sub-agent delegation. ACON research showed 26 to 54% token reduction while preserving 95%+ accuracy by prioritizing reasoning traces over raw tool outputs.
4. Verification loop design. Computational verification (tests, linters) provides deterministic ground truth. Inferential verification (LLM-as-judge) catches semantic issues but adds latency. Martin Fowler’s Thoughtworks team frames this as guides (feedforward, steer before action) versus sensors (feedback, observe after action).
5. Permission and safety architecture. Permissive (fast but risky, auto-approve most actions) versus restrictive (safe but slow, require approval for each action). The choice depends on the deployment context.
6. Tool scoping strategy. More tools often mean worse performance. Vercel removed 80% of tools from v0 and got better results. Claude Code achieves 95% context reduction via lazy loading. The principle: expose the minimum tool set needed for the current step.
7. Harness thickness. How much logic lives in the harness versus the model. Anthropic bets on thin harnesses and model improvement. Graph-based frameworks bet on explicit control. Anthropic regularly deletes planning steps from Claude Code’s harness as new model versions internalize that capability.

The harness is the product

Two products using identical models can have wildly different performance based solely on harness design. The TerminalBench evidence is clear that changing only the harness moved agents by 20+ ranking positions.

The harness is not a solved problem or a commodity layer. It’s where the hard engineering lives like managing context as a scarce resource, designing verification loops that catch failures before they compound, building memory systems that provide continuity without hallucination, and making architectural bets about how much scaffolding to build versus how much to leave to the model.

The field is moving toward thinner harnesses as models improve. But the harness itself isn’t going away. Even the most capable model needs something to manage its context window, execute its tool calls, persist its state, and verify its work.

The next time your agent fails, don’t blame the model but rather look at the harness.

Thanks for reading!

Introduction

Several AI/ML systems do not just fail because the model is not good enough. They fail because everything around the model was not built to last.

Consider a scenario: A team trains a model, it performs well in tests, and it is deployed to production. Yet conversion rates remain flat. A question from the finance team cannot be answered. Engineers spend weeks investigating why a recommendation system with 94% accuracy is making the product no better. This is not an edge case, rather a reflection of how real-world systems behave.

The companies that eventually get this right, do so by learning lessons that do not always have much to do with algorithms. Their progress comes from how they design systems, handle constraints, and adapt to failure.

This article is the final piece in our MLOps/LLMOps course. Here, we take a look at concrete, real-world examples. We examine a set of carefully chosen case studies drawn from real systems. Each case study focuses on the decisions that shaped the system, why specific approaches were chosen and the constraints teams operated under.

The examples span big tech, fintech, banking, e-commerce, etc. offering a grounded view of how modern AI/ML systems are actually built and sustained.

#1) The fundamental misunderstanding

Booking.com: Model performance ≠ Business performance

In 2019, Booking.com published a paper at KDD that has changed how most companies today think about ML.

It described 150 deployed production models and one uncomfortable lesson: improving a model's accuracy often did not improve the business metric it was supposed to affect.

There were several reasons for this:

• Value saturation: the model had already captured most of the available gain, and further accuracy improvements had nothing much left to unlock.
• Segment saturation: when testing a new model against the current one, the two models increasingly agree on what to show users, shrinking the population actually exposed to any difference. The testable segment becomes too small to move aggregate metrics.
• Proxy metric over-optimization: the model had learned to maximize something measurable that was only loosely correlated with what the business actually cared about.

• Uncanny valley effect: as a model becomes too accurate (predicting user behavior so precisely that it feels like the system knows too much), it can unsettle users, producing a negative effect on business value.

Their solution was to treat randomized controlled trials (RCTs) as mandatory infrastructure, not optional validation.

This means every single model gets validated through an RCT before it stays in production. This was not a qualitative review or a conversation about the AUC, but rather an actual experiment measuring whether users behaved differently in a way that matters to the business.

The deeper insight was about how teams construct problems. Switching a preferences model from click data to natural language processing on guest reviews produced more business value than any model-level improvement had. The framing of the problem and project scoping mattered more than the sophistication of the solution.

Claude Code users typically treat the .claude folder like a black box. They know it exists. They’ve seen it appear in their project root. But they’ve never opened it, let alone understood what every file inside it does.

That’s a missed opportunity.

The .claude folder is the control center for how Claude behaves in your project.

It holds your instructions, your custom commands, your permission rules, and even Claude’s memory across sessions. Once you understand what lives where and why, you can configure Claude Code to behave exactly the way your team needs it to.

This newsletter walks you through the entire anatomy of the folder, from the files you’ll use daily to the ones you’ll set once and forget.

Two folders, not one

Before diving in, one thing worth knowing upfront: there are actually two .claude directories, not one.

The first lives inside your project, and the second lives in your home directory:

The project-level folder holds team configuration. You commit it to git. Everyone on the team gets the same rules, the same custom commands, the same permission policies.

The global ~/.claude/ folder holds your personal preferences and machine-local state, like session history and auto-memory.

CLAUDE.md: Claude’s instruction manual

This is the most important file in the entire system. When you start a Claude Code session, the first thing it reads is CLAUDE.md. It loads it straight into the system prompt and keeps it in mind for the entire conversation.

Simply put: whatever you write in CLAUDE.md, Claude will follow.

If you tell Claude to always write tests before implementation, it will. If you say “never use console.log for error handling, always use the custom logger module,” it will respect that every time.

A CLAUDE.md at your project root is the most common setup. But you can also have one in ~/.claude/CLAUDE.md for global preferences that apply across all projects, and even one inside subdirectories for folder-specific rules. Claude reads all of them and combines them.

What actually belongs in CLAUDE.md

Most people either write too much or too little. Here’s what works.

Write:

• Build, test, and lint commands (npm run test, make build, etc.)
• Key architectural decisions (”we use a monorepo with Turborepo”)
• Non-obvious gotchas (”TypeScript strict mode is on, unused variables are errors”)
• Import conventions, naming patterns, error handling styles
• File and folder structure for the main modules

Don’t write:

• Anything that belongs in a linter or formatter config
• Full documentation you can already link to
• Long paragraphs explaining theory

Keep CLAUDE.md under 200 lines. Files longer than that start eating too much context, and Claude’s instruction adherence actually drops.

Here’s a minimal but effective example:

That’s ~20 lines. It gives Claude everything it needs to work productively in this codebase without constant clarification.

CLAUDE.local.md for personal overrides

Sometimes you have a preference that’s specific to you, not the whole team. Maybe you prefer a different test runner, or you want Claude to always open files using a specific pattern.

Create CLAUDE.local.md in your project root. Claude reads it alongside the main CLAUDE.md, and it’s automatically gitignored so your personal tweaks never land in the repo.

The rules/ folder: modular instructions that scale

CLAUDE.md works great for a single project. But once your team grows, you end up with a 300-line CLAUDE.md that nobody maintains and everyone ignores.

The rules/ folder solves that.

Every markdown file inside .claude/rules/ gets loaded alongside your CLAUDE.md automatically. Instead of one giant file, you split instructions by concern:

Each file stays focused and easy to update. The team member who owns API conventions edits api-conventions.md. The person who owns testing standards edits testing.md. Nobody stomps on each other.

The real power comes from path-scoped rules. Add a YAML frontmatter block to a rule file and it only activates when Claude is working with matching files:

Claude won’t load this file when editing a React component. It only loads when it’s working inside src/api/ or src/handlers/. Rules without a paths field load unconditionally, every session.

This is the right pattern once your CLAUDE.md starts feeling crowded.

The commands/ folder: your custom slash commands

Out of the box, Claude Code has built-in slash commands like /help and /compact. The commands/ folder lets you add your own.

Every markdown file you drop into .claude/commands/ becomes a slash command.

A file named review.md creates /project:review. A file named fix-issue.md creates /project:fix-issue. The filename is the command name.

Here’s a simple example. Create .claude/commands/review.md:

Now run /project:review in Claude Code and it automatically injects the real git diff into the prompt before Claude sees it. The ! backtick syntax runs shell commands and embeds the output. That’s what makes these commands genuinely useful instead of just saved text.

Passing arguments to commands

Use $ARGUMENTS to pass text after the command name:

Running /project:fix-issue 234 feeds issue 234’s content directly into the prompt.

Personal vs. project commands

Project commands in .claude/commands/ are committed and shared with your team. For commands you want everywhere regardless of project, put them in ~/.claude/commands/. Those show up as /user:command-name instead.

A useful personal command: a daily standup helper, a command for generating commit messages following your convention, or a quick security scan.

The skills/ folder: reusable workflows on demand

You now know how commands work. Skills look similar on the surface, but the trigger is fundamentally different. Here’s the distinction before we go any further:

Skills are workflows that Claude can invoke on its own, without you typing a slash command, when the task matches the skill’s description. Commands wait for you. Skills watch the conversation and act when the moment is right.

Each skill lives in its own subdirectory with a SKILL.md file:

The SKILL.md uses YAML frontmatter to describe when to use it:

When you say “review this PR for security issues,” Claude reads the description, recognizes it matches, and invokes the skill automatically. You can also call it explicitly with /security-review.

The key difference from commands: skills can bundle supporting files alongside them. The DETAILED_GUIDE.md reference above pulls in a detailed document that lives right next to SKILL.md. Commands are single files. Skills are packages.

Personal skills go in ~/.claude/skills/ and are available across all your projects.

The agents/ folder: specialized subagent personas

When a task is complex enough to benefit from a dedicated specialist, you can define a subagent persona in .claude/agents/. Each agent is a markdown file with its own system prompt, tool access, and model preference:

Here’s what a code-reviewer.md looks like:

When Claude needs a code review done, it spawns this agent in its own isolated context window. The agent does its work, compresses the findings, and reports back. Your main session doesn’t get cluttered with thousands of tokens of intermediate exploration.

The tools field restricts what the agent can do. A security auditor only needs Read, Grep, and Glob. It has no business writing files. That restriction is intentional and worth being explicit about.

The model field lets you use a cheaper, faster model for focused tasks. Haiku handles most read-only exploration well. Save Sonnet and Opus for the work that actually needs them.

Personal agents go in ~/.claude/agents/ and are available across all projects.

settings.json: permissions and project config

The settings.json file inside .claude/ controls what Claude is and isn’t allowed to do. It’s where you define which tools Claude can run, which files it can read, and whether it needs to ask before running certain commands.

The complete file looks like this:

Here’s what each part does.

The $schema line enables autocomplete and inline validation in VS Code or Cursor. Always include it.

The allow list contains commands that run without Claude asking for confirmation. For most projects, a good allow list covers:

• Bash(npm run *) or Bash(make *) so Claude can run your scripts freely
• Bash(git *) for read-only git commands
• Read, Write, Edit, Glob, Grep for file operations

The deny list contains commands that are blocked entirely, no matter what. A sensible deny list blocks:

• Destructive shell commands like rm -rf
• Direct network commands like curl
• Sensitive files like .env and anything in secrets/

If something isn’t in either list, Claude asks before proceeding. That middle ground is intentional. It gives you a safety net without having to anticipate every possible command upfront.

That said, you can also have settings.local.json for personal overrides. It has the same idea as CLAUDE.local.md. Create .claude/settings.local.json for permission changes you don’t want committed. It’s auto-gitignored.

The global ~/.claude/ folder

You don’t interact with this folder often, but it’s useful to know what’s in it.

~/.claude/CLAUDE.md loads into every Claude Code session, across all your projects. Good place for your personal coding principles, preferred style, or anything you want Claude to remember, regardless of which repo you’re in.

~/.claude/projects/ stores session transcripts and auto-memory per project. Claude Code automatically saves notes to itself as it works: commands it discovers, patterns it observes, and architecture insights. These persist across sessions. You can browse and edit them with /memory.

~/.claude/commands/ and ~/.claude/skills/ hold personal commands and skills available across all projects.

You generally don’t need to manually manage these. But knowing they exist is handy when Claude seems to “remember” something you never told it, or when you want to wipe a project’s auto-memory and start fresh.

The full picture

Here’s how everything comes together:

A practical setup to get started

If you’re starting from scratch, here’s a progression that works well.

Step 1. Run /init inside Claude Code. It generates a starter CLAUDE.md by reading your project. Edit it down to the essentials.

Step 2. Add .claude/settings.json with allow/deny rules appropriate for your stack. At minimum, allow your run commands and deny .env reads.

Step 3. Create one or two commands for the workflows you do most. Code review and issue fixing are good starting points.

Step 4. As your project grows and your CLAUDE.md gets crowded, start splitting instructions into .claude/rules/ files. Scope them by path where it makes sense.

Step 5. Add a ~/.claude/CLAUDE.md with your personal preferences. This might be something like “always write types before implementations” or “prefer functional patterns over class-based.”

That’s genuinely all you need for 95% of projects. Skills and agents come in when you have recurring complex workflows worth packaging up.

The key insight

The .claude folder is really a protocol for telling Claude who you are, what your project does, and what rules it should follow. The more clearly you define that, the less time you spend correcting Claude and the more time it spends doing useful work.

CLAUDE.md is your highest-leverage file. Get that right first. Everything else is optimization.

Start small, refine as you go, and treat it like any other piece of infrastructure in your project: something that pays dividends every day once it’s set up properly.

Recap

In the previous chapter (Part 13), we explored how LLM inference actually works and how to optimize it.

We began by grounding ourselves in the core performance metrics such as TTFT, TPOT, throughput (TPS/RPS), latency percentiles, and goodput.

We then built a precise mental model of inference by breaking it into two distinct phases: prefill (compute-bound) and decode (memory-bound).

This distinction became central to understanding why different optimizations work the way they do.

From there, we explored key optimization techniques across the inference stack, starting with continuous batching to maximize GPU utilization and throughput under variable-length workloads.

Then we understood about KV caching to eliminate redundant computation and essentially trade compute for memory.

After KV caching, we learned about PagedAttention & prefix caching to fix memory fragmentation and reuse shared prefixes across requests, significantly improving effective memory usage and throughput.

We also saw that KV cache quantization can further reduce memory footprint and enable larger batch sizes or longer contexts.

We then looked at attention-level optimizations, including MQA, GQA, and FlashAttention.

Next, we explored speculative decoding, which leverages a smaller draft model to accelerate generation without sacrificing output quality, along with its practical trade-offs like acceptance and memory overhead.

We also discussed prefill-decode disaggregation, where systems separate compute-bound and memory-bound workloads across different hardware pools, and how its effectiveness depends on workload characteristics.

Finally, we covered parallelism strategies (data, tensor, pipeline, and expert parallelism) to scale inference across multiple devices.

Finally, we explored hands-on experiments demonstrating: the impact of KV caching, the speedup from speculative decoding, and the performance gains of vLLM.

Overall, this chapter established that LLM inference is not just about running a model, but about carefully managing compute, memory, and the trade-offs.

If you haven’t yet gone through Part 13, we recommend reviewing it first.

Read it here:

LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques

LLMOps Part 13: Exploring the mechanics of LLM inference, from prefill and decode phases to KV caching, batching, and optimization techniques that improve latency and throughput.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will discuss the fundamentals of LLM serving, exploring self-hosting of models, API-based access, and inference with vLLM.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

If you have a language model and want to make it accessible through an API that can be invoked by others, then this article serves as the operations manual for the fundamentals of that journey.

While there are many similarities with traditional ML deployment, serving a large language model introduces a different set of challenges.

LLMs are resource-intensive, often consuming significant VRAM even when idle. In naive setups, requests might be handled sequentially, meaning a single long-running generation can block all subsequent users. Cold starts are slower, and scaling is more complex due to the heavy compute and memory requirements.

In contrast, conventional ML systems typically assume lightweight models with fast initialization and low per-request latency, assumptions that might not hold strong in the case of LLMs.

In the sections ahead, we’ll build an understanding of the core fundamentals and concepts behind serving LLMs. We'll try to leave you with enough clarity and intuition to replicate, adapt, and extend the setup demonstrated.

Accessing inference

The very first thing we need to understand is how an application will access LLM API for inference.

The LLM deployment and serving landscape broadly splits into two categories:

• API providers (OpenAI, Anthropic, etc.): These are inference services. You send a request, you get a response. You do not manage hardware, you do not worry about GPU provisioning, and you do not implement optimizations. The provider handles all of that. However, you still need to manage your application layer.
• Self-hosted inference: This means you run the model yourself. You provision GPUs (either on-premises or in the cloud), use a model inference and serving mechanism (like vLLM, TGI, etc.), and manage the entire stack. This gives you full control over model selection, inference configuration, data privacy, and cost structure. It also means you are responsible for everything: GPU allocation, model loading, optimization, and more.

Deployment topology

Where the model runs is a strategic decision, not just a technical one. The choice affects data security, cost structure, operational overhead, and the teams responsible for keeping things running.

On-premises

Teams choose on-prem LLM deployments primarily for three reasons: data security and compliance, predictable cost at scale, and performance control.

For regulated industries like healthcare, finance, and government, sending user data to a third-party inference API is often not an option. An on-prem deployment keeps all inference traffic inside the organization's own network perimeter. There is no data leaving the building.

Cost predictability is the second driver. Cloud GPU pricing can be volatile, and token-based pricing from hosted API providers becomes expensive at scale. A fleet of owned or co-located GPUs has a roughly fixed monthly cost. Once the infrastructure investment is amortized, the marginal cost of additional inference is just power and operations. For steady, high-volume workloads, this math often favors on-prem.

The downsides however are also quite real:

• Upfront capital expenditure is substantial
• Operational complexity is high
• Iteration speed suffers when a new model architecture requires different hardware or driver versions and your on-prem cluster may not be ready for that.

Cloud deployments

Cloud deployments, whether managed inference APIs or self-hosted models on rented GPU instances, offer the inverse tradeoffs.

• No upfront cost.
• Access to the latest GPU generations on demand.
• Horizontal scaling in minutes.
• No capacity planning headache.

However, the costs are variable and can surprise you at scale. You pay for what you use, which is great for experimentation and bursty workloads but can scale quickly as volume grows.

Data leaves your infrastructure, which creates compliance complexity for sensitive workloads. You are also dependent on provider SLAs and GPU availability, which can be constrained during periods of high demand.

Cloud is the right default for early-stage deployments, development and staging environments, workloads with highly variable or unpredictable traffic, and any use case where speed of iteration matters more than cost optimization.

Hybrid setup

Similar to inference access patterns, many deployments end up hybrid, combining on-prem baseline capacity with cloud overflow. The pattern works as follows: steady, predictable traffic runs on owned on-prem hardware (lower marginal cost). When traffic spikes beyond on-prem capacity, overflow traffic routes to cloud GPUs that spin up on demand.

Recap

In the last chapter (Part 12), we explored how language models can be adapted via fine-tuning.

We began by discussing the central question of when fine-tuning is actually worth doing. We studied the reasons to fine-tune and the reasons to avoid it.

After that, we moved to understanding PEFT techniques. In particular, we explored LoRA and QLoRA. We understood that LoRA reduces the number of trainable parameters, while QLoRA combines ideas of LoRA with quantization.

Next, we shifted our focus to alignment fine-tuning. We explored three major preference optimization approaches: RLHF, DPO, and GRPO. Each of these methods provides a different way to push model behavior closer to human preferences or task-specific reward signals.

Then, we looked at the role of data in fine-tuning and emphasized the quality and structure of training data.

Finally walked through a hands-on GRPO-based demo using Unsloth, TRL, and the GSM8K dataset.

Overall, the previous chapter established fine-tuning as a tool for adapting LLMs, while also making it clear that it should be used thoughtfully and only when simpler approaches are not enough.

If you haven’t yet gone through Part 12, we recommend reviewing it first, as it helps maintain the natural learning flow of the series.

Read it here:

LLM Fine-tuning: Techniques for Adapting Language Models

LLMOps Part 12: Understanding LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will be exploring LLM inference and optimization. We will understand how inference works and what techniques are used to optimize it.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Key throughput and latency metrics

Understanding and optimizing inference without understanding the right metrics is probably not a very smart thing to do. Thus, let’s first establish how we measure LLM inference performance in the first place:

• Time to first token (TTFT): The time taken to generate the first token after sending a request. It basically evaluates how fast the model can start responding. Think of TTFT as the startup latency for a query.
• Time per output token (TPOT): Once the first token is out, TPOT measures the steady-state speed of generation: the average time to produce each subsequent token. A lower TPOT means the model can produce tokens faster, leading to higher tokens per second. Formally, if an output has $N$ tokens and the total generation time after the first token is $T$, then:

• End-to-end latency (E2E): The total time from request to receiving the final output token.
• Throughput: How much work the system can handle per unit time. Throughput is often measured in two ways:
  • Requests per second (RPS): tells how many requests can be completed per second.
  • Tokens per second (TPS): measures token processing rate, including input tokens (input TPS) and output tokens (output TPS) across all requests.

• Latency percentiles (p95, p99) capture the tail experience. A system with an average TTFT of 200ms may have a p99 TTFT of 2 seconds, meaning 1% of users wait 10× longer. Thus, our objectives/SLOs should also target p95 or p99, not just averages (especially important if consistency is critical).
• Goodput is the fraction of requests meeting all objective/SLO constraints simultaneously (e.g., TTFT < 500ms AND TPOT < 50ms).

Now that we understand the metrics, let’s go ahead and build a precise mental model of how the LLM inference mechanism works.

How LLM inference works

Autoregressive language models generate tokens one at a time, where each new token depends on all previous tokens. This dependency is the fundamental constraint that makes understanding inference and optimization challenging.

The two phases of LLM inference

Critically, LLM inference isn’t a uniform workload; it has two distinct phases:

• Prefill phase: The model processes the input tokens to compute the intermediate states (keys and values), which are used to generate the first output token. Because the full extent of the input is known, at a high level, this is a highly parallelized matrix-matrix operation, and the GPU can reach high utilization (the operation is compute-bound). The K and V projections are stored in what we call the KV cache for later reuse (more on it later in the chapter). The prefill phase completes by building the full KV cache for the prompt and producing the logits from which the first output token is sampled. The decode phase then generates subsequent tokens one at a time. The time until this first token is generated maps directly to the TTFT metric and is dominated by the prefill phase.

• Decode phase: The model generates output tokens one by one. At each step, only the single newest token's Q, K, V projections are computed. The new K and V vectors are appended to the KV cache, and the new Q vector attends against all cached K vectors. This results in low-parallel, matrix-vector kind of operations with poor hardware utilization. The arithmetic intensity plummets: tons of memory accesses for a tiny operation. Thus, the decode phase is usually memory-bandwidth-bound and hence maps to the TPOT metric.

Recap

In the last chapter (Part 11), we completed our journey of understanding how evaluation works in LLM systems.

We began by exploring multi-turn evaluation. Unlike single-turn scenarios where one prompt produces one response, conversational systems require evaluating behavior across an entire dialogue. A response in later turns often depends on things that happened earlier in the conversation.

Next, we shifted our focus to tool evaluation. We studied that in tool-enabled applications, the final text response is only a small part of the system’s behavior. The underlying sequence of tool decisions such as which tools were selected, in what order they were executed, and what arguments were passed, can determine whether the task actually succeeds.

After that, we moved to understanding the role of Langfuse in the evaluation space. We discussed how we can capture the lifecycle of every request as a trace, which records inputs, outputs, latency, token usage, and intermediate operations. We also demonstrated how evaluation results can be attached to traces.

Next, we introduced red teaming, which focuses on evaluating how LLM systems behave under adversarial conditions. We explored DeepTeam for automated red teaming, and saw it generate adversarial prompts targeting vulnerabilities.

Finally, we discussed two important operational considerations that influence pipelines in systems: latency and cost.

Overall, the previous chapter broadens the concept of evaluation, presenting it as a systemic discipline.

If you haven’t yet gone through Part 11, we recommend reviewing it first, as it helps maintain the natural learning flow of the series.

Read it here:

Evaluation: Multi-turn Conversations, Tool Use, Tracing, and Red Teaming

LLMOps Part 11: Understanding evaluation of conversational LLM systems, tool evaluations, tracing with Langfuse, and automated red teaming.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will be exploring LLM fine-tuning. We'll understand parameter-efficient training methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

Fine-tuning refers to adapting a pre-trained LLM to a specific task or domain by further training all or part of its weights. It can dramatically improve an LLM’s performance on specialized tasks or formats.

Fine-tuning is especially popular for enhancing instruction-following ability, making a model better at obeying user prompts and style guidelines. Today, with the rapid increase of open-source models of all sizes, fine-tuning has become far more popular and attractive than in the early GPT-3 era.

However, still, fine-tuning is a significant investment. It requires high-quality data, ML expertise, and substantial compute resources.

A common question is when to fine-tune versus using prompting or RAG alone. In practice, fine-tuning is attempted only after exhausting prompt-based methods, and often used in tandem with prompting (e.g. a fine-tuned model plus good prompts) in real-world scenarios.

Let’s now delve deeper into this chapter by exploring the advantages and limitations of LLM fine-tuning, briefly reviewing some of the major fine-tuning techniques, and examining emerging approaches such as model merging.

Why fine-tune an LLM? And why not?

Reasons to fine-tune

The primary motivation is to improve a model’s quality or specificity beyond what prompting alone can achieve. Fine-tuning can unlock latent capabilities of a model that are hard to elicit via prompts.

Common use cases include:

• Task/Domain specialization: If a model wasn’t trained sufficiently on your domain or task, fine-tuning on in-domain data can dramatically boost performance. For example, an out-of-the-box model might handle standard SQL but fail on a niche dialect; fine-tuning on that dialect will teach the model the needed syntax. Similarly, a legal LLM can be refined on legal documents to improve its legal reasoning.

• Format and style tuning: Fine-tuning is often used to enforce structured output formats (JSON, XML, markdown, etc.) or a specific style/tone. While prompt instructions can coax format, a well fine-tuned model will natively produce the format reliably.
• Instruction following and alignment: Instruction-tuned models are fine-tuned to better follow human instructions and avoid undesired outputs. Nowadays, it’s standard for LLM providers to release a base model and a supervised-finetuned (SFT) version for dialogue. SFT uses high-quality (instruction, response) pairs to teach the model to respond helpfully. This unlocks usability improvements that pure pre-training doesn’t yield.
• Bias and safety mitigation: Fine-tuning on carefully curated data can correct unwanted biases or behaviors in the base model.
• Efficiency via smaller models: A well-finetuned small model can many-a-times outperform a larger general model on a narrow task. Fine-tuning thus enables deploying lightweight models that achieve required accuracy, which is beneficial for cost and latency (since a smaller model is cheaper and faster to run).

In summary, fine-tuning shines when you need custom behavior or high accuracy in a specific setting that an already available model wasn’t probably trained for, explicitly. It can yield a model that is better suited to your application than any general-purpose model.

Reasons NOT to fine-tune

Despite its benefits, fine-tuning is neither trivial nor always the right choice:

• Prompting or RAG may suffice: There are many improvements can often be achieved to a good extent with prompt engineering, providing context, or retrieval augmentation without the cost of training, especially improvements where external knowledge is the primary factor. Hence, before fine-tuning, one should try instructing the model or supplying reference context to see if performance is acceptable. Fine-tuning is generally a last resort when other methods fail to meet requirements.

• Risk of over-specialization: Fine-tuning on a specific task can degrade performance on other tasks the model used to do well (a form of catastrophic forgetting). In a multi-task application, this means you may have to fine-tune on all desired tasks or use separate models per task. Developing and maintaining multiple fine-tuned models is operationally complex.
• Maintenance and model freshness: A fine-tuned model can become stale. If a new model is released that significantly outperforms your fine-tuned model, you face a tough choice: stick with your older fine-tuned model, or adopt the new model and incur the cost of again fine-tuning on your data. In rapidly evolving fields, this cycle can repeat often.
• Data and requirements: Fine-tuning requires significant upfront investment in data. You need a high-quality dataset of task-specific examples. Obtaining and curating this data (or generating it with AI and cleaning it) can be slow and expensive, especially for complex tasks requiring domain expertise.

• Computational cost: Fine-tuning (especially full fine-tuning) is computationally intensive. Hence, if you are just prototyping an idea, jumping straight into fine-tuning is rarely the first step; it makes sense only once you’re convinced that a custom model is needed and worth the investment.

In practice, many teams follow a progression: start with prompt (and context) engineering and off-the-shelf models, only proceed to fine-tune when necessary for quality or latency reasons.

Next, let’s explore some effective techniques for fine-tuning large language models (LLMs).

Memory-efficient fine-tuning: PEFT

A major challenge in fine-tuning large models is memory. Full fine-tuning of all model weights can be prohibitively expensive in terms of GPU VRAM. Parameter-efficient fine-tuning (PEFT) approaches avoid this problem by reducing the number of trainable parameters, thereby cutting memory and compute while aiming to preserve performance.

Let's now survey the two of the most popular and prominent PEFT methods:

LoRA: Low-Rank Adaptation of LLMs

LoRA is by far the most popular method in the PEFT family of techniques. LoRA builds on a simple insight: neural network weight updates often lie in a low-dimensional subspace. This means that when we fine-tune a model, the change we need to make to the original weights does not require a full-rank update. Instead, the update matrix can often be well approximated by a low-rank decomposition composed of much smaller matrices.

By learning these low-rank adapters while keeping the original model weights frozen, LoRA drastically reduces the number of trainable parameters and the memory required for fine-tuning.

Intuition: Imagine a 1000-dimensional parameter space, but the useful changes you need to make to adapt the model only lie along 8 important directions. Instead of searching the entire 1000-dimensional space, you can restrict learning to those 8 directions and still capture the necessary behavior change. LoRA exploits this idea by constraining weight updates to a low-rank subspace.

How LoRA works

Consider a single weight matrix in the model. Call this matrix $W$ and let the dimension be 4096, then $W$ is a $4096×4096$ matrix with about 16.7 million parameters.

In standard fine-tuning, you'd compute a gradient for every one of those 16.7 million entries and update them all. But, LoRA does something different: it freezes $W$ entirely and instead learns a small correction to it.

LoRA represents the weight update as the product of two small matrices:

where $A$ is an $n×r$ matrix and $B$ is an $r×m$ matrix. The key parameter here is $r$, the rank hyperparameter, which is typically very small (for example, 8 or 16).

Thus, the effective weight used during the forward pass becomes:

During training, only $A$ and $B$ receive gradients and get updated. The original weight matrix $W$ stays exactly as it was after pre-training. Hence, we're not modifying the model, we're learning a lightweight patch on top of it.

How it helps?

Continuing with our earlier assumed data, the original matrix $W$ has $4096×4096$ weight matrix with rank $r=8$ (say):

Therefore:

• Original: $4096×4096=16,777,216$ parameters
• LoRA: $(4096×8)+(4096×8)=65,536$ parameters

That's about just 0.4% of the original, and across the whole model, this has a dramatic effect.

Initialization and scaling

At the start of training, $ΔW$ is initialized to zero so that the model begins exactly where pre-training left off. Typically, $A$ is initialized with small random values (from a Gaussian distribution) and $B$ is initialized to all zeros, so that $A×B=0$ initially.

LoRA also introduces a scaling factor $α$, so the actual update is:

Here:

• The division by $r$ normalizes the update magnitude so that different rank choices behave consistently.
• The $\alpha$ parameter provides explicit control over how strongly the LoRA adaptation affects the original model.

At inference time

Once training is done, you have two options:

Option 1:

Merge the weights. Compute $W′ = W+\frac{α}{r}⋅(A×B)$ once, and replace $W$ with $W′$ in the model. Now the model runs at exactly the same speed as the original. The LoRA matrices are "baked in."

Option 2:

Keep them separate. Store $A$ and $B$ alongside the frozen model and add their product on the fly during inference. This adds a small amount of computation, but it means you can easily toggle the adaptation on or off, or swap in different LoRA adapters for different tasks without touching the base model.

This second option is especially powerful: you can have one base model and dozens of smaller LoRA adapter files, each specializing the model for a different task.

Where LoRA is applied?

In a transformer, every layer has several weight matrices: the query, key, value, and output projections in the attention mechanism, plus the weight matrices in the feed-forward network. LoRA can be applied to any subset of these.

The most common practice is to apply LoRA to the attention projection matrices (the query and value ones are the most popular targets).

With LoRA now covered, let's go ahead and briefly explore about quantization and QLoRA.

Quantization and QLoRA

Our discussion on LoRA addressed one axis of efficiency: reducing the number of trainable parameters. Let's now tackle the second, complementary axis: reducing the precision of the numbers used to store model weights. This is called quantization, and when combined with LoRA, it produces one of the most efficient techniques.

What is quantization?

Early deep learning models were typically trained using 32-bit floating point (float32). Modern large language model training, however, usually relies on mixed-precision training, where most computations use 16-bit formats such as float16 or bfloat16, while certain values like optimizer states are still maintained in float32 for numerical stability. Quantization means representing these values with even fewer bits: for example, 8-bit integers and 4-bit integers.

The memory savings are proportional and direct. For example, storing model weights in 16-bit precision requires roughly half the memory of 32-bit floats. Similarly, moving from 16-bit weights to 4-bit quantized weights reduces memory usage by about another four times (roughly).

For inference (just running the model), 8-bit quantization has been reliable for years, and 4-bit inference has become standard practice lately with negligible quality loss. But training in low precision is harder, because gradients need to be accurate to make useful updates. This is where QLoRA comes in.

QLoRA

QLoRA makes 4-bit training practical by combining two ideas:

• Store the frozen base model in 4-bit precision to save memory.
• Keep the LoRA adapter matrices in 16-bit precision so that gradient computation stays accurate.

The base model weights are treated as a compressed, read-only lookup. During the forward pass, each 4-bit weight is temporarily dequantized back to 16-bit, used in the computation, and then discarded. Gradients flow backward through this dequantization step and into the LoRA matrices $A$ and $B$, which are the only things being updated.

In other words, you never actually train in 4-bit. You store in 4-bit and compute in higher precision. The frozen weights just need to be close enough to their original values that the model still behaves correctly, and the LoRA matrices handle all the actual learning.

NF4

Not all 4-bit representations are equal. A naive approach would space the 16 possible values (that 4 bits can represent) uniformly across the weight range. But neural network weights are not uniformly distributed. They tend to follow a roughly normal distribution. NF4 (NormalFloat 4-bit) exploits this.

It spaces the 16 quantization levels so that they are information-theoretically optimal for normally distributed data. More levels are packed near zero (where most weights live) and fewer levels are placed in the tails.

The result: NF4 preserves significantly more of the original model's knowledge than naive 4-bit quantization.

What QLoRA made possible

QLoRA as a technique made it possible to let a person with one high-end GPU do what previously required a multi-node cluster. Thus, the practical barrier to fine-tuning large models dropped by an order of magnitude.

However, there is a tradeoff too. The primary cost of QLoRA is wall-clock time. Every weight that participates in a forward or backward pass must be dequantized on the fly. This extra step adds overhead.

In practice, however, this is almost always an acceptable tradeoff, because the alternative is not "slower training" but rather "no training at all" on a specific piece of hardware.

Quantization beyond training

Quantization is arguably even more impactful at inference time. Serving a model requires fitting it into memory first. Smaller precision means:

• More models per GPU (or larger models on the same GPU)
• Higher throughput (more tokens per second)
• Lower cost per query

Nowadays, it is common practice to deploy LLMs in 8-bit or 4-bit precision. The quality loss is typically negligible for chat and generation tasks. Some deployment setups use mixed precision: 4-bit for most layers and 8-bit or 16-bit for a few layers that are especially sensitive to quantization error.

In summary, both LoRA and QLoRA are essentially ways to minimize changes to the model while still adapting it. They prevent us from having to update billions of weights, making fine-tuning cheaper, and more accessible, and have proven to achieve performance comparable to full model fine-tuning.

Next, let’s explore alignment fine-tuning, where the focus is shaping how a model behaves so its responses better match human intentions and expectations.

Alignment fine-tuning: RLHF, DPO, and GRPO

Recap

In part 10, we expanded our understanding of evaluation by exploring model benchmarks and the evaluation of LLM-powered applications within their real operational context.

We began by exploring model capability benchmarks. We understood that these benchmarks attempt to measure the general intelligence, reasoning ability, and knowledge breadth of large language models.

We discussed widely used benchmarks such as MMLU, HellaSwag, TruthfulQA, etc. An important takeaway from this discussion was that benchmark scores are useful for narrowing down candidate models, but they should not be treated as definitive indicators of performance for a particular application context.

From there, we shifted our focus to application-level evaluation. We first discussed test sets. Then we examined several key evaluation metrics, including deterministic ones such as exact match, contains-answer, and token-level F1.

For RAG pipelines, we explored retrieval-specific metrics including Recall@K, Precision@K, and Mean reciprocal rank (MRR).

We also examined groundedness (faithfulness) evaluation, which measures whether the generated answer is supported by the retrieved context. One widely used technique here is Question-Answer Generation (QAG), which decomposes generated outputs into claims, converts them into verification questions, and checks them against the source context.

We then explored the role of LLM-as-a-Judge methods. We emphasized that the reliability of an LLM judge depends heavily on prompt, rubric design and policy layer. Building on this, we examined G-Eval, a structured LLM-based evaluation framework.

Finally, we moved into practical tooling, focusing on the DeepEval framework. Using DeepEval, we demonstrated how to define test cases, attach evaluation metrics, and run evaluation programmatically. We also looked at how DeepEval supports RAG evaluation. Additionally, we explored DAG-based evaluation. DAG metrics are particularly useful for evaluations such as checking structure, verifying the presence of required sections, or enforcing formatting constraints.

Overall, the chapter extended our evaluation toolkit beyond the fundamentals.

If you haven’t yet gone through Part 10, we strongly recommend reviewing it first, as it establishes the conceptual foundation essential for understanding the material we’re about to cover.

Read it here:

Evaluation: Model Benchmarks and LLM Application Assessment

LLMOps Part 10: Understanding model benchmarks, LLM application evaluation, and tooling.

Daily Dose of Data ScienceAvi Chawla

In this final chapter on LLM evaluation, we will understand evaluation of multi-turn systems, tool use evals, tracing, and red teaming.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Evaluating multi-turn conversation

By now, single-turn evaluation is a solved problem for us, more or less. You feed a prompt to a model, get a response, and compare it against a reference answer. Multi-turn evaluation is harder. The quality of turn five depends on everything that happened in turns one through four. A response that looks correct in isolation might contradict something the model said two turns ago.

This section covers the concepts and tooling that matter for multi-turn conversations.

Two levels of evaluation

Multi-turn evaluation can operate at two granularities:

• Turn-level evaluation assesses each individual exchange. For pipelines, we can easily reuse most of your single-turn evaluation machinery here. The key difference is that we pass the full conversation history as context to the judge, not just the last user message. Turn-level scoring is how you pinpoint where a conversation breaks down. If a five-turn dialogue fails at the end, turn-level scores might reveal the real problem started at turn three (say).

• Task-level evaluation answers a different question: did the conversation accomplish the user's goal? For a customer support bot, that might mean the issue was resolved. For a coding assistant, it might mean the final snippet runs. Task-level evaluation often requires either explicit success criteria embedded in the test cases or a goal-extraction step where it is inferred what the user was trying to accomplish and then check whether it happened.

Metrics

Some key signals to observe and evaluate in multi-turn systems include:

• Context retention: whether the model remembers and uses information from earlier turns. If it forgets, that is a failure.
• Coherence: whether the dialogue flows naturally from one message to another.
• Relevancy: whether the dialogue stays on topic or does the system deviate to nonsensical tangents.

Now, a point to note here is, we may not always need to manually implement all of these metrics ourselves. This is where evaluation frameworks become essential. Let’s now explore how DeepEval enables the evaluation of multi-turn conversations through its metrics.

Implementing multi-turn evaluation with DeepEval

DeepEval provides a practical framework for multi-turn evaluation. It represents a dialogue as a ConversationalTestCase, which is a sequence of Turns. We then apply the conversational metrics.

Here is a concrete example:

This script, instead of evaluating a single prompt-response pair, evaluates an entire dialogue between a user and an assistant.

• The ConversationalTestCase and Turn classes are used to represent the conversation itself. Each message in the dialogue is represented as a Turn, with a role and the corresponding message content.
• After defining the conversation, the script initializes three evaluation metrics.
  • TurnRelevancyMetric constructs sliding windows of turns for each turn, before using the LLM to determine whether the last turn in every sliding window has an "assistant" content that is relevant to the previous conversational context found in the sliding window.
  • KnowledgeRetentionMetric, evaluates whether the assistant correctly remembers and uses information from earlier turns in the conversation.
  • Finally safe_advice, a ConversationalGEval metric, which is a modified implementation of standard G-Eval LLM-as-judge evaluation. Using this we can determine whether our LLM chatbot responses are up to standard with our custom criteria throughout the conversation.
• All three metrics are configured with thresholds and an evaluation model (openai/gpt-4o-2024-08-06).
• The script then runs the evaluations in a standalone manner by calling .measure(test_case) on each metric.

• Finally, the results are printed.

Overall, this example illustrates how with DeepEval we can evaluate multi-turn dialogue quality, checking for relevance, context retention, and domain-specific safety rules.

Apart from these, DeepEval provides several other multi-turn evaluation metrics. We encourage readers to explore them in the documentation as a self-learning activity.

Introduction to LLM Metrics | DeepEval by Confident AI - The LLM Evaluation Framework

deepeval offers 50+ SOTA, ready-to-use metrics for you to quickly get started with. Essentially, while a test case represents the thing you’re trying to measur, the metric acts as the ruler based on a specific criteria of interest.

DeepEval Logo

Conversation simulation

Recap

In Part 9, we began exploring the evaluation space of LLM applications, laying the groundwork, covering challenges and a practical taxonomy of evaluation methods.

We began by examining why LLM evaluation is different from traditional software or classical ML evaluation. Unlike deterministic systems, LLMs generate open-ended, probabilistic outputs. This introduces subjectivity, non-determinism, multi-dimensional quality criteria, and emergent failure modes such as hallucinations and bias.

From there, we introduced a structured taxonomy of evaluation methods: intrinsic, deterministic, and subjective.

We first explored intrinsic metrics, such as entropy, cross-entropy, and perplexity. These metrics quantify how well a model approximates the true data distribution. The key takeaway was that intrinsic metrics are measure of language modeling, not task-level usefulness.

We then moved to deterministic evaluation methods, which apply when ground truth or structural correctness exists. These included functional correctness, exact match and classification metrics, n-gram overlap metrics such as BLEU and ROUGE, embedding-based similarity metrics such as BERTScore, and strict format validation using schema checks.

We emphasized that deterministic metrics are objective and automatable, but often fail to capture nuances of open-ended language generation.

Next, we examined subjective evaluation methods, which become essential when outputs are open-ended and multi-dimensional. We discussed human evaluation (ratings, rankings, open feedback), along with its cost, scale and variability constraints.

We then explored LLM-as-a-judge, where strong models are prompted to score outputs based on explicit rubrics.

We also introduced pairwise comparative evaluation and Elo rating systems, explaining how relative comparisons can produce more stable rankings.

Finally, in the hands-on demo, we grounded these ideas in practice using a summarization task. Two candidate summaries were evaluated using BLEU, ROUGE, BERTScore, and an LLM-as-a-judge pipeline. The demo illustrated a crucial insight: no single metric captures the full picture.

Altogether, the chapter served as a foundational guide to the evaluation of LLM applications.

If you haven’t yet gone through Part 9, we strongly recommend reviewing it first, as it establishes the conceptual foundation essential for understanding the material we’re about to cover.

Read it here:

Evaluation: Fundamentals

LLMOps Part 9: A foundational guide to the evaluation of LLM applications, covering challenges and a practical taxonomy of evaluation methods.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will briefly discuss about benchmarks and build on the foundations from the last chapter and move toward understanding deeper task-specific methodologies, and tooling for evaluation of LLM applications.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Evaluation benchmarks

In the previous part, we’ve covered how to evaluate a model’s outputs on particular tasks. Now, for a moment, let’s zoom out and discuss how do we assess the general capabilities and quality of an LLM in a broad sense.

This is important for model selection in LLMOps. Before using a model or building an app on it, you want to know its strengths, weaknesses, and how it compares to alternatives. The field has developed many benchmarks and standardized tests for this purpose.

We’ll overview some common benchmarks and metrics used, and how to interpret them.

MMLU

MMLU (massive multitask language understanding) is a benchmark covering 57 subjects including history, math, science, law, etc., with difficulty from high school to expert level. It’s a set of multiple-choice questions. The metric is accuracy (%) on these questions.

It’s for measuring a model’s breadth of knowledge and reasoning. Non-expert human performance on MMLU is around 34%, whereas top models (Gemini 3 Pro, Claude Sonnet, etc.) go beyond 90% on it. If your use case requires broad knowledge, MMLU score is a good indicator. Many models report their MMLU in papers and on leaderboards.

As models continue to become more capable, many now achieve exceptionally high scores on the original MMLU benchmark. To better differentiate performance at the higher end, we have MMLU-Pro, which introduces a more challenging 10-option multiple-choice format.

This increased difficulty leads to more realistic scores, making it a more discriminative benchmark for evaluating current AI models.

HellaSwag

A commonsense reasoning benchmark where each question is a sentence or paragraph with a blank, and the model must choose the best ending from options.

It was designed adversarially such that surface cues are misleading, so models really need deeper understanding. Accuracy on HellaSwag reflects a model’s commonsense reasoning ability. It’s often used in open LLM leaderboards.

TruthfulQA

A benchmark to test whether a model tells the truth versus repeating common myths or falsehoods. It has questions (many about common misconceptions, false beliefs and tricky facts) and checks if the model’s answer is true. It tests models in both free-form generation and multiple-choice formats.

Many models often struggle with TruthfulQA, tending to output plausible but incorrect answers if a misconception is popular (because they mimic training data). For applications where factual accuracy is crucial, a model’s TruthfulQA score can be an important indicator, in evaluating model choice.

BIG-Bench

Beyond the Imitation Game benchmark (BIG-Bench) is a collection of over 200 diverse tasks. It includes everything from logical puzzles and mathematics to creative tasks and novel problem types.

BIG-Bench was used to test models for emergent abilities, i.e., tasks where performance jumps as models get larger. It’s huge and eclectic; not typically one number but a suite of metrics. A subset called BIG-Bench Hard (BBH) consists of 23 particularly challenging tasks unsolved by smaller models. An even complex one is BIG-Bench Extra Hard (BBEH).

If a model does well on BBEH tasks, it’s considered quite advanced. BIG-Bench is mostly research-focused, but you might reference it if comparing frontier models’ advanced reasoning capabilities.

Recap

In the previous chapter (LLMOps Part 8), we examined memory context and temporal awareness as critical components of context engineering within LLM systems.

We began by drawing a clear distinction between short-term and long-term memory.

Short-term memory was defined as immediate context within the session or more precisely, what's part of the "active" prompt. It is ephemeral, bounded, and directly readable by the model. We discussed practical strategies such as last-N turns and rolling summaries pattern that preserves coherence while keeping token growth under control.

We then defined long-term memory as information retained across sessions or that persists even as short-term context moves on. It is implemented through external storage such as vector databases.

Here, we emphasized a critical architectural principle: persistent information is not automatically usable. It must be retrieved, filtered, and injected into working context before the model can reason over it. We also explored storage strategies, retrieval frequency, caching, pruning and cost considerations.

From there, we shifted to dynamic and temporal context injection. Unlike stored memory, dynamic context is not pre-authored or stored, but assembled or updated in real-time based on the current state.

We showed how these injections move systems beyond static Q&A into adaptive, state-aware behavior. Here, temporal grounding for memory emerged as a central theme: without timestamps and recency-aware ranking, memory becomes unreliable. Time metadata transforms memory from an archive into a living representation of evolving user state.

Next, we implemented a basic hands-on memory-enabled assistant. The system combined short-term memory (turns + summary), long-term semantic memory (ChromaDB + embeddings) with timestamps, dynamic time injection, query rewriting for retrieval, memory relevance gating, and audit logging. The goal was to expose the mechanics of storing, retrieving, summarizing, injecting, and governing memory in a transparent and inspectable way. By walking through each component, we built intuition about how memory systems behave under the hood.

Finally, we addressed context failure modes that frequently arise when context is poorly engineered: context poisoning, distraction, clash, token wastage, and latency issues.

Altogether, the chapter reframed memory and dynamic context as engineered subsystems within LLM applications. Together, they extend the effective intelligence of the model and transform a stateless predictor into a stateful, adaptive and aware system.

If you haven’t yet gone through Part 8, we strongly recommend reviewing it first, as it helps maintain the natural learning flow of the series.

Read it here:

Context Engineering: Memory and Temporal Context

LLMOps Part 8: A concise overview of memory, dynamic and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and some of the common context failure modes in agentic applications.

Daily Dose of Data ScienceAvi Chawla

In this and the next few chapters, we will be exploring evaluation methods and approaches for LLM-based applications. This chapter, in particular, focuses on building a strong understanding of the fundamental concepts.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Unique challenges in LLM evaluation

Evaluating LLMs is fundamentally different, and often harder, than evaluating traditional software or even classical ML models. Unlike a deterministic function or classifier, an LLM generates open-ended, probabilistic outputs in natural language.

This introduces several challenges:

• Subjective and non-deterministic outputs: Many LLM tasks (creative writing, dialogue, etc.) don’t have a single objectively “correct” answer. An LLM’s response quality often lies on a spectrum, making strict pass/fail judgments difficult. Two answers can both be valid, or one may be “better” in style or clarity, a subjective call that’s hard to reduce to a simple metric. Moreover, identical prompts can yield different outputs on different runs due to randomness, so evaluation must account for variability.

• Lack of ground truth: For tasks like open-ended Q&A, or chatbot behavior, we often lack a perfect ground truth. Often human references are also just one of many possible good outputs. This complicates reference-based evaluation, measuring model output just against a fixed reference might undervalue valid answers that differ in wording or approach.

• Multifaceted quality criteria: The quality of an LLM output is multi-dimensional. It may need to be factually correct, relevant, coherent, concise, safe, stylistically appropriate, etc. No single scalar metric captures all these aspects. Hence evaluating “How good is this response?” often requires combining multiple criteria.

• Scale and automation needs: Human evaluation (e.g. having people read and score outputs) is considered the gold standard for subjective tasks, but it is slow, expensive, and not scalable for the rapid iteration cycles. LLM-driven applications might produce thousands of responses, or have new model versions daily. We can’t feasibly have humans check everything. We need automated or AI-assisted evaluation methods that approximate human judgment.

• Emergent behaviors and failure modes: LLMs can fail in unexpected ways, for example, hallucinating plausible-sounding but false information, making biased or toxic remarks, leaking system prompts, or failing on adversarial inputs. Evaluating these requires creative testing beyond standard accuracy metrics. We often must proactively design evals targeting potential failure modes to catch issues.

Hence, we can say, LLM evaluation requires a mix of exact, objective metrics for tasks where ground truth or certain fixed structure exists, and human-like judgment for open-ended tasks. We need to handle subjectivity, ensure broad coverage of quality aspects, and perform effective evaluation with clear judgment calls and contextual understanding.

Now, in the next section, we’ll discuss the types of evaluation methods that have emerged to meet these challenges.

Taxonomy of evaluation methods

Broadly, evaluation can be divided into three main categories:

• Intrinsic evaluation
• Deterministic evaluation
• Subjective evaluation

Let’s break down each category and the key techniques therein:

Intrinsic evaluation

Before we discuss evaluation for task-specific performance, it is critical that we also have a fair idea about the intrinsic metrics utilized during the pre-training and fine-tuning phases.

The capabilities of an LLM model are inextricably tied to its language modeling efficiency, which measures the mathematical divergence between the learned probability distribution and the true distribution of the training data.

Entropy

Entropy measures how much information, on average, a token carries. Higher the entropy, the more information a token carries, and hence more bits are needed to represent a token.

Let's illustrate this via a very intuitive example by Chip Huyen.

Figure (a): Consider you want to create a language to describe locations in a square. If your language has only two tokens, each token can tell you whether the location is up or down. Now, since there are only two tokens, one bit is sufficient to represent them, making the entropy of this language 1.

Figure (b): Now suppose, if your language has four tokens, each token can give you a more specific location: top-left, top-right, bottom-left, or bottom-right. However, since there are now four tokens, you need two bits to represent them, making the entropy of this language 2.

Hence we can say, entropy calculates how difficult it is to predict what comes next in a language. The lower a language’s entropy, i.e., the lesser the information a token carries, the more predictable that language.

In the example discussed above, the language with only two tokens is easier to predict since you have to predict from only two possible tokens compared to four in the case of the second language.

Hence we can conclude, entropy is highest when outcomes are equally likely and lowest when outcomes are deterministic. Thus, in language modeling:

• A highly structured domain (HTML, SQL, JSON) has lower entropy.
• Casual conversation or creative texts have higher entropy.

Cross entropy

When we train a language model on some data, the goal is to get the model to learn the distribution of the training data. A language model’s cross entropy on a dataset measures how difficult it is for the language model to predict what comes next in this data.

So if $P$ is the true distribution of the training data and $Q$ is the distribution learned by the language model, the cross-entropy $H(P,Q)$ is expressed mathematically as:

where, $H(P)$ is entropy of training data and $D_{KL}$ represents the Kullback-Leibler divergence, measuring how the learned distribution diverges from the true distribution.

Quick note for those who might not know:

KL divergence is calculated as follows:

It measures the expected extra information required to encode samples drawn from the true distribution $P$ when using an approximating distribution $Q$. In simple terms, it quantifies how inefficient $Q$ is at representing $P$.

Imagine this. Say $P$ and $Q$ were identical. This should result in zero inefficiency:

And, hence:

Implication: A language model cannot achieve a cross-entropy lower than the inherent entropy of the dataset ($H(P)$). This is the theoretical lower bound of performance.

A language model is trained to minimize its cross entropy with respect to the training data.

Perplexity

Perplexity (PPL) is the exponentiated cross-entropy, representing the model's absolute uncertainty when predicting the next token. If measured in bits (base 2):

Popular ML frameworks (TensorFlow, PyTorch, etc.), use natural log, making perplexity the exponential of $e$:

A lower perplexity value indicates that the model assigns a higher probability to the true token sequence, or we can also say, the more the uncertainty the model has in predicting what comes next in a given dataset, the higher the perplexity.

With this we have understood about the intrinsic metrics, now let's go ahead and take a look at deterministic evaluation methods.

Deterministic evaluation methods

Recap

In the previous chapter (part 7), we explored the discipline of context engineering: the practice of designing the information environment in which an LLM application operates.

We began by reframing the context window as a form of working memory, and showed why the core job of context engineering is maximizing signal under strict capacity constraints.

From there, we built a practical taxonomy of context types that show up in real systems: instruction context, query context, knowledge context (RAG), memory context, tool context, user-specific context, environmental and temporal context.

Next, we moved from "what context is" to "how context is constructed". We introduced modular, conditional context construction.

We explored chunking as a core design decision and walked through modern chunking strategies. We also clarified how chunking behaves differently for summarization-style workloads, where coverage replaces relevance as the optimization goal.

We then covered the standard retrieval stack: vector search as a candidate generator, the need to tune precision vs. recall, and improvements like contextual retrieval.

After that, we introduced summarization and context compression as context management tools, and discussed query-guided summarization, filtering, deduplication, structured compaction, and prompt compression techniques like LLMLingua.

Moving ahead, we examined re-ranking and selection, especially the common bi-encoder + cross-encoder pattern that retrieves broadly and then tightens precision with a slower but more accurate scorer.

Next, we also explored governance and assembly. The core takeaway was that context construction is not a single step, it is an engineered pipeline that balances relevance, coverage, and efficiency.

Finally, we ended with three hands-on demos:

• Semantic and AST-based chunking
• Bi-encoder retrieval with cross-encoder re-ranking
• Prompt compression with LLMLingua.

By the end of the chapter, we had moved from thinking of context as “extra text in the prompt” to seeing it as a structured, retrieval-driven, and governable system component, one that often determines whether an LLM application feels intelligent and dependable, or noisy and brittle.

If you haven’t yet gone through Part 7, we strongly recommend reviewing it first, as it helps maintain the natural learning flow of the series.

Read it here:

Context Engineering: An Introduction to the Information Environment for LLMs

LLMOps Part 7: A conceptual overview of context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs.

Daily Dose of Data ScienceAvi Chawla

In this chapter of discussing context engineering we will explore selected aspects of memory in LLM applications and dynamic and temporal context.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Memory systems

Humans have short-term and long-term memory; analogously, AI systems with LLMs can benefit from a notion of short-term context and long-term memory.

Short-term memory

By short-term we usually refer to the immediate context within the session or more precisely, what's part of the "active" prompt. This is basically the conversation history or any information stored directly in the prompt for the current inference.

It’s ephemeral by nature. Short-term memory covers the immediate dialogue (last few messages) and maybe a brief summary of earlier ones. It’s fast to access (the model reads it directly) but limited in capacity.

The design question is often how much of the recent conversation to include verbatim. A common approach is to do something like: include the last N turns. If over limit, start trimming the oldest. This works to keep coherence for recent references, but if the user goes “N messages ago we said XYZ”, the model might have already dropped that from short-term memory.

Here, as mentioned earlier, systems typically combine recent verbatim dialogue with a rolling summary of older context. But when even summaries start getting too generic and/or trimming is unavoidable, long-term memory is used to reintroduce relevant information (memories).

Long-term memory

This refers to information retained across sessions or that persists even as short-term context moves on. Typically implemented via external storage: a vector database (or a vector store, depending on use case) for semantic memories, a relational DB for structured data, etc.

Long-term memory allows the system to recall things that were said or learned earlier. For example, a support chatbot could “remember” a customer’s issue from yesterday when they come back today by looking it up in a user memory store.

Approach

Whenever something potentially important comes up in conversation (like the user shares a piece of information that could be useful), you embed that sentence and store it in a vector DB with metadata.

Later, you query the vector DB with the new user query to see if any stored information is relevant.

What to store in long-term memory

A core design decision is what actually gets stored. One option is to persist full conversation logs verbatim. This is the safest choice from a compliance, audit, or debugging perspective, because nothing is lost. The downside is obvious: raw logs grow quickly and are expensive to retrieve and inject into context.

A common alternative is to store summaries instead. After a conversation (or after a logical phase), the LLM generates a concise semantic summary capturing information, decisions, facts, and intent. This drastically compresses memory and keeps retrieval lightweight. The trade-off is information loss: details that seem unimportant today may become relevant later especially for compliance-sensitive use cases.

Hence, in practice, many robust (and compliance-sensitive) systems do both: full logs + summaries.

Full logs are archived in cheaper storage for safety, traceability and compliance, while summaries are what actually get retrieved and injected into prompts. This gives you compression for runtime use without permanently discarding information.

When to retrieve long-term memory

The simplest strategy is to retrieve long-term memory on every user query, using the current query (or conversation state) as the retrieval key. This is common because vector search is relatively cheap, and it avoids edge cases.

That said, if the conversation is clearly staying within a single topic for a while, you could retrieve memory only once at the start of that topic and reuse it across turns. This slightly reduces overhead but adds logic and state management complexity.

Caching retrieved memory

Another robust and widely used pattern is to introduce a memory cache that sits between the model and long-term storage. Instead of retrieving from long-term memory on every query, the system first checks this cache.

When relevant memories are fetched from long-term storage, they may be added to the cache directly or based on some threshold. For subsequent user queries, the system consults the cache first. If the cached memory is still relevant to the current conversation state, it can be reused directly without performing another long-term retrieval.

This has a few important benefits. It reduces repeated vector searches for the same information, overall lowers latency, and simplifies prompt construction during continuous conversations on the same topic. It also creates a natural separation between "active" context (what’s currently in use) and "archived" context (everything stored long term).

If the cache lookup fails or confidence in relevance drops, the system can always fall back to long-term memory retrieval and refresh the cache accordingly. In practice, this pattern gives you most of the robustness of per-turn retrieval while being more efficient and easier to reason about during multi-turn interactions.

Memory pruning and cleanup

Over time, long-term memory can accumulate noise: duplicated facts, outdated preferences, temporary context that no longer matters, or even incorrect entries. If left unchecked, this can pollute retrieval and degrade response quality.

To prevent this, memory systems should support pruning and maintenance, such as:

• Deduplicating near-identical entries
• Clustering and merging semantically similar memories
• Decaying or removing memories that are rarely retrieved
• Marking entries as outdated based on time or newer conflicting information

Cost considerations

Long-term memory adds cost in two places: storing embeddings and performing vector searches. However, these costs are usually modest compared to calls we make to LLMs, compounding token costs (especially in a fully sequential setup and multi-turn conversations).

In fact, effective memory retrieval can reduce overall cost. By injecting the right context early, the model is less likely to ask follow-up clarification, hallucinate, or drift off topic; each of which would otherwise require extra tokens and extra model calls.

The real cost is often complexity, not money. Memory systems require careful design, evaluation, and maintenance to ensure they help in the manner intended.

In summary, a well-designed memory system (short-term + long-term) extends the effective context of the LLM beyond a fixed window, enabling continuity and accumulation of knowledge over time. It’s a key part of context engineering for any stateful AI interaction.

Next, let’s discuss injecting dynamic and temporal context.

Dynamic and temporal context injection

So far, we talked about fetching context in a fairly static way (stored memory). But many applications require injecting dynamic context, information that changes with time or is generated on the fly, into the model’s prompt.

Dynamic Context means the context is not pre-authored or stored, but assembled or updated in real-time based on the current state.

Examples:

• The current date/time: Many prompts explicitly include date to help the model avoid confusion with its training cutoff and handle temporal questions.
• Real-time data: If a user asks “What’s the stock price of X right now?”, you need to fetch that via an API. The result is dynamic context, it didn’t exist until now. Similarly, weather info, news headlines, latest database entries; all are dynamic.
• User’s current interaction state: If the user is filling a form and the model assists, the partially filled fields could be dynamic context on each turn. Or in a code assistant, the current file contents or cursor position is context that changes as user edits.
• Tool results: In an agent performing actions, each time it runs a tool (API call, code execution), the output is new context fed in. That we covered under tool context, but it’s inherently dynamic because it’s a function of user input and current environment.
• Generative chain-of-thought: Some approaches feed the model’s own prior reasoning steps back in as context for further reasoning (like reflecting or refining). This is also dynamic self-generated context.

Temporal context specifically refers to context that has a time dimension:

• The simplest temporal context is the current date/time injection, as mentioned.
• Another is knowledge that decays with time, for example, “the current top trending topics on Twitter” changes daily. Hence, we might implement strategic forgetting for time-limited facts.

• Temporal memory: as conversations progress, older context might be summarized (as we talked), effectively injecting a “summary so far” at some regular interval. That summary can be considered a temporal context injection technique, you compress older context over time.

Injection methods

A few common dynamic and temporal context injection methods are:

Event-driven context refresh

This means if certain events happen (time ticks, new data arrival), the context is updated.

For instance, if the conversation crosses midnight and date changed, maybe update the "Today’s date" context if it’s persistent. Or if a monitored data source (like stock price) changes significantly while the user is interacting, maybe spontaneously update context or notify the model. Usually though, it’s user-driven (user asks and system fetches at that time).

Scheduled context injection

In some agent scenarios, the agent might have tasks that run periodically and then feed results in.

For example, an agent that monitors an email inbox every hour and then in the next conversation turn mentions any new important email. That means the context pipeline might include a step “check for new emails (tool), if found, include summary in prompt.” So at different times, context content changes because environment changed.

User-specific dynamic context

If a user’s current context like location changes (they moved to a different city), ideally the system knows and updates what it tells the model.

If location is used in answers, and user moves, you want “User is in London now” context instead of the old “User in Paris.” This could be handled by always querying a profile service for current information at prompt time.

Memory aging

We touched on summarization as one method. Another is strategic forgetting (dropping if assumed no longer needed). Some systems do “forget” intentionally things that are resolved or won't come up again, to minimize risk of retrieval picking up unnecessary stuff.

Recap

In the previous chapter (part 6), we started our discussion by introducing prompt versioning. We explained why even small prompt changes can have outsized and unpredictable effects, and why treating prompts as non-versioned text could be dangerous.

From there, we laid out the core principles of prompt versioning: separating prompts from application code, enforcing immutability of prompt versions, adopting systematic versioning schemes, and use of metadata.

We also emphasized that prompt changes should be treated as runtime configuration changes, not code changes, and must be governed with the same rigor as any other critical system component.

We then expanded this discussion into prompt templates, the form in which prompts are most commonly used in real applications. We showed how templates enable dynamic prompt construction while preserving structure and consistency, and why different templates should exist for different formats and use cases.

Next, we moved into the security dimension with defensive prompting. We examined why prompt-based attacks matter, especially in high-stakes systems, and categorized common attack families.

Crucially, we reframed prompt security as a layered problem, spanning model-level behaviors, prompt-level constraints, and system-level safeguards, rather than something that can be “solved” with a single clever prompt.

Next, to ground the ideas of prompt versioning in practice, we walked through a hands-on prompt versioning workflow using Langfuse.

Finally, we explored a few modern and practical prompting ideas that help teams steer behavior beyond normal practices: verbalized sampling, role prompting and prompt repetition.

By the end of the chapter, we had moved from viewing prompts as static strings to understanding them as managed, versioned, and secure components of an LLM system.

If you haven’t yet gone through Part 6, we strongly recommend reviewing it first, as it helps maintain the natural learning flow of the series.

Read it here:

Context Engineering: Prompt Management, Defense, and Control

LLMOps Part 6: Exploring prompt versioning, defensive prompting, and techniques such as verbalized sampling, role prompting and more.

Daily Dose of Data ScienceAvi Chawla

In this chapter we zoom out from prompt engineering to the broader discipline of context engineering, and understand the much larger information environment that drives model behavior.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

Context engineering can be understood as a collection of methods and techniques used to supply information and instructions to large language models in applications in a deliberate and structured way.

An LLM’s context window functions like its working memory. It contains the prompt, response, recent conversation history, retrieved knowledge, etc. This window is finite, which makes efficient use of that space critical.

Context engineering is therefore about maximizing signal within limited capacity, deciding what information to include for a given model invocation, what to leave out, and how to structure what remains so the model can reason effectively.

The ultimate goal of context engineering is to bridge the gap between a static model and the dynamic world in which an application operates. Through context, the model can be connected to external knowledge via retrieval, to past interactions via memory, to real-time data or actions via tools and perception, and to user-specific details such as preferences or profiles.

In production systems, this is what often determines whether an AI system feels intelligent and useful, or disconnected and unreliable.

Now, to make this more concrete, let’s establish a taxonomy of context types that we might include in an LLM’s input, depending on the use case.

Taxonomy of context

When we talk about “context” for LLM applications, it’s not one monolithic thing. We can break it down into categories, each representing a type of information that might be included in the model’s input.

Here’s a taxonomy of context often considered in LLMOps:

Instruction context

This is the part of the input that tells the model what its task or role is. It includes the system prompt or any high-level directives. Essentially, it’s the instructions or rules governing the model’s behavior.

It usually does not contain factual data for the task itself, but rather sets the stage and defines how the model should operate (persona, style, boundaries, etc.). Instruction context is crucial to align the model’s responses with the desired tone and policy.

Without it, the model might default to a generic behavior or something undesired. In systems engineering terms, this is part of the configuration for the model on each request. It’s often static or changes infrequently.

Query or user context

This is the immediate input from the user: their question, command, or message. It defines the problem to solve right now. If we consider a chat, the latest user message is the core of the query context.

Knowledge context

This is additional information provided to help answer the query, typically fetched from some database or documents. It’s often implemented via retrieval-augmented generation (RAG).

Knowledge context can include company documentation, knowledge base articles, textbook snippets, etc. It ensures the model isn’t just relying on parametric memory (which might be outdated or insufficient) but has access to the latest or specific facts.

For example, if a user asks “What’s our refund policy?”, your system might pull the refund policy text from your FAQ and include it in the prompt as context. When building pipelines, a lot of effort goes into getting this knowledge context right (finding the right pieces, chunking them, etc.).

Note: We'll not be covering everything about RAG in depth here, since we already have a detailed course on it. We recommend the readers unfamiliar with RAG checking that out:

RAG Course

A hands-on series on building production-grade RAG systems. It covers the fundamentals of RAG, naive RAG, RAG evaluation, RAG optimization, Multimodal RAG, Graph RAG, Vision RAG, etc. (with implementation).

Daily Dose of Data ScienceAvi Chawla

Memory context

This refers to including previous interactions or events so the model has continuity. In a chat scenario, the memory context would be the past dialogue turns that are relevant to keep the conversation coherent.

In a more general setting, memory might include things like what actions the agent has already taken, what the intermediate results were, or any conclusions already reached. Broadly, we break memory into:

• Short-term: talks about what’s in the active prompt (or immediate context within the session).

• Long-term: knowledge and experience across sessions (or what’s stored elsewhere and fetched).

But conceptually, memory context is anything that happened before now that the model should “remember” in order to respond appropriately. For example, if earlier the user said their name or preferences, you want the model to remember that later in the session.

Or suppose if this is the 5th step in a multi-step response chain, the model should have the prior steps’ outcomes in context.

Checkout the articles linked below for a detailed discussion on memory for AI systems:

A Practical Deep Dive Into Memory for Agentic Systems (Part A)

AI Agents Crash Course—Part 8 (with implementation).

Daily Dose of Data ScienceAvi Chawla

A Practical Deep Dive Into Memory for Agentic Systems (Part B)

AI Agents Crash Course—Part 9 (with implementation).

Daily Dose of Data ScienceAvi Chawla

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part A)

AI Agents Crash Course—Part 15 (with implementation).

Daily Dose of Data ScienceAvi Chawla

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part B)

AI Agents Crash Course—Part 16 (with implementation).

Daily Dose of Data ScienceAvi Chawla

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part C)

AI Agents Crash Course—Part 17 (with implementation).

Daily Dose of Data ScienceAvi Chawla

Tool context

When an LLM uses external tools (like search engines, calculators, databases, code execution), the outputs of those tools become context for the model. In an agent loop, after the model decides on an action and you execute it, you feed the result back into the model as an observation. This is context in the sense that it’s additional info. given to the model mid-task.

For instance, the model says “Action: call weather API for London today,” your system does it and gets “It’s 15°C and sunny,” and then you supply to the model: “Observation: 15°C and sunny in London.” Now the model has that data in its context for the next step, perhaps to report it to the user.

Tool context thus is a dynamic, often real-time form of context that expands what the model can do beyond its training data.

Note: We'll not be covering tooling in depth here, since we already have detailed discussion on tool use with LLMs as part of our Agents and MCP courses. We recommend the readers unfamiliar with tooling checking out the AI Agents course:

AI Agents Course

A series of technical deep dives on AI Agents that covers fundamentals and backgrounds, Flows, Knowledge, Memory, implementation of Agentic Patterns from scratch, and much more (with implementations).

Daily Dose of Data ScienceAvi Chawla

User-specific context

This category includes any information tailored to the specific user or session that can help personalize or contextualize the response. It might be a user profile (e.g., their name, location, membership status), their past interactions or preferences, or even dynamic context like current date/time as it relates to the user (maybe the user’s local time for greeting, etc.).

Including user-specific context allows the model to tailor answers, for instance, if you know the user’s proficiency level is beginner, you might include that as context: “The user is a beginner in programming.” Then the model can adjust the complexity of its answer accordingly.

Another example: if the user has an open support ticket, the context might include summary of that issue so the model doesn’t ask them to repeat it.

Environmental and temporal context

Sometimes listed separately, this includes things like the current date and time, the platform or device info., or other environmental variables that might influence the model’s response. For example, telling the model "Today’s date is 2026-01-19". Why do this? If the user asks “Is the event coming up soon?”, the model knowing today’s date can determine that.

Now that we have defined the different types of context, let’s move on to some important techniques and principles for context construction and management of these different context elements.

Principles and techniques for context construction

Context construction involves several principles and techniques to assemble model input from multiple information sources.

Rather than a single fixed procedure, real-world systems employ modular, conditional stages that are composed differently depending on the query, task, and operational constraints.

This section surveys commonly used techniques in practice, with a particular focus on retrieval-centric context assembly.

Typical modules/stages involved:

Recap

In the previous chapter (part 5), we laid the foundations of prompt engineering, a subset of context engineering.

We started by building a basic understanding of prompt engineering and context engineering, and explained why prompts are best viewed as control surfaces that shape model behavior probabilistically.

We then grounded this idea with the fundamentals of in-context learning, showing how zero-shot and few-shot prompting work.

Next, we introduced a systematic prompt development workflow. This framed prompting similar to software development: specification, debugging, evaluation, and versioning.

From there, we covered the major prompt types used in real applications: system prompts, user prompts, few-shot examples, instruction + context prompts, and formatting prompts.

We then moved beyond basics into widely used prompting techniques: chain-of-thought prompting, ReAct as a reasoning-and-tool-use control loop, and self-consistency as an “ensemble at inference time” to improve reliability when correctness matters.

Finally, we tied the concepts together with a hands-on implementation: a small math-solver pipeline that combines system instructions, strict formatting, chain-of-thought, and demonstrates a pattern where reasoning is logged for audits while only the clean final answer is shown to the user.

By the end of the chapter, we had a clear mental model of how prompts influence model behavior, how to iterate on prompts, and how advanced prompting patterns trade simplicity for capability.

If you haven’t yet gone through Part 5, we strongly recommend reviewing it first, as it lays the conceptual groundwork for everything that follows.

Read it here:

Context Engineering: Foundations, Categories, and Techniques of Prompt Engineering

LLMOps Part 5: An introduction to prompt engineering (a subset of context engineering), covering prompt types, the prompt development workflow, and key techniques in the field.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will be going deeper into prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting and more.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Prompt versioning

Prompt versioning means treating each prompt as a versioned artifact with a clear history and explicit relationships to how the system behaves. In production environments, prompts are as critical as code and configuration.

Small changes in wording, structure, or constraints can significantly alter model outputs, sometimes in non-obvious ways.

Without proper versioning, we lose traceability, rollback capability, reproducibility, and the ability to govern or reason about prompt-driven behavior. Debugging becomes guesswork, and even minor prompt tweaks can silently introduce regressions.

A production-ready approach to prompt versioning follows a set of core principles as discussed below:

Separate prompts from application code

Prompts should not live inline inside application logic. Embedding prompt text directly in functions or API calls tightly couples behavior changes to code deploys.

Instead, prompts should be stored externally, for example as files in a repository, entries in a database, or records in a registry.

This separation of concerns allows prompts to evolve independently of code, reduces the risk of unintended rollouts, and makes prompt changes more visible and auditable.

Immutable prompt versions

Each prompt version should be immutable. Once a version is created, it must never be edited in place.

If a prompt needs to change, even slightly, a new version should be created. This mirrors how release artifacts are handled in mature software systems. Any reference to a version identifier must always resolve to the same prompt text and context.

Immutability is what makes logs, evaluations, audits, and incident investigations trustworthy. Without it, historical behavior cannot be reliably reconstructed.

Clear and systematic versioning

Prompt versions should follow a consistent and explicit versioning scheme. Many teams adopt semantic versioning using a three-part identifier such as major.minor.patch like 1.0.0 where:

• A major version indicates a breaking or structural change in prompt behavior.
• A minor version reflects additive or behavioral improvements.
• A patch version captures small refinements or wording tweaks.

This structure communicates intent and risk to anyone consuming the prompt and aligns prompt evolution with established engineering conventions.

Metadata and context tracking

Versioning is not just about storing text. Each prompt version should carry metadata that explains its purpose and provenance.

At minimum, metadata should include who created or modified the prompt, when it was changed, target model and parameters.

In more mature systems where large teams needs to collaborate, this can expand to include linked evaluation results, supported use cases, and environment tags such as development, staging, or production.

Testing and evaluation as a gate

Prompt changes and versions should be evaluated using consistent testing.

Evaluation may include automated test sets, task-specific metrics, failure analysis, or structured human review. Results should be recorded and used as promotion gates.

Regression handling and rollback

Because LLM outputs are probabilistic, even small prompt changes can introduce unexpected regressions. For this reason, prompt updates must be treated as runtime configuration changes, not code changes. This is another reason why decoupling of prompts from application logic is important.

A robust prompt versioning strategy includes an immediate rollback mechanism. If a new prompt version causes degraded quality, incorrect behavior, or format violations, we should be able to switch back to the last known stable prompt at runtime, without redeploying the application or rebuilding infrastructure.

This is typically achieved by resolving the “active” prompt version dynamically (for example, via an alias) rather than hard-coding a specific version in the application. Rolling back then becomes a simple pointer change, similar to reverting a feature flag.

Prompt regressions should be handled with the same rigor as code regressions: monitored, quickly reversible, and isolated from deployment cycles.

To summarize: prompt versioning in production is fundamentally about explicit management. Prompts should be stored separately from code, versioned immutably, enriched with metadata, evaluated systematically, deployed through controlled aliases and monitored in operation. When treated this way, prompts evolve predictably, just like any other core component in a well-engineered production system.

With this, we have covered the key concepts of prompt versioning and will explore them further in the hands-on section of this chapter. Let us now move on to another core topic: prompt templates and their role in prompt management.

Prompt templates

Most real-world LLM applications need to format prompts dynamically: inserting user input or contextual data into the prompt. Hardcoding a single static prompt is rarely enough.

This is where prompt templates come in. A prompt template is essentially a string with placeholders that get filled in at runtime. Managing these templates well is important for clarity and consistency.

You can use simple string formatting or more sophisticated template engines to manage prompts. For instance, consider the simple template below:

Here {itinerary_details} is a placeholder for data that will be plugged in (e.g., the user’s flight info). By using a template, you ensure the structure and wording of the prompt remain consistent every time, and only the variable parts change. This reduces human error.

Organizing templates in code or config files is also useful. For example, you might have a YAML or JSON file that stores prompts with keys, and your application loads them.

This ties back to prompt versioning, templates can be versioned just like full prompts. In fact templates are the prompts that are mostly in use and talked about for AI applications.

However, with templates, there are two very important things that we must keep in mind:

Recap

In Part 4, we explored key decoding strategies, sampling parameters, and the general lifecycle of LLM-based applications.

We began by introducing the concept of decoding, and how it functions. We explored the key decoding strategies in detail: greedy decoding, beam search, top-K, top-P, and min-P strategies.

We then went ahead and understood the some of the major generation parameters, like temperature, top-P, top-K, max tokens, etc. in detail.

We also examined the complete text generation workflow and sketched a mental map of how LLMs function.

Moving ahead, we grounded the learned concepts about decoding and generation parameters with hands-on experiments. We compared the greedy and beam search decoding strategies, and showed how they operate at the token level.

We also compared different values for generation parameters (temperature, top-P, top-K), clearly demonstrating the influence the model’s responses.

Finally, we transitioned towards more of LLMOps by exploring the general lifecycle of LLM-based applications and its different stages.

By the end of Part 4, we had developed a clear understanding of the core concepts and components underlying LLMs, along with a strong LLMOps mindset gained through understanding the application lifecycle.

Although this chapter is not strongly related to the previous part, we strongly recommend reviewing Part 4 first, as it helps maintain the natural learning flow of the series.

Read it here:

Building Blocks of LLMs: Decoding, Generation Parameters, and the LLM Application Lifecycle

LLMOps Part 4: An exploration of key decoding strategies, sampling parameters, and the general lifecycle of LLM-based applications.

Daily Dose of Data ScienceAvi Chawla

In this and the next few chapters, we will be exploring context (and prompt) engineering. This chapter, in particular, focuses on the fundamentals of prompt engineering.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

Large Language Models (LLMs) have unlocked powerful new capabilities in software, but harnessing them in production systems (LLMOps) requires more than just plugging in an API.

Two critical disciplines have emerged in recent years:

• Prompt engineering
• Context engineering

Prompt engineering focuses on how we design the textual inputs (prompts) to guide an LLM’s behavior, while context engineering is about managing the entire information flow, structuring the surrounding data, tools, and environment that feed into the model. Together, these techniques enable us to build reliable and efficient LLM-powered applications.

Now let’s go ahead and explore the core concepts of prompt engineering, a key subdomain of context engineering. We will begin with the fundamental principles, followed by a practical workflow for developing effective prompts. Finally, we will examine various prompt types and advanced prompting techniques.

Fundamentals of prompting

A prompt is the input given to an LLM to draw out a response, it can be a question, instruction, or any text the model completes or responds to.

From an engineering perspective, a prompt is not just casual text; it is a piece of logic we design to program the model’s behavior (often called “soft programming”).

In essence, prompt engineering means writing instructions that effectively leverage the knowledge already present in the model. Unlike traditional software where we write deterministic code, prompting is more of an interactive, probabilistic programming: we guide a black-box model with instructions, then observe how it behaves, and refine our approach.

Why is prompt engineering needed?

LLMs are trained to continue text in a plausible way, but to get the specific output we want, we must craft the input skillfully. A poorly designed prompt can lead to irrelevant, inconsistent or incorrect outputs, even from a very capable model.

On the other hand, a well-designed prompt can make even a weaker model perform surprisingly well on a given task. Prompt engineering has therefore become “the skill of designing prompts that guide a generative AI model toward the kind of response you actually want”.

The system perspective

From a system standpoint, prompting is a first-class component of the application. We treat prompts like code: we design them methodically, test them, version-control them, and continuously improve them.

Importantly, prompt engineering is often the most accessible way to adapt an LLM to your needs without model retraining. It requires no updates to the model’s weights, instead, you leverage the model’s already present knowledge by providing instructions and context.

This makes prompt engineering fast to iterate on and deploy, which is why many early LLM applications relied solely on prompting. However, it’s not a silver bullet, more complex tasks often require going beyond skillful prompting to exploring a broader context pipeline, fine-tuning, or additional data.

In-context learning

Prompting is closely tied to the concept of in-context learning. Researchers discovered with GPT-3 that large models can learn tasks from examples given in the prompt itself, without any parameter updates.

For instance, if you prompt the model with a few example question-answer pairs (few-shot prompting), the model can infer the pattern and apply it to a new question. If you provide no examples, expecting the model to solve the task from just instructions, that’s zero-shot prompting.

Few-shot prompting (sometimes called few-shot in-context learning) often improves accuracy by showing the model what format or style of answer is expected. For example, adding 5 examples makes it a “5-shot” prompt. GPT-3’s paper “Language Models are Few-Shot Learners” highlighted this ability.

However, there are diminishing returns with newer advanced models: experiments have shown GPT-4 sometimes doesn’t gain much from few-shot examples on certain tasks compared to zero-shot. This is likely because newer models are better at following instructions out-of-the-box, so a clear zero-shot instruction often suffices for them.

But in niche domains, a few examples can still boost performance significantly if the model’s training data lacked those patterns. In practice, you should experiment to find the optimal number of examples for your task: balancing performance versus the added prompt length (which increases cost and latency).

Note that few-shot prompts use up context space, so you can’t include too many examples given the context window is limited.

In summary, the fundamental idea is that prompts provide both instructions and context to the model. They can include a description of the task, relevant information or data, and examples of the task being performed.

Overall, prompt engineering is fundamentally about constructing these elements in a way that the model reliably produces the desired behavior. We now turn to a systematic approach to developing prompts.

Systematic prompt development workflow

Developing an effective prompt is an iterative engineering process. Rather than guess-and-check ad hoc, it helps to follow a structured workflow:

Define the task and success criteria

Start by clearly defining what you want the model to do (e.g., “extract the total price from an invoice email and output a JSON”). Identify what a correct output looks like and any constraints (formatting, tone, length, etc.).

This is analogous to writing a spec; it will guide your prompt design and evaluation criteria.

Draft an initial prompt

Create a first version of the prompt. At a minimum, state the instruction clearly. Depending on the task, you might include some examples or a specific format.

Recap

Before we dive into Part 4, let’s briefly recap what we covered in the previous chapter.

In Part 3, we moved beyond the foundational translation layer of tokenization and embeddings, and stepped into the core mechanism that actually drives the intelligence, particularly attention.

We began by introducing the attention mechanism, the central innovation behind transformers. We explored self-attention in detail. We broke down how input embeddings are projected into Query, Key, and Value vectors, and how scaled dot-product attention computes relevance scores between tokens.

To make this concrete, we also walked through a small illustrative example with manually defined $Q$, $K$, and $V$ vectors.

We then extended the discussion to multi-head attention and saw why a single attention computation is insufficient to capture the many parallel relationships present in language, and how splitting the embedding space into multiple subspaces allows different heads to focus on different aspects.

We also examined how head outputs are concatenated and why the final output projection matrix $W^O$ is necessary to recombine and mix information across heads.

Next, we introduced causal masking, a critical concept for autoregressive language models. We explained why models must be prevented from attending to future tokens during training, and how this is enforced by adding an upper-triangular mask of negative infinity values before the softmax operation.

After covering attention, we broadened our perspective to architectural choices. We contrasted dense transformer architecture with mixture-of-experts (MoE) models, explaining how MoE introduces sparse activation through expert selection.

We then shifted from architecture to learning dynamics by examining pretraining and fine-tuning. Pretraining was portrayed as learning the statistical distribution of language via next-token prediction at massive scale, producing a capable but indifferent model.

Fine-tuning, particularly instruction tuning, was presented as behavioral conditioning that reshapes this distribution, making the model cooperative, task-aware, and aligned with user intent.

Finally, we grounded these ideas with hands-on experiments. We inspected next-token probability distributions to show that model outputs are fundamentally probabilistic, and that even a single-word change in a prompt can reshape the distribution.

We also directly compared pretrained and instruction-tuned models under identical prompts, clearly demonstrating the difference between text completion and instruction-following behavior.

By the end of Part 3, we had developed a clear understanding of how tokens exchange information, how architectural choices affect efficiency and behavior, and why pretraining and fine-tuning lead to different model personalities.

If you haven’t yet gone through Part 3, we strongly recommend reviewing it first, as it lays the conceptual groundwork for everything that follows.

Read it here:

Building Blocks of LLMs: Attention, Architectural Designs and Training

LLMOps Part 3: A focused look at the core ideas behind attention mechanism, transformer and mixture-of-experts architectures, and model pretraining and fine-tuning.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will be exploring decoding strategies, generation parameters, and the broader lifecycle of LLM-based applications.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

LLM decoding

There is a common misconception that large language models directly produce text. In reality, they do not generate text outright. Instead, at each step, they compute logits, which are scores assigned to every token in the model’s vocabulary. These logits are then converted into probabilities using the softmax function.

As we saw in the hands-on section of the previous chapter, given a sequence of tokens, each possible next token has an associated probability of being selected. However, how the actual choice of the next token is made has not been discussed by us so far.

Large language models that are autoregressive predict the next token based on all previously generated tokens. Consider a sequence of tokens $w = w_1, w_2, \ldots, w_t$. The joint probability of the entire sequence can be factorized using the chain rule of probability as:

For each token $w_i$, the term $P(w_i \mid w_1, \ldots, w_{i-1})$ represents the conditional probability of that token given the preceding context. At every generation step, the model computes this conditional probability for every token in its vocabulary.

This naturally leads to an important question: how do we use these probabilities to actually generate text?

This is where decoding strategies come into play. Decoding strategies define how a single token is selected from the probability distribution at each step.

Fundamentally, decoding is the process of converting the model’s raw numerical outputs into human-readable text.

While the model provides a probability distribution over the entire vocabulary, the decoding strategy determines which specific token is chosen and appended to the sequence before moving on to the next step.

How LLM decoding works (in simple language)?

As we already know by now, at its core, an LLM is a next-token predictor that, given a sequence of words or sub-words (tokens), calculates possibilities over its entire vocabulary for what the next most likely token should be.

Decoding strategies are the set of rules used to translate these raw probabilities into coherent, human-readable text by selecting a token. The process is autoregressive: each newly chosen token is added to the sequence and used as part of the input for predicting the subsequent token, continuing until a stopping condition (like an "end-of-sequence" token or maximum length) is met.

Decoding strategies

Now that we know the fundamental meaning of decoding, let's go ahead and discuss the four major decoding strategies themselves. These strategies define how the next token is chosen at each step. We will explain each and compare their behavior:

Greedy decoding

Greedy decoding always picks the highest probability token at each step:

It’s simple and fast (no extra search). The advantage is that it produces the single most likely sequence according to the model’s learned distribution (or at least a local optimum of that).

However, greedy outputs often suffer from repetition or blandness. This is especially true for open-ended generation, where the model’s distribution has a long tail.

For example, a model can get stuck repeating some statement because the model keeps choosing the highest probability continuation, which might circle back to a previous phrase.

Greedy decoding can be appropriate in more constrained tasks.

For instance, in translation, greedy often does well for language pairs with monotonic alignment, and it’s used in real-time systems due to speed. But for tasks like dialog or storytelling, greedy is usually not what you want if you expect diverse and engaging outputs.

Beam search

Beam search is an extension of greedy search that keeps track of multiple hypotheses (paths) at each step instead of one.

If beam width is $B$, it will explore the top $B$ tokens for the first word, then for each of those, explore top $B$ for the second word (so $B^2$ combinations, but it keeps only the top B sequences by total probability), and so on.

In essence, beam search tries to approximate the globally most likely sequence under the model, rather than making greedy local choices. This often yields better results in tasks where the model’s probability correlates with quality (like translation or summarization).

However, beam search has known downsides for LLMs in open generation:

• It tends to produce repetitive outputs too (the model maximizing probability often means it finds a loop it likes). In fact, beam search can be worse than greedy search for repetition. It has been observed that beyond a certain beam size, the output quality significantly decreases for storytelling tasks.
• Beam search can result in length bias: the model often assigns a higher probability to shorter sequences (since each additional token multiplies probabilities, lowering it). Without a length penalty, beam search may prefer ending early to get a higher average score. This is why a length penalty is introduced to encourage longer outputs when appropriate.

In practice, beam search is favored in tasks where precision and logical consistency trump creativity, for example, in summarization (you want the most likely fluent summary), translation, or generating code (where randomness could introduce errors). Meanwhile, for conversational AI or creative generation, beam search can make the model too rigid and prone to generic responses, so sampling is preferred.

Top-K sampling

We will discuss the top-K and top-P parameters in a later section. For now, it’s enough to understand that with top-K sampling, at each generation step, the model samples from at most the top K most probable options.

It's a simple truncation: the low-probability tail is cut off. This avoids weird tokens but still leaves randomness among the top tokens.

Note that if K is large (e.g. K = 100 or so), it’s almost like full sampling except excluding really unlikely picks, which hardly changes the distribution. If K is very small (e.g. K = 2 or 3), the model becomes only a bit random, it’s like “restricted sampling” where it might alternate between a couple of likely continuations, leading to some limited variation but not a lot.

This type of strategy tends to improve output quality and coherence because low-probability tokens often are low for a reason (either they are nonsensical in context or extremely rare completions). As a result, top-K sampling can reduce the chance of nonsensical completions while still allowing the model to surprise us by not always picking the top option.

However, like every strategy, there are certain gray areas here too. One drawback is that choosing K is not obvious. A fixed K might be too high in some contexts and too low in others. For example, if the model is very sure about the next word (distribution is peaked), then whether K = 500 or K = 5 doesn’t matter because maybe only 3 tokens had significant probability anyway. But if the distribution is flat (lots of possibilities), limiting to K might prematurely cut off some plausible options.

Despite the above limitation, top-K is computationally efficient and easy to implement, so it’s quite popular.

Nucleus sampling

Nucleus (top-P) sampling includes the smallest set of possible tokens whose cumulative probability is greater than or equal to P.

Nucleus sampling addresses the adaptiveness issue. With top-P, you might sample from 2 tokens in one case (if the model is confident) or 20 in another (if unsure), whatever number is needed to reach the cumulative probability P.

This ensures that the tail is cut off based on significance, not an arbitrary count. So in general, top-P tends to preserve more contextually appropriate diversity. If a model is 90% sure about something, top-P = 0.9 will basically become greedy (only that token considered). If a model is very uncertain, top-P=0.9 might allow many options, reflecting genuine ambiguity.

The quality of top-P outputs is often very good for conversational and creative tasks. It was shown to produce fluent text with less likelihood of incoherence compared to unfiltered random sampling.

That said, one still needs to be reasonable about the choice of P (too high P, like 0.99, brings back long-tail weirdness; too low P, like 0.7, might cut off some normal continuations).

Min-p sampling

Min-P sampling is a dynamic truncation method that adjusts the sampling threshold based on the model's confidence at each decoding step.

Unlike top-P, which uses a fixed cumulative probability threshold, min-P looks at the probability of the most likely token and only keeps tokens that are at least a certain fraction (the min-P value) as likely.

So if your top token has 60% probability and min-P is set to 0.1, only tokens with at least 6% probability make the cut. But if the top token is just 20% confident, then the adapted 2% threshold lets many more candidates through.

Recap

Before we dive into Part 3, let’s briefly recap what we covered in the previous part of this course.

In Part 2, we started with our exploration of the concepts and the core building blocks of large language models. We began by understanding the fundamental concept behind tokenization.

Next, we discussed subword tokenization and understood why it is superior to word-level or character-level tokenization.

After that, we explored the three most common subword-based tokenization algorithms: byte-pair encoding, WordPiece algorithm and Unigram tokenization. We also learned about the byte-level BPE and understood how it helps in solving the OOV problem.

After tokenization, we examined and explored the concept of embeddings and why they are required. We also understood that tokenization is one stage of the two-stage critical translation process, with the other stage being embedding, which maps the tokens into a continuous vector space.

Moving forward, we explored the types of embeddings: token embeddings and positional embeddings. We understood that token embeddings are learned representations that map token IDs to vectors, and positional embeddings inject position information.

Next, we examined and learned about the two types of positional embeddings: absolute positional embeddings and relative positional embeddings.

Finally, we took a hands-on approach to tokenization and embeddings. We compared different GPT tokenizers and examined how token embeddings function as a lookup table.

By the end of Part 2, we had a clear understanding of the two-stage translation process. Tokenization converts raw text into discrete symbols drawn from a finite vocabulary, while embeddings convert those symbols into continuous vector representations that neural networks operate on.

If you haven’t yet studied Part 2, we strongly recommend reviewing it first, as it establishes the conceptual foundation essential for understanding the material we’re about to cover.

Read it here:

Building Blocks of LLMs: Tokenization and Embeddings

LLMOps Part 2: A detailed walkthrough of tokenization, embeddings, and positional representations, building the foundational translation layer that enables LLMs to process and reason over text.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we shall further examine key components of large language models, focusing on the attention mechanism, the core differences between transformer and mixture-of-experts architectures, and the fundamentals of pretraining and fine-tuning.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Attention mechanism

“Attention” is the key innovation that enabled Transformers to leapfrog previous RNN-based models. At a high level, an attention mechanism lets a model dynamically weight the influence of other tokens when encoding a given token.

In other words, the model can attend to relevant parts of the sequence as it processes each position. Instead of a fixed-size memory, attention provides a flexible lookup: for each token, the model can refer to any other token’s representation to inform itself, with learnable weights indicating how much to use each other token.

Hence, in a nutshell, attention is the mathematical engine that allows the model to route information between tokens, enabling it to model long-range dependencies.

Self-attention

The most common form of attention is self-attention (specifically scaled dot-product self-attention). In self-attention, each position in the sequence sends queries and keys to every other position and receives back a weighted sum of values.

At its heart, it is a differentiable dictionary lookup. For every token in a sequence, the model projects the input embedding into three distinct vectors, concretely, for each token (at position $i$), the model computes:

• a Query vector $Q_i$​: Represents what the current token is looking for (e.g., a verb looking for its subject).
• a Key vector $K_i$: Represents what the current token "advertises" itself as (e.g., "I am a noun, plural").
• a Value vector $V_i$​: The actual content information the token holds.

Important note:

In a transformer, input token embeddings are linearly projected into Query (Q), Key (K), and Value (V) representations using three separate learned weight matrices.

Let the input embedding matrix be: $X \in \mathbb{R}^{n \times d_{\text{model}}}$, where $n$ is the sequence length and $d_{\text{model}}$ is the embedding dimension.

During training, the model learns three distinct weight matrices: $W_Q \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_K \in \mathbb{R}^{d_{\text{model}} \times d_k}$ and $W_V \in \mathbb{R}^{d_{\text{model}} \times d_v}$. Here, $d_k$ is the length of a query or key vector and $d_v$ is the length of a value vector.

The Query, Key, and Value matrices are computed as:

• $Q = X W_Q$
• $K = X W_K$
• $V = X W_V$

As mentioned above, $Q_i$, $K_i$ and $V_i$ are all derived from the embedding (or the output of the previous layer) via learned linear transformations. You can think of:

• The Query as the question this token is asking at this layer.
• The Key as the distilled information this token is offering to others.
• The Value as the actual content to be used if this token is attended to.

The attention weights between token $i$ and token $j$ are computed as the (scaled) dot product of $Q_i$​ and $K_j$​. Intuitively, this measures how relevant token $j$’s content is to token $i$’s query. All tokens $j$ are considered, and a softmax is applied to these dot products to obtain a nice probability distribution that sums to 1.

Deconstructing the Formula:

• $Q_i \cdot K_j$: Computes the dot product between the query of token $i$ and the key of token $j$, measuring their relevance.
• $\sqrt{d_k}$ (scaling factor): As the dimensionality $d_k$​ increases, dot-product magnitudes grow, pushing softmax into regions with very small gradients. Dividing by $\sqrt{d_k}$​​ keeps the variance of scores roughly constant and stabilizes training.
• Softmax: Applied over all $j$, it normalizes the attention scores for a fixed query $Q_i$​, producing a distribution of attention weights that sum to 1.
• $V_j$ (Aggregation): The output is a weighted sum of value vectors, producing a context-aware representation for token $i$.

Here's the formula depicted as a diagram:

Dummy example (illustrative)

Now, let's walk through a basic example to understand this better.

Suppose we have a tiny sequence of 2 tokens and a very small vector dimension (2-D for simplicity). We'll manually set some Query/Key/Value vectors to illustrate different attention scenarios:

• Token 1 has query $Q_1 = [1,\,0]$
• Token 2 has query $Q_2 = [1,\,1]$
• Token 1’s key $K_1 = [1,\,0]$
• Token 2’s key $K_2 = [0,\,1]$
• Token 1’s value $V_1 = [10,\,0]$
• Token 2’s value $V_2 = [0,\,10]$

Here, token 1’s query is [1,0] which perfectly matches token 1’s own key [1,0] but is orthogonal to token 2’s key.

Token 2’s query [1,1] is equally similar to both keys. Now we compute attention:

Here is the step-by-step breakdown using the summation formula explicitly for each token index $i$. This method highlights how a single token "looks at" every other token $j$ (including itself) to construct its own output.

Given:

• $d_k = 2$, so scaling factor $\sqrt{d_k} \approx 1.414$.

• Token 1 ($j=1$): $K_1 = [1, 0]$, $V_1 = [10, 0]$
• Token 2 ($j=2$): $K_2 = [0, 1]$, $V_2 = [0, 10]$

Let's compute the attention output for Token 1.

We want to find $\text{AttentionOutput}_1$, which will be a vector that denotes the representation of token 1 based on other tokens.

Thus, we fix $i=1$ and compute attention over all $j$ values → $j=1, 2$.

So the structure of the output is:

Next, we compute raw similarity scores using dot products:

We compare the query of token 1 with every key ($Q_1 \cdot K_j$).

So the raw scores are:

• $s_{1,1}$ = 1
• $s_{1,2}$ = 0

Next, we scale the scores by dividing each score by $\sqrt{2}$:

Moving on, Softmax is applied across the index $j$, while $i$ stays fixed.

Simplifying the exponential terms, we get the attention scores as:

Now we combine the value vectors ($V$) using these attention weights:

Similar to above, we can find $\text{AttentionOutput}_2$. Consider this as a self-learning exercise, and here are some hints and results for verification:

• We fix $i=2$ and sum over $j=1, 2$.
• Weight $\alpha_{2,1}$ will come out to be: $0.5$
• Weight $\alpha_{2,2}$ will come out to be: $0.5$
• Weighted sum of values: $[5, \, 5]$

Here's the complete example summarized in the form of a diagram:

Now that we know how the calculations work, let's answer the question, "What does the output actually depict?"

The attention output depicts a contextualized representation of each word based on its relationships (weights) with other words.

For token 1, the attention mechanism calculated weights of 0.67 for itself and 0.33 for token 2. This means the output vector is not just the static definition of $V_1$.

It is a compound concept built from a $67\%$ share of $V_1$ and a $33\%$ share of $V_2$. The model has kept the core identity of the word strong, but has added a slight flavor of the other token.

Similarly, for token 2, the attention mechanism calculated weights of 0.50 for itself and 0.50 for token 1. This means token 2’s representation depends equally on itself and token 1.

In summary, the outputs depict the flow of information. It shows exactly how much context each token absorbed from its neighbors to update its own representation.

With this, we understand the fundamental idea of self-attention. We can now extend this discussion by introducing multi-head attention, which builds on the same principle while enabling the model to capture information from multiple representation subspaces simultaneously.

Multi-head attention

In our previous example, we had a single attention mechanism processing the tokens. However, language is too complex for a single calculation to capture everything. A sentence contains grammatical structure, semantic meaning, and references.

To solve this, transformers use multi-head attention (MHA) in practice.

Hence, let's go ahead and understand the conceptual ideas behind multi-head attention.

The dimensionality split

This is where the projection mathematics becomes essential. We take the massive "main highway" of information ($d_{\text{model}}$) and split it into smaller "side roads" (subspaces).

• $d_{\text{model}}$ (the full embedding): Each input token starts as a vector $X \in \mathbb{R}^{d_{\text{model}}}$ (e.g., 512). This vector contains all token information entangled together.
• $h$ (number of heads): We split the workload into $h$ parallel heads (e.g., $h=8$).
• $d_k, d_v$ (the head dimensions): Each head operates on a lower-dimensional subspace. In standard architectures (like GPT/BERT), we divide the dimensions equally, say:

Learnable projections

For each head $i$, the model learns three matrices (which we discussed earlier also):

• $W_i^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}$
• $W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$
• $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$

The complete flow

Firstly, the input vector is split into parts, where each part denotes a head:

Each segment of the input goes through the standard attention formula independently, with its own weight matrices:

This results in 8 (total heads) separate vectors, each of size 64.

Next, we stack the output vectors side-by-side to restore the original width.

For example, size: $8 \times 64 = 512$ (Back to $d_{\text{model}}$).

Finally, we multiply by one last matrix, $W^O$.

This $W^O$ (output weight matrix) is the final learnable linear layer in the MHA block. You can think of it as the "manager" of the attention heads.

To understand $W^O$ properly, let's look at the sizes involved:

• As mentioned above, the MHA is the concatenated output of all 8 heads. Size: $8 \text{ heads} \times 64 \text{ ($d_v$)} = 512$ dimensions.
• Desired output: The vector that goes to the next layer (the feed-forward network): $512$ ($d_{\text{model}}$).

Therefore, $W^O$ is a square matrix of size $512 \times 512$.

You might ask: "If the concatenation is already size 512, and we want size 512, why not just stop there? Why multiply by another matrix?"

There's a very critical reason. After concatenation, the vector is segmented:

• Dimensions 0-63: Contain only information from Head 1 (e.g., Grammar).
• Dimensions 64-127: Contain only information from Head 2 (e.g., Vocabulary).
• and so on.

These parts haven't "talked" to each other yet. They are just sitting side-by-side.

Multiplying by $W^O$ allows the model to take a weighted sum across all these segments. It mixes the learnings of different heads to create a unified representation:

Here is the complete concept in the form of a diagram:

With this, we have completed understanding the key details of multi-head attention (MHA) and attention in general. However, before moving on to another topic, there is one more crucial concept we need to understand: causal masking.

Causal masking

In Generative LLMs (like GPT), we use causal language modeling (CLM), where the objective is to predict the next token.

The Problem:

Recap

Before we dive into Part 2 of the LLMOps phase of our MLOps/LLMOps crash course, let’s briefly recap what we covered in the previous part of this course.

In Part 1, we explored the fundamental ideas behind AI engineering and LLMOps. We began by defining what LLMOps is and examining the goals it aims to achieve.

Next, we discussed foundation models and the AI application stack, focusing on its three layers.

After that, we explored several fundamental LLM concepts, including what “large” means in the context of LLMs, different types of language models, emergent abilities, prompting methods (zero-shot and few-shot), a brief overview of attention, and the limitations of LLMs.

Next, we examined the shift from traditional machine learning models to foundation model engineering, highlighting key changes such as adaptation strategies, increased compute demands, and new evaluation challenges.

Moving forward, we examined the key levers of AI engineering: instructions, context, and the model itself. We begin by optimizing prompts; if that is insufficient, we enrich the system with supporting context. Only when these approaches fall short do we turn to modifying the model.

Finally, we took a brief look at the key differences between MLOps and LLMOps.

By the end of Part 1, we had a clear understanding that, despite differences from MLOps, foundation model engineering and LLM-based application development are fundamentally systems engineering disciplines.

If you haven’t yet studied Part 1, we strongly recommend reviewing it first, as it establishes the conceptual foundation essential for understanding the material we’re about to cover.

Read it here:

The Complete LLMOps Blueprint: Foundations of AI Engineering and LLMs

LLMOps Crash Course Part 1 (MLOps/LLMOps Phase-II).

Daily Dose of Data ScienceAvi Chawla

In this chapter, and the next few, we will cover the fundamental concepts and core building blocks of large language models, analyzing each component and stage in detail.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

This chapter, and the next few ones, discuss the internal mechanics of large language models.

We will build a strong mental model of the technical fundamentals of LLMs: from how they convert text into numbers and apply attention, to how they are trained and generate text.

By the end of the subsequent set of chapters, you should be comfortable with concepts such as tokenization, embeddings, attention mechanisms, pretraining, and how LLMs generate text using various decoding strategies and parameters.

Additionally, we will discuss the general lifecycle of LLM-based applications, outlining the key stages involved from initial design and development through deployment and ongoing maintenance.

Let’s begin with the very first step in any language model pipeline: converting raw text into numerical representations usable by the model.

Tokenization

At the heart of an LLM’s input processing is tokenization: breaking text into discrete units called tokens, and assigning each a unique token ID, because before a neural network can process human language, that language must be translated into a format the machine can understand: numerical vectors.

Tokenization is one stage of this two-stage critical translation process, with the other stage being embedding, which maps the tokens into a continuous vector space.

Early techniques relied on word-level tokenization (splitting by spaces) or character-level tokenization. Both approaches have significant limitations.

Word-level tokenization results in massive vocabulary sizes and cannot handle unseen words, while character-level tokenization produces excessively long sequences that dilute semantic meaning and increase computational cost.

Hence, modern LLMs adopt subword tokenization, which strikes an optimal balance. Here, tokens are often words and subwords, depending on frequency of occurrence. For example, the sentence “Transformers are amazing!” might be tokenized into pieces like ["Transform", "ers", " are", " amazing", "!"].

More formally, the reasons behind using subword units are:

Subwords capture meaning

Unlike single characters, subword tokens can represent meaningful chunks of words. For instance, the word “cooking” can be split into tokens “cook” and “ing”, each carrying part of the meaning.

A purely character-based model would have to learn “cook” vs “cooking” relationships from scratch, whereas subwords provide a head start by preserving linguistic structure.

Vocabulary efficiency

There are far fewer unique subword tokens than full words. Using subword tokens reduces the vocabulary size dramatically compared to word-level tokenizers, which makes the model more efficient to train and use.

For example, instead of needing a vocabulary containing every form of a word (run, runs, running, ran, etc.), a tokenizer might include base tokens like “run” and suffix tokens like “ning” or past tense markers. This keeps the vocabulary (and therefore model size) manageable.

Most LLMs have vocabularies on the order of tens to hundreds of thousands of tokens, far smaller than the number of possible words.

Handling unknown or rare words

Subword tokenization naturally handles out-of-vocabulary words better by breaking them into pieces. If the model encounters a new word it wasn’t trained on, it can split it into familiar segments rather than treating it as an entirely unknown token.

This helps the model generalize to new words by understanding their components.

Now that we understand the basics of subword tokenization, let's take a quick look at the key techniques we use to implement subword tokenization.

Subword tokenization algorithms

The three dominant algorithms in this space are byte-pair encoding (BPE), WordPiece, and Unigram tokenization.

These algorithms build a vocabulary of tokens by starting from characters and iteratively merging common sequences to form longer tokens. The result is a fixed vocabulary where each token may be a word or a frequent subword.

Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) is the standard for models like GPT-2, GPT-3, GPT-4, and the Llama family. It is a frequency-based compression algorithm that iteratively merges the most frequent adjacent pairs of symbols into new tokens.

Mechanism

• Initialization: The vocabulary is initialized with all unique characters in the corpus.
• Statistical Counting: The algorithm scans the corpus and counts the frequency of all adjacent symbol pairs (e.g., "e" followed by "r").
• Merge Operation: The most frequent pair is merged into a new token (e.g., "er"). This new token is added to the vocabulary.
• Iteration: This process repeats until a pre-defined vocabulary size (hyperparameter) is reached.

Recap

In the previous part (Part 16) of this AI agents crash course, we focused on memory as a first-class design concern for agentic systems.

We began by understanding sequential memory, where the entire conversation history is appended and sent to the LLM on every turn.

Next, we introduced sliding window memory, which places a hard bound on context size by retaining only the most recent messages. This significantly stabilizes token usage and latency, but at the cost of forgetting older information once it falls outside the window.

To address this limitation, we moved on to summarization-based memory, where older conversation segments are compressed into a running summary while recent turns remain in full detail.

We then extended this idea further with compression and consolidation, introducing importance-aware memory management.

Throughout the chapter, we grounded every concept in a customer support chatbot example, analyzing how each memory strategy affects behavior, latency, and token consumption in realistic workflows.

If you haven’t gone through the previous chapter yet, we strongly recommend reading it first, as it lays the foundation for the next step in our memory optimization journey.

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part B)

AI Agents Crash Course—Part 16 (with implementation).

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will continue our discussion on memory optimization, learning about:

• Retrieval-based memory
• Hierarchical memory
• OS-like memory

Let’s continue.

Retrieval-based memory

So far, all the techniques we discussed so far have been short-term:

• Sequential memory: send the entire conversation so far.
• Sliding window: keep only the last few turns.
• Summarization: compress older parts into a running summary.

All of these live inside a single thread. Once the thread ends, that memory is gone.

For real, production-grade agentic systems, that’s not enough. You want your agent to:

• remember a user’s preferences across conversations
• recall past issues for the same user/account
• reuse knowledge learned last week in a new session

Instead of keeping everything inside the thread, we need to:

• store durable memory items in a long-term store
• retrieve only the relevant ones for the current query
• stitch them into the context alongside the current conversation

In Part 15, we briefly saw how long-term memory works in LangGraph using the store abstraction.

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part A)

AI Agents Crash Course—Part 15 (with implementation).

Daily Dose of Data ScienceAvi Chawla

Here we’ll walk through it step by step, and then wire it into a retrieval-based memory setup for our customer support agent.

Memory store in LangGraph

Checkpointers give us continuity inside a thread. As long as we use the same thread ID, LangGraph will restore the previous state and continue the conversation from where you left off.

But checkpointers alone cannot:

• share information between different threads
• carry knowledge across sessions or tickets
• build a persistent profile for a user

Imagine a user who opens three support tickets over a month:

• Ticket 1 – billing issue
• Ticket 2 – access issue
• Ticket 3 – workspace performance issue

With only checkpointers, each ticket is its own island. The agent has no way to reuse what it learned in Ticket 1 when answering Ticket 2 or 3.

This is the exact reason for the need to have stores as an external database that can store important conversations that can be retrieved on demand.

LangGraph stores long-term memories as JSON documents in a store.

Each memory is organized under:

• A namespace is like a folder and is represented as a tuple like (user_id, "memories").
• A key is like a file name within that folder.

Moreover, you can write and read from the store in any thread. In LangGraph we use the InMemoryStore to implement this.

The good thing about LangGraph is that it takes care of all the backend infrastructure required for the store implementation. We don't need to worry about managing it but it is still useful to understand the basics.

Here’s an implementation of the InMemoryStore:

Here:

• InMemoryStore creates a store which is essentially just a dictionary in memory.
• Memories are grouped by a namespace, which is a tuple of strings.

We use (user_id, "memories") here, but you could just as well use (project_id, "docs") or (team_id, "preferences") whatever suits your use case.

To save a memory, we use the put method:

In this snippet:

• memory_id is a unique key inside this namespace.
• memory is any JSON-serializable document such as a simple dictionary.
• put stores this (key, value) pair under the given namespace.

To read memories back, we use the search method:

Each memory item returned by search is an Item class with:

• value: the memory or user preference saved
• key: the unique key used (memory_id)
• namespace: the namespace for this memory
• created_at / updated_at: timestamps

That’s the core concept:

• put(namespace, key, value) to save a memory
• search(namespace, ...) to retrieve a memory

So far, search returns all items for a namespace and follows a simple retrieval strategy based on traditional keyword search. This doesn’t scale well if you have hundreds or thousands of memories per user.

In real systems, we want semantic retrieval that first converts text into vector embeddings and store these embeddings in an index. At query time, we embed the query and find the closest memories using a similarity metric.

Let us first configure our embedding model before going to the implementation part. We will again be using OpenRouter for this.

InMemoryStore can be configured with an embedding index like this:

Here:

• embed is an embedding function (OpenAI embeddings).
• dims is the dimensionality of the embedded vectors.
• fields tells the store which fields of our values to embed.
  • "food_preference" will embed the value of this key in the dict (i.e. I like pizza).
  • "$" is a catch-all for the entire object.

Now, when we put memories into the store, vectors are computed and stored behind the scenes. We can then issue semantic queries:

Here:

• query is a natural language search string.
• limit controls how many top matches you want.
• The store embeds the query, computes similarity with stored embeddings, and returns the best matches.

We can also control which parts of the memories get embedded by configuring the fields parameter or by specifying the index parameter when storing memories:

This is the core of retrieval-based memory systems:

• define what you save
• define how it is embedded
• ask semantic questions and get back relevant chunks of memory

Before we start to implement this strategy for our support agent it is important to note that the InMemoryStore is prefect for local experiments, unit tests and small prototypes but not enough for a production-grade app.

For production workloads, we almost always want a robust vector database backend that can handle millions of memory items, is scalable with efficient read/write operations and supports low-latency search and retrieval.

Implementation

Now that the store and semantic search pieces are clear, we can design a retrieval-based memory layer for our customer support agent.

Here's the workflow:

• We add important long-term facts about a user into the store (plan, workspace names, previous issues, etc.).
• When the user opens a new ticket and asks a question, the agent:
  • looks up these long-term memories via semantic search
  • injects the relevant ones into the prompt
  • answers using both the current conversation and the retrieved context

We’ll keep the design simple and focused on retrieval. Let's define our state:

Recap

In Part 15, we deliberately started from the ground up discussing how large language models are inherently stateless.

The model does not remember what happened earlier unless you show it again in the prompt. All of the memory we see in real systems is a product of managing context outside the model.

To make that concrete, we built a truly stateless agent in LangGraph that accepts user input and outputs a response. Each call built a fresh prompt from scratch using only the current input, sent it to the LLM, and returned a one-off answer.

The moment we asked a follow-up question, the agent had no idea about the previous query or its own response. Every request was independent. There was no conversation history and no concept of memory at all.

From there, we layered in the LangGraph mental model that we used throughout the article:

• State as the single object that flows through the graph, updated at each step.
• Nodes as small, focused functions that read from state and return updates.
• Edges as the control flow that decides which node runs next.
• Conditional edges for loops and branches.

Once that was clear, we introduced the memory side of the LangGraph ecosystem.

On the short-term side, we moved from a simple state to a MessagesState that holds a growing list of messages storing the conversation history. We wired this into a node and compiled the graph with a checkpointer plus a thread ID.

LangGraph then started persisting the conversation for that thread as a series of checkpoints, so that each turn could see the full interaction so far without us manually passing the history around.

On the long-term side, we looked at the store abstraction in LangGraph for durable memory. Instead of keeping everything inside a single thread, we wrote JSON documents into a store and then retrieved them on future runs.

This gave us cross-session memory for things like user preferences, project metadata, and past decisions. We also saw how threads, checkpoints, and stores were all tied together through the config layer.

By the end of Part 15, we had not optimized anything. But we did answer a more fundamental question, "Where does memory actually live in a LangGraph-based system, and how does it behave over time?"

If you haven’t explored Part 15 yet, we strongly recommend going through it first, since it sets the foundations and flow for what's about to come.

Read it here:

A Practical Deep Dive Into Memory Optimization for Agentic Systems (Part A)

AI Agents Crash Course—Part 15 (with implementation).

Daily Dose of Data ScienceAvi Chawla

In this chapter, we’ll understand and learn about memory optimization in agentic workflows, with necessary theoretical details and hands-on examples.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

To quickly reiterate, Part 15 was about understanding the LangGraph ecosystem, how it represents workflows as graphs over a shared state, and how threads, checkpoints, and stores combine to give you short-term and long-term memory layers.

This and the upcoming part (Parts 16 and 17) are about taking that foundation and putting it to use in understanding memory optimization strategies in depth. Here we start using that understanding to build agents that are not just stateful, but efficient and production-ready.

Once we move beyond demo scripts and into production, stuffing everything into the context window stops working almost immediately. Token limits are real. Costs grow fast. High latency means slow responses and poor user experience.

Therefore, it is quite clear that throwing more tokens at the model does not automatically give us better behavior.

So the question is not “How do I give the model more memory?” The question is “How do I give the model the right memory at the right time, for optimum performance?”

That is what memory optimization is about.

In this chapter, we will treat memory as a first-class design problem and walk through a set of practical strategies to manage it in LangGraph. For each strategy, we will focus on three things:

• Intuition: What problem it solves, where it fits in the stack, and what it trades off.
• Implementation: How to express it using LangGraph’s building blocks: state, nodes, edges, threads, checkpoints, and stores.
• Behavior: How it changes cost, latency, and the quality of responses in realistic flows.

We will keep everything grounded and run a single example of a customer support chatbot for a SaaS product.

This agent will need to:

• Handle long, messy support threads with follow-ups and clarifications
• Refer back to earlier messages in the same ticket without re-asking everything
• Remember user-level information across tickets (plans, preferences, past issues)
• Stay within reasonable latencies

We will be looking at an example of a customer support agent and evolve our agentic system step by step:

• Start with a sequential memory baseline where we send the entire conversation to the LLM on every turn.
• Introduce sliding windows to bound how much recent context we carry into each call.
• Add summarization so that older parts of the conversation are compressed rather than dropped.
• Layer in retrieval over a long-term store so we can pull back only the most relevant past interactions.
• Move to hierarchical memory where session-level context, user profiles, and product knowledge live in clearly separate tiers.
• Add OS-like memory management, where we treat context as a limited budget and explicitly choose what stays in active vs passive memory states.

All of this will be implemented in LangGraph with code you can adapt and reuse for developing your own agents.

To summarize our system evolution as a diagram:

To provide you an in depth understanding of the concepts, we would build these memory frameworks using only the fundamental components of LangGraph discussed previously.

The same strategies could also be implemented using more abstract components but we won't use them for this tutorial.

By the end of this and the next part, you should be able to look at your own agent and choose the right memory strategy based on the decision frameworks we will discuss.

Sequential memory

In Part 15, we ended with a simple but powerful idea wherein we store all messages inside a list and persist that state with a checkpointer and a unique thread ID. This effectively gives us an agent with short-term memory inside a conversation.

That exact sequential pattern is the first technique we shall discuss.

In the baseline agent with no memory, the state consisted of just the user query and the LLM response, and every call built a prompt from scratch with only the current input. As a result, the agent could not recall any prior information when answering the next user query.

In the sequential approach, the state holds a growing list of messages, and we always send the full conversation history to the model on each turn. This creates a linear chain of memory, preserving everything that has been said so far.

This gives us an agent that can handle follow-ups correctly.

This strategy is the simplest of all and will serve as a reference implementation to compare all other memory strategies against.

Before we dive into the code, let’s build an intuition for how this works in plain terms:

• The user starts a conversation with the AI agent.
• The agent generates a response.
• This user-AI interaction is saved as a single block of text. Let's call it a turn.
• For each subsequent turn, the agent takes the entire conversation history (Turn 1 + Turn 2 + Turn 3, and so on) and combines it with the new user query.
• This massive block of text is sent to the LLM to generate the next response.

Here's a diagram, summarizing the flow:

Implementation

Now, let's look at how we can implement this in LangGraph.

Before moving on, let's first complete our setup and install the required dependencies.

We will use OpenRouter as our main LLM provider for this tutorial. It provides a single API to connect to all the LLM providers out there.

Go to OpenRouter, create an account and get your API key, and store it in a .env file:

Next, we load this key into our environment as follows:

OpenRouter is compatible with the OpenAI chat completions API, so we can use ChatOpenAI with a custom base_url to define our LLM.

model is the model identifier from OpenRouter. You can swap this out for Claude, Gemini, or any other open source model if you like.

Now we are all set to get on with the code.

Let's first define our state as follows:

Here:

• messages will hold the entire conversation history: user messages, AI replies, tool calls, tool results.
• operator.add tells LangGraph to append new messages instead of overwriting the list.

This is exactly what we need for the conversation to grow over time and gives us short-term memory within a single thread.

Next, we set up our LLM node for response generation:

Here:

• The node reads state["messages"], prepends a system message, and sends the entire history to the LLM.
• LLM returns a single new message, which will be appended to the conversation list due to operator.add.

Now we compile the graph and enable checkpointing to persist state between calls:

Here we are creating a simple graph:

• It has a single node, chat_llm, which handles one turn of conversation.
• The graph starts at START, runs chat_llm, and then reaches END.
• We compile it with an InMemorySaver checkpoint. This is what allows LangGraph to save and restore the conversation state across multiple calls.

Sample conversation

We now have a full-history, sequential memory agent in LangGraph.

But what good is our agent if it can't answer user queries reliably. So, now let's test by simulating a simple support ticket flow for our customer support agent.

To quantify the impact of this strategy and to easily compare different memory strategies later, we will analyze their token usage and latency as we go along the conversation.

Turn 1:

For this turn:

• messages contains one HumanMessage describing the issue.
• chat_llm_node returns an answer.
• Checkpointer saves a checkpoint for thread_id="ticket-seq" containing both the user message and the assistant's reply.

Turn 2:

Introduction

So, while learning MLOps, we explored traditional machine learning models and systems.

We learned how to take them from experimentation to production using the principles of MLOps.

Now, a new question emerges:

What happens when the “model” is no longer a custom-trained classifier, but a massive foundation model like Llama, GPT, or Claude?

Are the same principles enough?

Not quite.

Modern AI applications are increasingly powered by large language models (LLMs), which are systems that can generate text, reason over documents, call tools, write code, analyze data, and even act as autonomous agents.

These models introduce an entirely new set of engineering challenges that traditional MLOps does not fully address.

This is where AI engineering and LLMOps come in.

AI engineering, and specifically LLMOps (Large Language Model Operations), are the specialized practices for managing and maintaining LLMs and LLM-based applications in production, ensuring they remain reliable, accurate, secure, and cost-effective.

LLMOps aims to manage language models and the applications built on them by drawing inspiration from MLOps. It applies reliable software engineering and DevOps practices to LLM-based systems, ensuring that all components work together seamlessly to deliver value.

Now that we are starting the LLMOps phase of our MLOps and LLMOps crash course, the aim is to provide you with a thorough explanation and systems-level thinking to build AI applications for production settings.

Just as in the MLOps phase, each chapter will clearly explain necessary concepts, provide examples, diagrams, and implementations.

As we progress, we will see how we can develop the critical thinking required for taking our applications to the next stage and what exactly the framework should be for that.

We’ll begin with the fundamentals of AI engineering and LLMOps, incorporate related concepts, and progressively deepen our understanding with each chapter.

As for this part, we will focus on laying the foundation by exploring what LLMOps is, why it matters, and how it shapes the development of modern AI systems.

Let's begin!

Fundamentals of AI engineering & LLMs

Large language models (LLMs) and foundation models, in general, are reshaping the way modern AI systems are built.

Out of this, AI engineering has emerged as a distinct discipline, one that focuses on building practical applications powered by AI models (especially large pre-trained models).

It evolved out of traditional machine learning engineering as companies moved from training bespoke ML models to harnessing powerful foundation models developed by others.

In essence, AI engineering blends software engineering, data engineering, and ML to develop, deploy, and maintain AI-driven systems that are reliable and scalable in real-world conditions.

At first glance, an “AI Engineer” might sound like a rebranding of an ML engineer, and indeed, there is significant overlap, but there are important distinctions.

AI engineering emphasizes using and adapting existing models (like open-source LLMs or API models) to solve problems, whereas classical ML engineering often centers on training models from scratch on curated data.

AI engineering also deals with the engineering challenges of integrating AI into products: handling data pipelines, model serving infrastructure, continuous evaluation, and iteration based on user feedback.

The role sits at the intersection of software development and machine learning, requiring knowledge of both deploying software systems and understanding AI model behavior.

The AI application stack

AI engineering can be thought of in terms of a layered stack of responsibilities, much like traditional software systems. At a high level, any AI-driven application involves three layers:

• The application itself (user interface and logic integrating AI)
• The model or model development layer
• The infrastructure layer that supports serving and operations

Application development (top layer)

At this layer, engineers build the features and interfaces that end-users interact with, powered by AI under the hood.

With powerful models readily available via libraries or APIs, much of the work here involves prompting the model effectively and supplying any additional context the model needs.

Because the model’s outputs directly affect user experience, rigorous evaluation is crucial at this layer. AI engineers must also design intuitive interfaces and handle product considerations (for example, how users provide input to the LLM and how the AI responses are presented).

This layer has seen explosive growth in the last couple of years, as it’s easier than ever to plug an existing model into an app or workflow.

Model development (middle layer)

This layer is traditionally the domain of ML engineers and researchers. It includes choosing model architectures, training models on data, fine-tuning pre-trained models, and optimizing models for efficiency.

When working with foundation models, model development typically involves adapting an existing pretrained model rather than training one from scratch. It can involve tasks like fine-tuning an LLM on domain-specific data and performing optimization (e.g., quantizing or compressing).

Data is a central piece here: preparing datasets for fine-tuning or evaluating models, which might include labeling data or filtering and augmenting existing corpora.

Hence, even though foundation models are used as a starting point, understanding how models learn (e.g., knowledge of training algorithms, loss functions, etc.) remains valuable in troubleshooting and improving them as per the desired use case.

Infrastructure (bottom layer)

At the base, AI engineering relies on robust infrastructure to deploy and operate models.

This includes the serving stack (how you host the model and expose it), managing computational resources (typically means provisioning GPUs or other accelerators), and monitoring the system’s health and performance.

It also spans data storage and pipelines (for example, a vector database to store embeddings for retrieval), as well as observability tooling to track usage and detect issues like downtime or degraded output quality.

Based on the three layers and whatever we've learned so far, one key point is that many fundamentals of Ops have not changed even as we transition to using LLMs. Similar to any Ops lifecycle, we still need to solve real business problems, define success and performance metrics, monitor, iterate with feedback, and optimize for performance and cost.

However, on top of those fundamentals, AI engineering introduces new techniques and challenges unique to working with powerful pre-trained models.

Next, let’s take a look at some fundamental points that will help us understand what large language models (LLMs) really are.

LLM basics

Large language models (LLMs) are a type of AI model designed to understand and generate human-like text. They are essentially advanced predictors: given some input text (a “prompt”), an LLM produces a continuation of that text.

Under the hood, most state-of-the-art LLMs are built on the transformer architecture, a neural network design introduced in 2017 (“Attention Is All You Need”) that enables scaling to very high parameter counts and effective learning from sequential data like text.

Attention Is All You Need

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

arXiv.orgAshish Vaswani

Several characteristics and terminology related to LLMs:

Scale (the “large” in LLM)

LLMs have a huge number of parameters (weight values in the neural network). This number can range from hundreds of billions to trillions.

There isn’t a strict threshold for what counts as “large”, but generally, it implies models with at least several billion parameters.

The term is relative and keeps evolving (each year’s “large” might be “medium” a couple of years later), but it contrasts these models with earlier “small” language models (like older RNN-based models or word embedding models with millions of parameters).

Training on massive text corpora

LLMs are trained in an unsupervised manner on very large text datasets, essentially everything from books, articles, websites (Common Crawl data), Wikipedia, forums, etc., up until a certain cut-off date.

The training objective is often to predict the next word in a sentence (more formally, next token, since text is tokenized into sub-word units). By learning to predict next tokens, these models learn grammar, facts, reasoning patterns, and even some world knowledge encoded in text.

The training process involves reading billions of sentences and adjusting weights to minimize prediction error. Through this, LLMs develop a statistical model of language that can be surprisingly adept at many tasks.

Generative and autoregressive

Most LLMs (like the GPT series) are autoregressive transformers, meaning they generate text one token at a time, each time considering the previous tokens (the prompt plus what they’ve generated so far) to predict the next.

This allows them to generate free-form text of arbitrary length. They can also be directed to produce specific formats (JSON, code, lists) via appropriate prompting. LLMs fall under the Generative AI category as well, since they create new content rather than just predicting a label or category.

Another kind of LMs is masked language models.

A masked language model predicts missing tokens anywhere in a sequence, using the context from both before and after the missing tokens. In essence, a masked language model is trained to be able to fill in the blank. A well-known example of a masked language model is bidirectional encoder representations from transformers, or BERT.

Masked language models are commonly used for non-generative tasks such as sentiment analysis. They are also useful for tasks requiring an understanding of the overall context, like code debugging, where a model needs to understand both the preceding and following code to identify errors.

Emergent abilities

One intriguing aspect discovered is that as LMs get larger and are trained on more data, they start exhibiting emergent behavior, i.e., capabilities that smaller models did not have, seemingly appearing at a certain scale.

For example, the ability to do multi-step arithmetic, logical reasoning in chain-of-thought, or follow certain complex instructions often only becomes reliable in the larger models.

These emergent abilities are a major reason why LLMs took the world by storm. At a certain size and training breadth, the model is not just a mimic of text, but can perform non-trivial reasoning and problem-solving. It’s still a statistical machine, but it effectively learned algorithms from data.

Few-shot and zero-shot learning

Before LLMs, if you wanted a model to do something like summarization, you’d train it specifically for that. LLMs introduced the ability to do tasks zero-shot (no examples, just an instruction in plain language) or few-shot (provide a few examples in the prompt).

For instance, you can paste an article and say “TL;DR:” and the LLM will attempt a summary, even if it was never explicitly trained to summarize, because it has seen enough text to infer what “TL;DR” means and how summaries look.

This was a revolutionary shift in how we interact with models: we don’t always need a dedicated model per task; one sufficiently large model can handle myriad tasks given the right prompt.

This is why prompt engineering became important; the model already has the capability, we just have to prompt it correctly to activate that capability.

Transformers and attention

For a bit of the technical underpinnings, transformers use a mechanism called self-attention, which allows the model to weigh the relevance of different words in the input relative to each other when producing an output.

This means the model can capture long-range dependencies in language (e.g., understanding a pronoun reference that was several sentences back, or the theme of a paragraph).

Transformers also lend themselves to parallel computation, which made it feasible to train extremely large models using modern computing (GPUs/TPUs). This architecture replaced older recurrent neural network approaches that couldn’t scale as well.

Limitations

It’s important to remember LLMs don’t truly “understand” in a human sense. They predict text based on patterns, i.e., basically, they are probabilistic. This means they can be right for the wrong reasons and wrong with high confidence.

For example, an LLM might generate a very coherent-sounding but completely made-up answer to a factual question (hallucination). They have no inherent truth-checking mechanism; that’s why providing context or integrating tools is often needed for high-stakes applications.

It’s also worth noting that making a model larger yields diminishing returns at some point. The jump from 100M to 10B parameters yields a bigger improvement than the jump from 10B to 50B, for example.

So just because an LLM is extremely large doesn’t always mean it’s the best choice. There might be sweet spots in the size vs performance vs cost trade-off. Engineers often choose the smallest model that achieves the needed performance to keep latency/cost down.

For instance, if a 7B model can do a task with 95% success and a 70B model can do it with 97%, one might stick with 7B for production due to the huge difference in resource requirements, unless that extra 2% is mission-critical.

In summary, an LLM is like an extremely knowledgeable but somewhat alien being: it has read a lot and can produce answers on almost anything, often writing more fluently than a human, but it might not always be reliable or know its own gaps. It’s our job to coax the best out of it with instructions and context, and curtail its weaknesses with evaluations.

The shift from traditional ML models to foundation model engineering

Traditionally, deploying an AI solution usually means developing a bespoke ML model for the task: gathering a labeled dataset, training a model, and integrating it into an application.

This “classical” ML engineering was very model-centric; you’d often start from scratch or from a small pre-trained base, forming the data → model → product flow.

Introduction

In previous articles, we’ve covered the foundational aspects of building and optimizing LLM-powered systems, including topics like Retrieval-Augmented Generation (RAG), multimodal integration, and model evaluation techniques.

Foundations of RAG systems

A practical and beginner-friendly crash course on building RAG apps (with implementations).

Daily Dose of Data ScienceAkshay Pachaar

These discussions laid the groundwork for building systems capable of sophisticated tasks.

However, as these systems scale, ensuring their performance remains robust and trustworthy becomes the real challenge. That is why building reliable and impactful LLM-powered systems requires more than just deploying a model.

It demands continuous evaluation to ensure quality and observability so that we can identify issues post-deployment.

Thus, this article will help you develop skills to examine how evaluation and observability can work together seamlessly to make LLM applications more robust and practical:

• Evaluation ensures that the system delivers accurate, consistent, and task-relevant results.
• Observability offers a real-time view into the system’s inner workings, capturing everything from model drift to bottlenecks in performance.

Technically speaking, the focus of this article is Opik by CometML, which is an open-source framework that provides a specialized platform to address exactly these concerns.

GitHub - comet-ml/opik: From RAG chatbots to code assistants to complex agentic pipelines and beyond, build LLM systems that run better, faster, and cheaper with tracing, evaluations, and dashboards.

From RAG chatbots to code assistants to complex agentic pipelines and beyond, build LLM systems that run better, faster, and cheaper with tracing, evaluations, and dashboards. - comet-ml/opik

GitHubcomet-ml

With Opik, we can:

• Track LLM performance across multiple metrics, including relevance, factuality, and coherence.
• Monitor system behavior in real-time, identifying areas for improvement.
• Seamlessly integrate with popular LLMs (like OpenAI or Ollama) and tools (like LlamaIndex).

This is possible because Opik offers several tools to track, evaluate, and monitor machine learning systems in production, enabling teams to bridge the gap between development and real-world deployment.

And a great part about Opik is that whether you’re using OpenAI, Ollama, or any other LLM provider, Opik makes it straightforward to monitor your applications, log every interaction, and evaluate results against predefined metrics.

Here's the outline for this article:

• We'll start by setting up the environment and configuring Opik.
• Next, we are going to trace a simple Python function so that you understand how Opik works.
• After that, we are going to track a simple LLM core so you can use the same for the AI applications that you're building.
• Moving on, we are going to build a RAG application using LlamaIndex, which we shall evaluate and monitor this application in our Opik dashboard.

Let's begin!

If you prefer to follow along with a video tutorial instead, you can watch this video below:

Below, we have the text version of the above video, along with detailed and step-by-step code instructions.

Installation and setup

We'll start by creating an account on comet.com.

Once you will create an account, it will give you two options to choose from—select LLM evaluation (Opik):

Once done, you will find yourself in this dashboard, where you can also find your API key on the right:

Next, in your current working directory, create a .env file:

Copy the API key shown in your dashboard and paste it as follows:

To configure Opik, run the following code.

Executing the above code will open a panel to enter the API key obtained above. Enter the API key there, and done—Opik has been configured.

In some parts of this article, we will be using a locally running model using Ollama. Ollama provides a platform to run LLMs locally, giving you control over your data and model usage.

Here's a step-by-step guide on using Ollama.

• Go to Ollama.com, select your operating system, and follow the instructions.

• If you are using Linux, you can run the following command:

• Ollama supports a bunch of models that are also listed in the model library:

library

Get up and running with large language models.

Once you've found the model you're looking for, run this command in your terminal:

The above command will download the model locally, so give it some time to complete. But once it's done, you'll have Llama 3.2 3B running locally, as shown below which depicts Microsoft's Phi-3 served locally through Ollama:

That said, for our demo, we would be running the Llama 3.2 1B model instead since it is smaller and will not take much memory:

Finally, install the open-source Opik framework, LlamaIndex, and LlamaIndex's Ollama integration module as follows:

Setup and installation are done!

Data

Next, download the dataset as follows:

The above code will create a data folder in your current working directory.

Getting started with Opik

When it comes to monitoring machine learning workflows, simplicity is key.

In this respect, Opik offers an intuitive way to track and log your experiments, which can include advanced applications like Retrieval-Augmented Generation (RAG) pipelines or multi-agent systems.

Let’s dive into a quick demo to see how Opik simplifies the process.

Tracking a Simple Python Function

Let's start with a simple demo.

Imagine we want to track all the invocations to this simple Python function specified below:

To do this, Opik provides a powerful @track decorator that makes tracking a Python function effortless.

That's it!

By wrapping any function with this decorator, you can automatically trace and log its execution inside the Opik dashboard.

For instance, currently, our dashboard does not show anything.

If we run the above code, which is decorated with the @track decorator, and after that, we go to the dashboard, we will find this:

As depicted above, after running the function, Opik automatically creates a default project in its dashboard.

In this project, you can explore the inputs provided to the function, the outputs it produced, and everything that happened during its execution.

For instance, once we open this project, we see the following invocation of the function created above, along with the input, the output produced by the function, and the time it took to generate a response.

Also, if you invoke this function multiple times, like below...

...the dashboard will show all the invocations of the functions:

Opening any specific invocation, we can look at the inputs and the outputs in a clean YAML format, along with other details that were tracked by Opik:

This seamless integration makes it easy to monitor and debug your workflows without adding any complex boilerplate code.

To recap, these are the steps:

• Define Your Function: Start with any Python function—whether it’s as simple as adding two numbers or as complex as an LLM-based application.
• Apply the @track Decorator: Add the @track decorator to the function definition.
• Run and Observe: Execute the function, and the inputs, outputs, and other relevant data will be captured and displayed on the dashboard.

The true power of Opik lies in its flexibility. While this demo showcased a simple function, the same tracking capabilities can be extended to much more complex systems:

• RAG applications: Track every step of your Retrieval-Augmented Generation pipeline, from query formulation to retrieval and final generation.
• Multi-agent orchestration: Monitor interactions and outputs in applications involving multiple agents working together to solve a task.

All you need to do is wrap the relevant logic in a Python function and apply the @track decorator. It’s that straightforward.

Having understood how to track simple Python functions, the next step is to explore how Opik can be used to track LLM calls to monitor every interaction with large language models, analyze their performance, and gain actionable insights.

Let's do that below.

Tracking LLM calls with Opik

The purpose of this section is to show how Opik can log and monitor the interaction when the input includes both text and an image URL.

Before diving into Ollama, let's do a quick demo with OpenAI.

You would need an OpenAI API key for this, which you can get here: OpenAI API key.

Specify this in the .env file we created above as follows:

To load these API keys into your environment, run these:

Next, we shall be using Opik's OpenAI integration for this demo, which is imported below, along with the OpenAI library:

Moving on, we wrap the OpenAI client with Opik’s track_openai function. This ensures that all interactions with the OpenAI API are tracked and logged in the Opik dashboard. Any API calls made using this client will now be automatically monitored, including their inputs, outputs, and associated metadata.

Next, we define our multimodal prompt input as follows:

Finally, we invoke the chat completion API as follows:

Here, we make the API call using the chat.completions.create method:

• model: Specifies the LLM to use (gpt-4o-mini in this case).
• messages: Provides the multimodal input we defined earlier.
• max_tokens: Limits the number of tokens in the output to 300, ensuring the response remains concise.

Yet again, if we go back to the dashboard, we can see the input and the output of the LLM:

Opening this specific run highlights so many details about the LLM invocation, like the input, the output, the number of tokens used, the cost incurred for this specific run, and more.

This shows that by using track_openai, every input, output, and intermediate detail is logged in the Opik dashboard, for improved observability.

We can also do the same with Ollama for LLMs running locally.

Here's a quick demo.

The process remains almost the same.

We shall again use Opik's OpenAI integration for this demo, which is imported below, along with the OpenAI library:

Next, we again create an OpenAI client, but this time, we specify the base_url as https://localhost:11434/v1:

Next, to log all the invocations made to our client, we pass the client to the track_openai method:

Finally, we invoke the completion API as follows:

If we head over to the dashboard again, we see another entry:

Opening the latest (top) invocation, we can again see similar details like we saw with OpenAI—the input, the output, the number of tokens used, the cost, and more.

That was simple, wasn't it?

All the supported integrations and how to use them in your projects are available in the quickstart guide.

Experiment tracking with Opik

Now that we’ve seen how Opik works, it’s time to unlock its true potential by using it to evaluate and monitor LLM applications.

This involves creating an evaluation dataset, designing an experiment, and using Opik to track and analyze the results step by step.

Workflow

Here’s the workflow for setting up an evaluation experiment with Opik:

• Define the Dataset:
  • We’ll start by preparing a dataset for evaluation.
  • This dataset will include input queries and their corresponding expected outputs.

• This will allow us to measure how well the LLM application performs across several parameters like coherence, factfullness, etc. We discussed all of these metrics in detail in Part 2 of our RAG crash course series:

A Crash Course on Building RAG Systems – Part 2 (With Implementation)

A deep dive into evaluating RAG systems (with implementations).

Daily Dose of Data ScienceAvi Chawla

• Run the Experiment:
  • Each query item will be processed through the RAG application to produce a response

• During the retrieval process, we must have retrieved some context from the vector database before generating the response:

• Thus, the application’s output will then be compared to the expected output using predefined evaluation metrics, generating a feedback score. Moreover, we shall compare the expected context with the retrieved context to evaluate the retrieval pipeline.
• Track the Experiment:
  • Everything that happens within the above process is treated as an experiment.
  • Opik will trace and monitor the experiment, logging inputs, outputs, and scores for each dataset item in its dashboard for easy analysis.

Data

To run this experiment, we need some data, which we already downloaded earlier:

The above code will create a data folder in your current working directory.

Integrate LlamaIndex with Opik

Since we would be using LlamaIndex to build a RAG pipeline shortly, the next step is to integrate LlamaIndex with Opik.

This integration enables tracking all LlamaIndex operations like document chunking and ingestion to generate and retrieve queries.

To enable this tracking, you need to configure an Opik callback handler. This handler acts as a bridge between LlamaIndex and Opik, logging all operations in real time.

First, you’ll need to import Settings and CallbackManager from LlamaIndex, along with the LlamaIndexCallbackHandler from Opik:

Next, we create an instance of LlamaIndexCallbackHandler, which automatically logs all LlamaIndex operations to Opik.

Finally, we use LlamaIndex’s Settings to integrate the callback handler via the CallbackManager. This ensures that every operation performed by LlamaIndex is tracked.

Done!

With Opik’s integration for LlamaIndex, we can now focus on building a simple RAG application without worrying about tracking or monitoring manually.

Build a RAG application

Once LlamaIndex is integrated with Opik, the next logical step is to create a basic RAG pipeline.

Also, going ahead, our idea is to keep everything simple and focus on what's important, which is evaluation and observability.

This code below demonstrates how you can use LlamaIndex to load documents, build an index, and query it for insights.

We start with the necessary imports:

Next, we do the following:

• We use the SimpleDirectoryReader from LlamaIndex to read all the documents from the directory created above during data download.
• With the documents loaded, the VectorStoreIndex is created to serve as the foundation for efficient retrieval. This index maps document content into a vector space stored in memory.
• Once the index is ready, a query engine is created to perform semantic searches and answer specific questions based on the indexed data.

Done!

Next, we query our RAG pipeline as follows, which produces a response:

Yet again, if we go back to the dashboard, we see two things here:

• First, we have an index construction process, which we just ran when defining the RAG pipeline. Opening this specific trace shows everything that went into this process, like the data location, total embeddings, the time it took to embed and chunk the data, etc.

• Next, we have the step where we queried our vector database to produce a response, which highlights so many details about the LLM invocation, like the input, the output, the number of tokens used, the cost incurred for this specific run, the time spent in each process, and more:

• Moreover, if we look at the retrieve trace, it tells us all the retrieved context, which can be easily inspected the retrieved context for better debugging:

• Similarly, we can look at the subprocess, like the final prompt, the generation process, etc.

RAG evaluation and tracking with Opik

Now that we have a RAG pipeline ready, it's time to create a data set on which we can evaluate it. In order to do this, we have created a sample data set that has a question, an answer, and the context that is being used to arrive at this answer.

You can download this dataset below:

We have already learned how to generate this in Part 2 of our RAG crash course series so we are not going to discuss it again:

A Crash Course on Building RAG Systems – Part 2 (With Implementation)

A deep dive into evaluating RAG systems (with implementations).

Daily Dose of Data ScienceAvi Chawla

To get started, we first define a dataset client in Opik:

This immediately gets reflected in the datasets tab of the dashboard:

Now, we need to push our dataset to this dataset client.

To do this, we first create a dictionary of our question-answer-context triplets and upload it to our client using the insert method of the dataset object created above:

These records immediately get reflected within the dataset shown in the dashboard as shown below:

That said, you can also easily add a new data set item using the same format—the input, the expected out, put and any additional field that you want to add here.

For instance, say you have your application up and running in production and then you encounter a really interesting example that you want to be part of your evaluation set. So you can easily do that using the above code.

There's one more way.

• Click on “Create dataset item”:

• Next, you can populate these fields using the UI:

With that, we have everything ready to do an end-to-end evaluation, and it's time to put everything together and see how it works.

More specifically, we have already created a data set, and now we just need to create an evaluation task.

Inside that task, we are going to put the LLM application (which is our RAG app).

We shall also take all the expected output from the data set that we created to get feedback on how our application is working.

So the first thing we need to do is create an LLM application. This we have already seen above:

We start with the necessary imports:

Next, we do the following:

• We use the SimpleDirectoryReader from LlamaIndex to read all the documents from the directory created above during data download.
• With the documents loaded, the VectorStoreIndex is created to serve as the foundation for efficient retrieval. This index maps document content into a vector space stored in memory.
• Once the index is ready, a query engine is created to perform semantic searches or answering specific questions based on the indexed data.

Done!

Next, we use the @track decorator we used earlier in this article to define a function:

Inside this function, we have our RAG application. It takes a user query, which is the input string, and provides a string output, which is the output of the RAG application.

This my_llm_application is our final application now, which, in your case, can be anything—RAG, multimodal RAG, Agentic RAG, etc.

Next, we are going to use the same OpenAI integration to track all the LLM calls:

After this, we need to create an evaluation task that specifies the output that you get for a particular input. And inside this, we need to put in our LLM application:

After this, we create a data set client, which we already saw above:

The next part is to define all the evaluation metrics based on which we will evaluate our LLM application.

So we instantiate some evaluation metrics to evaluate our pipeline with as follows:

Based on the expected output and the output given by our application, we will get feedback scores for our LLM application.

Once this is done, it's time to run the evaluation, and we are going to put everything together—the data set, the evaluation task, the scoring metrics, all four metrics that we just discussed here, and the experiment configuration wherein we specify the model we intend to us as our evaluation system which acts as a judge and see like how things are working.

This is implemented below:

Done!

This produces the evaluation results shown below:

More specifically, these are the average scores we got for each of the four metrics on the five examples in our evaluation dataset.

We can also view these results in the Opik dashboard under the "Experiments" section:

Let's select our evaluation:

And here, we can see the individual scores for each of the 5 test samples we had in our evaluation dataset.

Moreover, if you go to the "Projects tab" and select your project, you can see all evaluation task-related activities:

If we open the very last evaluation task (which has a hallucination score of 1), it gives me a detailed overview of everything that happened for this specific example:

This includes the inputs of the dataset we had in this first item, the output that was produced by our LLM pipeline, the time taken to compute each of the evaluation metrics and much more.

Let's say you want to go into more details related to the hallucination of this evaluation example, we can select the "hallucination_metric" section to see the input, the output and the corresponding context it retrieved. Moreover, as shown below, it also shows a reason behind producing a particular score.

This makes it much easier to debug and understand what's happening and how you can make further improvements to your system.

Conclusion

With that, we come to an end of this deep dive on LLM evaluation and tracking using a relatively new but extremely powerful open-source framework—Opik.

As we saw above, each of the underlying technicalities for evaluation and tracking has already been implemented by Opik, so the only thing you are supposed to do is build your LLM apps.

Every aspect of evaluation and tracking is handled by Opik.

Moreover, in this demo, we only looked at OpenAI and Ollama, but all the supported integrations and how to use them in your projects are available in the quickstart guide.

In most cases, you would hardly have to add a few lines of code to your existing pipelines to have this robust framework integrated into your LLM applications.

The code for today's article is available here:

ai-engineering-hub/eval-and-observability at main · patchy631/ai-engineering-hub

Contribute to patchy631/ai-engineering-hub development by creating an account on GitHub.

GitHubpatchy631

As always, thanks for reading!

Any questions?

Feel free to post them in the comments.

Or

If you wish to connect privately, feel free to initiate a chat here:

Recap

In Part 17 of this MLOps and LLMOps crash course, we explored monitoring and observability in ML systems, along with the key tools that make up the ML observability stack.

We began by understanding the two types of monitoring: functional and operational monitoring.

After that, we explored Evidently AI, the functional monitoring tool, and walked through data-drift detection, data-quality analysis, and HTML dashboard generation, complete with code examples.

Next, we did a brief discussion on Prometheus and Grafana for operational monitoring and understood their significance for ML systems and teams.

Finally, we went hands-on covering functional and operational monitoring setup in FastAPI-based applications.

If you haven’t explored Part 17 yet, we recommend going through it first, since it sets the flow for what's about to come.

Read it here:

The Full MLOps Blueprint: Monitoring and Observability—Part B

MLOps and LLMOps Crash Course—Part 17.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we’ll understand and learn about CI/CD workflows in ML systems with necessary theoretical details and hands-on examples.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let’s begin!

Introduction

CI/CD stands for continuous integration and continuous delivery/deployment, which are standard DevOps practices that automate the software development lifecycle to deliver software changes faster and more reliably.

CI focuses on automatically integrating code changes and running tests, while CD automates the release of those validated changes to a testing or production environment. This automation ensures code quality, speeds up releases, and helps developers respond quickly to user feedback.

Traditional CI/CD focuses on delivering code changes quickly and safely. With ML, we need to extend this idea to data and models too.

This means our pipelines must not only run code tests, but also validate input data, retrain models, track model metrics, and then deploy or push those models.

So let's go ahead and dive deeper into the various aspects, starting with CI for ML systems.

CI for ML

Continuous Integration for ML means automating the validation of everything upstream of deployment: data, code, and model.

The goal is to catch problems early (at commit time) before a model is deployed. This section breaks CI into three parts unique to ML: Data CI, Code CI, and Model CI.

Data CI: validating data

In ML, data is essentially "code", it defines model behavior. Therefore, integrating new data (or data pipelines) requires rigorous checks just like new code does.

Data CI focuses on automatically testing data quality and detecting data drift as part of the integration process.

Schema and quality checks

Whenever new data is ingested or a new dataset version is used for training, the pipeline should validate that the data meets the expected schema and values.

Tools like Pandera allow you to define explicit expectations or schemas for data. For example, you can assert that certain columns exist, data types are correct, no critical nulls or out-of-range values, etc.

These checks can run in CI to prevent training on corrupt or invalid data. Using data validation frameworks (e.g., Pandera) enables schema checks and automated anomaly detection on your data pipeline.

For instance, using Pandera (version 0.27.0) we can define a schema and validate a Pandas DataFrame in a test:

This little script loads a CSV file into a pandas DataFrame and then uses a Pandera model called TrainingDataSchema to make sure every column in that DataFrame satisfies the rules declared for it.

The schema specifies that feature1 must be a non-null floating-point value greater than zero, feature2 must be an integer between zero and one hundred (inclusive), and label must be an integer equal to either zero or one.

When the DataFrame is validated using validate with lazy=True, Pandera checks every row and collects all violations instead of stopping at the first error.

The code wraps this validation in a try/except block so that if the data fails the schema checks, the exception provides a compact table (failure_cases) showing exactly which rows and values broke which constraint, along with a count of total errors.

If the data passes all checks, the script simply prints that validation succeeded.

In a CI context, this could be part of a test that fails if the data doesn't conform.

Data tests like this ensure that upstream data issues (like a schema change or unexpected distribution shift) are caught early, rather than silently corrupting the model training.

Data drift checks

Data drift refers to changes in the statistical distribution of data over time. While data drift is often monitored in production, you may also integrate drift detection in CI when new data is used for retraining.

For example, if a new training dataset is significantly different from the previous training data, you might want the pipeline to flag it before retraining (especially if such drift is unintended).

Tools like Evidently AI can help with that. You can programmatically compare a new dataset against a reference dataset and get a drift score.

We covered Evidently AI in Part 17 of this course, you can check it out below:

The Full MLOps Blueprint: Monitoring and Observability—Part B

MLOps and LLMOps Crash Course—Part 17.

Daily Dose of Data ScienceAvi Chawla

In CI, this could prevent retraining a model on data that is too different without additional review. Often, significant drift might require updating data processing or a model retraining strategy.

Data versioning

To enable reproducibility and continuous integration of data, teams use data version control tools like DVC. These tools let you track dataset versions similar to how Git tracks code versions.

In CI, you can then pull specific data snapshots for training or testing. For example, DVC can store data files in remote storage and link them to Git commits.

A CI pipeline might want to fetch the exact version of a dataset corresponding to the current code version before running tests or training. This ensures that experiments are reproducible and that model training in CI is using the correct data.

Tools such as DVC integrate with Git to track large data files and even pipeline dependencies, allowing reproducible training when data changes.

We covered versioning of data with DVC in Part 3 of this course, you can check it out below:

The Full MLOps Blueprint: Reproducibility and Versioning in ML Systems—Part A (With Implementation)

MLOps and LLMOps Crash Course—Part 3.

Daily Dose of Data ScienceAvi Chawla

In summary, Data CI treats data as a first-class citizen in integration tests. It guarantees that any new data entering the pipeline is valid, and it keeps an eye on distribution changes. This prevents “data bugs”, which can be just as harmful as code bugs, from propagating to model training.

Next, let's go ahead and take a look at CI for code.

Code CI: testing code

Code continuous integration for ML looks much like traditional CI on the surface: you run unit tests, enforce code style, and perform integration tests. However, the content of these tests is tailored to ML pipeline code.

Unit tests

Our feature engineering functions, data loaders, and utility functions can have unit tests. For example, if we have a function preprocess_data(df) that fills missing values or scales features, write tests for it using small sample inputs.

If you have a custom loss function or metric, test it on known inputs. These tests catch logical bugs early.

Example: If one_hot_encode() is supposed to produce a fixed set of dummy columns, a unit test could feed a sample input and verify the output columns match expectations.

Pipeline integration test (small training run)

A unique kind of test in ML CI is a small-scale training run to catch pipeline integration issues.

This isn't about achieving good accuracy, but rather about making sure the training loop runs end-to-end with the current code and data schema.

For example, you might take a tiny subset of the data (or synthetic data) and run one epoch of training in CI, just to ensure that the model can train without runtime errors (and perhaps that the loss decreases on that small sample).

This can catch issues like misaligned tensor dimensions, incompatible data types, or broken training scripts that unit tests might miss.

Configuration and dependency checks

ML projects often have configuration for hyperparameters, data paths, etc. A CI process can validate these too. For example, if you use a config YAML or JSON, you might have a test to load it and ensure required fields exist.

Additionally, because ML code depends on many libraries (numpy, pandas, torch, etc.), it's a good practice to pin versions and record dependencies. This avoids the classic “it works on my machine” issue when deploying code.

Property-based tests

We must understand that tests in ML can be brittle if they rely on exact outputs, since adding new data or changing logic can make a comparison fail. Instead, focus on properties.

For example, if adding more data should not reduce the length of the output, test for that. If a function is supposed to normalize data, test that the mean of output is ~0. If the sum of the model’s prediction probabilities should sum to 1, test that property. These are more robust than testing for an exact prediction value.

The main point for Code CI is: treat your ML pipeline code like normal software. Write tests for data transformations, model training routines, and even the inference logic (e.g., a test could call the model’s predict on a known input to see if it returns output of expected shape and type).

By continuously integrating code changes with tests, you ensure that refactoring or new features (like trying a new preprocessing step) don’t break existing functionality.

Next, let's go ahead and take a look at CI for models.

Model CI: testing models and quality

Beyond code and data, we also need to continuously test the model artifact and its performance. This is where ML CI adds new types of tests not seen in standard software:

Performance metric thresholds

After training a model (even in CI on a smaller scale or with a previous training run), you should automatically evaluate it on a validation set and verify key metrics.

For example, if you expect at least 85% accuracy on the validation data, your CI can assert that. If the model underperforms (maybe due to a bug or bad data), the pipeline should fail rather than deploy.

In practice, teams define "failure thresholds", e.g., if the new model’s AUC drops 5 points below the current production model, do not promote it.

Reproducibility tests

A tricky aspect of ML is randomness. You should aim for reproducible training (set random seeds, control sources of nondeterminism). As part of CI, you might run a reproducibility test: train the model twice with the same data and seed, and confirm that you get the same result within certain tolerance, in cases.

If a model is truly deterministic given a seed, then any drift might indicate non-deterministic behavior or external issues. This test might be expensive so perhaps not run on every commit, but it’s valuable when establishing a pipeline.

Another approach is comparing model outputs to a stored “baseline” output for a few sample inputs, plus, instead of storing entire output datasets, you can store expectations about them (like distribution, ranges) so that tests are less brittle.

Bias and fairness checks

ML integration tests can also include checks for ethical or regulatory concerns. For instance, after training, you might compute metrics across protected subgroups (male vs female, etc.) to detect large disparities. Tools like IBM’s AI Fairness 360 can automate parts of this.

Model artifact checks

Once a model is trained, there are some practical things to test about the artifact:

• Size check: If the model file is too large (maybe someone accidentally saved a model with debug info or a very high number of parameters), it could be impractical to deploy. A CI step can ensure the model file (e.g., model.pkl or model.onnx) is under a certain size.
• Dependency check: Ensure that all libraries needed to load and run the model are accounted for. For example, if you use pickle or joblib for a scikit-learn model, loading it requires the same Python environment with those library versions. If you export to ONNX, ensure your serving environment has an ONNX runtime.

• Serialization format validation: If you use a specific format, you might run a quick test that tries a round-trip: save the model, then load it back and see if it still predicts the same on a sample input. This catches issues where the model might not be properly serializable.

To summarize: Model CI introduces automated “gates” based on model evaluation. Instead of relying purely on a human to review a training run, the CI will fail fast if the model isn’t up to par (performance-wise or other checks).

These tests give confidence that models which pass CI are at least likely to be good candidates for deployment, or at least they aren't obviously bad.

With this, we now understand what continuous integration is, let's go ahead and explore continuous delivery (CD) for ML.

CD for ML

Continuous Delivery (CD) for ML takes the artifacts and results from CI (code, data, model that passed tests) and automates the process of packaging, releasing, and/or deploying them.

ML CD has to handle releasing not just application code, but also new model versions. It often integrates with specialized infrastructure like model serving platforms or orchestration on Kubernetes.

Let’s break down key aspects of CD for ML: