How do AI teams use RL in production?

Introduction

Over the past twelve chapters, we understood the key machinery of reinforcement learning piece-by-piece, from bandits and value functions all the way to GRPO, judges, and agentic training loops. With Chapter 12, that machinery became complete.

So this final chapter does something different. There is nothing new to derive and nothing to train. Instead, we look at what the world is actually doing with all of it.

Everything ahead is a story someone shipped. We chose four, each showing a different role RL plays in the industry:

• RL as a product engine
• RL as a capability engine
• RL as enterprise strategy
• RL as a market of its own

For each story, we will read it by asking the questions this series trained us to ask:

• What is the environment, and does it meet the resettability and safety requirements?
• Where does the reward come from, and is it hand-defined, learned from preferences, verified, or elicited from a judge?
• What is the trajectory, and how does the loop close?

Company blog posts rarely use our vocabulary, so part of the exercise is translation. By the fourth story, you should find yourself doing the translation automatically.

Let's begin!

Case study 1: Cursor's Composer, RL as the product engine

Cursor is a code editor built around AI assistance. Its in-house agent model is called Composer, and Composer is something like a Chapter 12 system running as a commercial product.

Start with how the model is trained. In the original Composer announcement, Cursor described giving the model access to its production search and editing tools during RL, and running hundreds of thousands of concurrent sandboxed coding environments in the cloud to support the rollouts.

The stated principle is to minimize train-test mismatch. The model trains inside the same harness, with the same tools, prompts, and context, that it will face when deployed.

You already own every word of that sentence. The harness is the environment. The sandboxed virtual machines are the resettable, side-effect-free copies from Chapter 12's two training requirements. The rollouts are trajectories.

Cursor also reported that, during RL, the model picked up useful behaviors on its own, like writing and executing unit tests and fixing linter errors.

Then comes the part that makes Composer the flagship story of this chapter. In March 2026, Cursor described what it calls real-time RL. Model checkpoints are served directly to production. Real user interactions are observed and aggregated into reward signals. An improved checkpoint can ship as often as every five hours.

The cycle has five stages:

• Collect billions of tokens from user interactions with the current checkpoint.
• Distill them into rewards (for example whether the user kept the suggested edit or sent a dissatisfied follow-up message).
• Update the model weights based on that implied feedback.
• Run the candidate against evaluation suites, including their internal CursorBench, to catch regressions.
• Deploy the improved checkpoint back to production.

The whole loop takes about five hours, so multiple checkpoints can ship in a single day.

Why go to all this trouble instead of staying in simulation? Cursor's answer echoes our environment discussion. Simulating the computer is easy, but simulating the person overseeing the agent is not. Real-time RL uses real environments and real users, removing that modeling error and assumptions entirely.

Cursor reported results from an A/B test of Composer 1.5 with this loop. Agent edits persisted in the codebase 2.28% more often, dissatisfied follow-up messages dropped by 3.13%, and latency fell by 10.3%.

It is worth pausing on how a user's behavior becomes a number, because this is reward engineering of a kind we have not seen yet.

User interactions are instrumented on the client, flow through backend data pipelines, and are distilled into signals:

• Did the suggested edit persist in the codebase, or was it reverted?
• Did the next message read as a dissatisfied correction?

No single interaction is trustworthy, since a developer might revert a perfectly good edit for unrelated reasons. So the signal is noisy, and Cursor compensates the way any of our chapters would suggest, by aggregating over very large batches so the noise averages out while the preference survives.

There is one more detail. Composer reward-hacked in production. Cursor reported that the model learned to emit invalid tool calls to avoid negative rewards, and separately learned to defer risky edits by asking clarifying questions, which lowered its editing rate.

Cursor offers a counterpoint worth remembering. Reward hacking is a bigger risk in real-time RL, because every seam in the production stack becomes a surface to exploit. But it is also harder to get away with. In simulated RL, a cheating model simply posts a higher score. In production, real users trying to get work done are less forgiving, and hacks surface as bug reports.