Agentic RL: Environments, Trajectories, and the Training

Recap

In Chapter 11, we solved the problem that RLVR left open. Verifiers only exist for tasks with an answer key and most agent work has no such key. So we asked where the reward comes from when the environment stays silent.

We started with an observation about GRPO itself. The optimizer consumes one number per response and normalizes within the group. It never inspects where that number came from. The reward source is an open slot in the loop.

We then drew the line between verifiable and non-verifiable tasks and understood that verifiability is a property of the outcome.

We examined the obvious fix, hand-written reward functions, and saw why they are brittle. Every threshold is a knob we must set ourselves.

The way out was the LLM-as-a-judge. Two properties made it fit GRPO almost perfectly. Relative scoring is reliable where absolute scoring is not. And GRPO only needs relative scores anyway. Scoring the whole group in one judge call turned evaluation into comparison.

Finally, we built a from-scratch judge scorer on a RAG example.

If you have not read Chapter 11, we recommend doing so first:

The Reward Signal Problem for Agents

RL Part 11: From verifiable rewards to LLM-as-a-judge.

Daily Dose of Data ScienceAvi Chawla

In this chapter, we will assemble the full training loop for multi-step agents. We'll define what an RL environment means for an LLM agent, and formalize the trajectory as the training unit. Then we'll run the complete rollout, score, and update cycle using OpenPipe's ART (Agent Reinforcement Trainer) framework and its RULER reward function.

As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.

Let's begin!

From one judged group to a training loop

Look closely at what Chapter 11 actually trained on. Each candidate in the judged group was a single response. One prompt went in, one completion came out, and the judge compared the completions. That shape matches a chatbot but not truly an agent.

An agent does not produce one completion. It reads a goal, calls a tool, reads the result, and calls another tool. Only after several steps does it produce the result.

This creates two gaps that Chapter 11 never closed.

First, we never defined where the agent acts. A multi-step agent needs tools, state, and a rule for when the episode ends.

Second, we never showed how multi-step interactions become training data. GRPO updates the policy from groups of scored samples. So what exactly is a sample when the agent acted eight times?

Closing the above gaps is the destination for this chapter.

The agentic setting

We need to say precisely what "the agent acts over many steps" means. The good news is that we already own the vocabulary. Chapter 2 gave us states, actions, and rewards. We only need to map them onto the new setting.

An agent episode looks like this. The agent receives a goal in natural language. It observes its situation. It acts, by emitting either a tool call or a message. The environment responds with an observation, such as a tool result. The agent acts again. This repeats until the episode ends, usually when the agent produces a final answer or hits a turn limit.

Each piece maps onto our old language:

• Action: one model output, typically a tool call or a message.
• Observation: what comes back, typically a tool result.
• State: everything the agent knows so far. For an LLM agent, that is the interaction history.
• Reward: a score for the episode. It usually arrives once, at the end.

Let us make the policy precise. The policy chooses the next action given that history. Now, one episode, run start to finish, produces a trajectory. We will write it as $\tau$. It is the full sequence of the goal, all actions, all observations, and the final answer. The training objective is then stated over whole trajectories:

Here,

• $J(\theta)$ is the quantity we want to maximize, as a function of the policy parameters $\theta$.
• The symbol $\tau$ is one complete trajectory. The notation $\tau \sim \pi_\theta$ says trajectories are generated by running the current policy in the environment.
• $R(\tau)$ is the reward assigned to the whole trajectory.
• The expectation $\mathbb{E}$ averages over the randomness in both the policy's sampling and the environment.

The intuition is the same one that has carried all along. Make good trajectories more likely and bad ones less likely. GRPO will estimate this "good versus bad" comparison, exactly as it did in Chapters 10 and 11. Nothing about the optimizer changes. What changes is the object being scored, from a response to a trajectory.

In summary, the agentic setting is our familiar MDP with three twists. The state is the interaction history, the setting is partially observable, and the reward is sparse and trajectory-level. The objective maximizes expected reward over whole trajectories. Everything else in this chapter builds the machinery to optimize it.

RL environments for agents

The word "environment" has followed us since the start. Gridworld was an environment, cliff walking was an environment, similarly cart pole, lunar lander, etc. Now let's see what the word means when the agent is an LLM.

An agentic RL environment is the world the agent is in. Concretely, it must supply four things: