The Reward Signal Problem for Agents
RL Part 11: From verifiable rewards to LLM-as-a-judge.
Recap
In Chapter 10, we resolved the tension that closed Chapter 9. Some tasks do not need a learned reward model at all. When correctness is a matter of fact, a verifier supplies the reward directly. We built this idea into Reinforcement Learning with Verifiable Rewards (RLVR).
We then explored how the critic becomes optional. Its job was to estimate the expected reward, so we could tell whether a reward was better than average. GRPO does this without a network. It samples a group of responses to the same prompt and averages their rewards. That average becomes the baseline.
Then we understood the group-relative advantage. It measures how much better a reward is than the group's average, scaled by how spread out the rewards are.
We plugged this advantage into PPO's clipped objective and controlled drift with a KL penalty, estimated cheaply through the k3 estimator.
Finally, the payoff was the collapse from four models to two. The verifier replaced the reward model (in RLVR setting), the group baseline replaced the critic, and only the policy and reference remained.
If you have not read Chapter 10, we recommend doing so first:
Avi Chawla
In this chapter, we will explore where the reward comes from when there is no answer key. We will look at why hand-written reward functions are brittle, how research and industry approach this problem, and the central idea that makes it tractable, using an LLM-as-a-judge.
As always, every notion will be explained through clear examples and walkthroughs to develop a solid understanding.
Let's begin!
GRPO does not care where the number comes from
The first thing to notice about GRPO is how little it asks for. It needs one number for each response in a group, nothing more.
Look back at the algorithm from Chapter 10. The policy generates a group of responses. Something assigns a reward to each. GRPO subtracts the group mean and divides by the standard deviation. That value becomes the training signal.
At no point does GRPO inspect where the reward came from. It could come from a math checker, a compiler, a human rating, or a Python script. To the optimizer, all of these are identical. They are just a list of scalars.
It helps to picture the same GRPO loop with different reward sources plugged into one slot. With a math verifier, the slot holds a calculator. With a compiler, it holds a test runner. With human feedback, it holds a panel of raters. With a script, it holds hand-written Python.
The loop around the slot is identical in every case, only the contents of the slot change.
In Chapter 10 that slot held a verifier, and it was easy. The environment handed us a clean signal at almost no cost. But the moment we step outside tasks that can be scored with a verifier, the empty slot in the loop becomes the hardest part of the system.
In summary, GRPO consumes a number per response and normalizes within the group. The source of that number is left entirely open. That open slot is where this chapter lives.
Verifiable and non-verifiable tasks
To see why the open slot is hard to fill, we need a clear line between tasks where a verifier exists and tasks where one does not.
A verifiable task is one whose outcome can be checked by a rule or a program:
- A math answer matches the key or it does not.
- A program passes the test suite or it fails.
- A SQL query returns the right rows or the wrong ones.
In each case, a deterministic check produces an exact verdict.
A non-verifiable task has no such check. Consider a RAG system, which retrieves documents and writes an answer grounded in them. There is no single correct answer to compare against. Many phrasings are valid, and no string match can tell a faithful answer from a plausible-sounding invented one.
The same gap appears across most real agent work. A customer support agent drafts a reply, and there is no compiler to run it through. A summarization agent condenses a twenty-page document, and there can be many good summaries.
Tool use sits in an interesting position here. Some tool-use tasks are fully verifiable. If an agent's job is to book a flight matching a set of constraints, you can check the final booking against those constraints. If its job is to query a database and return a row count, you can compare the count. The presence of tools does not block verification.
Other tool-use tasks are not verifiable at all. If an agent searches an inbox and writes a natural-language answer, there may be no single gold answer, and the quality of the writing matters as much as the facts. The tools are the same, but the outcome is what differs.