Supervised fine-tuning teaches a model to imitate answers. Reinforcement learning teaches it to find answers that score well. Since DeepSeek-R1 showed that a simple, critic-free RL algorithm applied to verifiable tasks can produce genuine reasoning gains, GRPO (Group Relative Policy Optimization) has become the default post-training step for teams that have a way to grade an output programmatically: maths, code, SQL, tool calls, JSON schemas, retrieval answers with known ground truth.
This tutorial walks through a working GRPO run on a single 8B model with TRL and PyTorch, then covers the parts that actually decide whether the run succeeds: reward design, KL control, throughput, and how to tell a real improvement from reward hacking. It assumes you are comfortable with the LoRA workflow in Fine-Tuning Llama with QLoRA.
1. When GRPO is the right tool
Reach for GRPO only when all three of these hold:
- You can score an output automatically. A unit test passes, a numeric answer matches, the JSON validates against a schema, the SQL returns the expected rows. If grading needs a human or a judge model on every sample, you are in expensive, noisy territory.
- SFT is already saturated. RL amplifies behaviour the model can already produce occasionally. If the base model never solves the task, no reward signal exists to reinforce; collect SFT data first.
- You have prompts, not answers. GRPO needs a dataset of tasks plus a grader. It does not need reference completions, which is exactly why it is attractive when labelled data is scarce.
If you mainly need tone, format or style, DPO or plain SFT is cheaper and more predictable. If you need knowledge, you need retrieval — see Fine-Tune or RAG or Both?.
2. How GRPO works, briefly
For each prompt, the policy samples a group of G completions (typically 4–16). Each is scored by your reward function. The advantage of each completion is its reward minus the group mean, divided by the group standard deviation:
A_i = (r_i - mean(r_1..r_G)) / (std(r_1..r_G) + eps)
That group baseline is the whole trick: PPO needs a separately trained value network to estimate the baseline, GRPO gets it for free from siblings of the same prompt. No critic means roughly half the memory and far fewer moving parts. The policy is then updated with a clipped policy-gradient objective plus a KL penalty against a frozen reference model to stop it drifting into gibberish.
Two consequences worth remembering. If all G completions in a group get the same reward, the advantage is zero and the prompt contributes no gradient — so a task where the model always fails, or always succeeds, is wasted compute. And the sampling step, not the backward pass, dominates cost: a GRPO step generates G completions per prompt.
3. Environment
pip install "torch>=2.6" "transformers>=4.48" "trl>=0.14" peft accelerate datasets vllm math_verify
We use Qwen2.5-7B-Instruct as the policy and GSM8K-style maths word problems as the task, because correctness is trivially checkable. Substitute your own prompt set and grader; the code shape does not change.
from datasets import load_dataset
SYSTEM = (
"Reason step by step inside <think></think> tags, "
"then give only the final answer inside <answer></answer> tags."
)
def to_prompt(example):
return {
"prompt": [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": example["question"]},
],
"target": example["answer"].split("####")[-1].strip(),
}
train = load_dataset("openai/gsm8k", "main", split="train").map(to_prompt)
Note that prompt is a chat list, not a string — TRL applies the chat template for you — and that we carry the ground truth along in a target column so the reward function can see it.
4. Reward functions
Rewards are plain Python functions receiving the batch of completions and any extra dataset columns. Keep them cheap, deterministic and bounded. We use two: one for correctness, one small one for format.
import re
from math_verify import parse, verify
ANSWER_RE = re.compile(r"<answer>(.*?)</answer>", re.S)
def correctness_reward(completions, target, **kwargs):
out = []
for c, t in zip(completions, target):
text = c[0]["content"] if isinstance(c, list) else c
m = ANSWER_RE.search(text)
if not m:
out.append(0.0); continue
try:
out.append(1.0 if verify(parse(t), parse(m.group(1))) else 0.0)
except Exception:
out.append(0.0)
return out
FORMAT_RE = re.compile(r"^<think>.*?</think>\s*<answer>.*?</answer>\s*$", re.S)
def format_reward(completions, **kwargs):
texts = [c[0]["content"] if isinstance(c, list) else c for c in completions]
return [0.2 if FORMAT_RE.match(t.strip()) else 0.0 for t in texts]
Design rules we apply on every client engagement:
- Correctness should dominate. Auxiliary rewards (format, length, latency) belong in the 0.1–0.2 range against a 1.0 correctness signal. Make the format bonus too large and the model learns to emit beautifully tagged wrong answers.
- Never reward length directly. Long chains of thought are a symptom of harder problems, not a goal. If you pay per token, add a small penalty above a token budget instead.
- Fail closed. A grader that throws should return 0.0, not crash the run — but log the rate. A parser bug that silently zeroes 30% of correct answers looks exactly like a model that will not learn.
- Version your grader. Reward code is training data. Changing it mid-run invalidates every curve you were watching.
5. Training configuration
from trl import GRPOConfig, GRPOTrainer
from peft import LoraConfig
cfg = GRPOConfig(
output_dir="out/qwen-grpo",
learning_rate=1e-6,
bf16=True,
per_device_train_batch_size=8,
gradient_accumulation_steps=4,
num_generations=8, # G: completions per prompt
max_prompt_length=512,
max_completion_length=1024,
beta=0.04, # KL coefficient
temperature=0.9,
use_vllm=True, # generation backend
vllm_mode="colocate",
gradient_checkpointing=True,
logging_steps=1,
save_steps=100,
num_train_epochs=1,
report_to="tensorboard",
)
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-7B-Instruct",
reward_funcs=[correctness_reward, format_reward],
args=cfg,
train_dataset=train,
peft_config=LoraConfig(r=32, lora_alpha=64, lora_dropout=0.0,
target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()
The settings that matter most:
| Knob | Sane start | What goes wrong |
|---|---|---|
learning_rate | 1e-6 to 3e-6 | SFT-scale rates (1e-5+) collapse the policy within tens of steps |
num_generations | 8 | Below 4 the group baseline is too noisy to be useful |
beta (KL) | 0.02–0.05 | Too low: drift, degenerate text. Too high: nothing moves |
temperature | 0.8–1.0 | Greedy sampling gives identical siblings and zero advantage |
max_completion_length | task-dependent | Truncated reasoning is graded as wrong; check your truncation rate |
Using LoRA rather than full fine-tuning is not just a memory trick here: it keeps the policy close to the reference model by construction, which makes RL noticeably more stable. Full-parameter GRPO on a 7B policy wants multiple nodes and FSDP2 — see Distributed Training with FSDP2.
6. Make generation fast, or the run will not finish
A naive GRPO step with HuggingFace generate spends 80–90% of wall-clock time sampling. That is why TRL integrates vLLM. Two modes:
- Colocate (
vllm_mode="colocate") — vLLM shares the training GPUs. Simplest, best for a single node; budget GPU memory withvllm_gpu_memory_utilization(0.3 is a reasonable start when training on the same cards). - Server (
vllm_mode="server") — runtrl vllm-serve --model <policy>on dedicated GPUs and point the trainer at it. Better utilization at multi-node scale, one more process to babysit.
Other throughput levers, in the order we try them: raise num_generations before raising batch size (more reward signal per generation pass), enable gradient_checkpointing, cap max_completion_length to the 95th percentile of useful lengths, and drop prompts whose groups are consistently all-correct — they contribute no gradient. A run that stalls at 20 seconds per step usually has a generation problem, not a training problem.
7. Reading the curves
Watch four numbers, not the loss (GRPO's loss value is close to meaningless in isolation):
- Mean reward — should trend up over hundreds of steps, jaggedly.
- Reward standard deviation within groups — if it collapses to ~0, learning has stopped: either the model solves everything or nothing.
- KL to reference — a slow climb is fine; a hockey stick means the policy is running away and
betais too low. - Completion length — a sudden explosion toward
max_completion_lengthis the classic prelude to degenerate repetition.
And always print completions. Every few dozen steps, dump three samples from the group with the highest reward and read them. Reward hacking is visible instantly to a human and invisible in any metric: models that emit the answer twice to satisfy a sloppy regex, wrap empty <think></think> tags, or exploit the fact that GSM8K answers are usually small integers.
8. Evaluate honestly, then serve
Hold out a test split the grader has never influenced and compare three checkpoints: the base model, your SFT model, and the GRPO model — at the same temperature, same system prompt, and pass@1 rather than pass@k. Report accuracy plus mean output tokens; an RL model that gains three points of accuracy while tripling token count may be a net loss on your inference bill.
Then merge the adapter and serve as usual:
from peft import PeftModel
from transformers import AutoModelForCausalLM
import torch
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-7B-Instruct", dtype=torch.bfloat16, device_map="cpu")
merged = PeftModel.from_pretrained(base, "out/qwen-grpo").merge_and_unload()
merged.save_pretrained("out/qwen-grpo/merged", safe_serialization=True)
Deployment details, including quantization and OpenAI-compatible endpoints, are in Serving an LLM with vLLM.
Common failure modes
- Flat reward from step one. The base model never succeeds. Do SFT first, or make the task easier (fewer digits, shorter programs) and curriculum up.
- Reward rises, quality falls. Your grader is exploitable. Tighten the parser, add a schema check, or add an adversarial test case to the reward.
- Loss spikes to NaN. Learning rate too high, or
max_completion_lengthtruncation interacting with a malformed chat template. Drop the LR tenfold and re-check the template first. - Great offline, bad in production. Training temperature and serving temperature differ, or the production system prompt is not the training system prompt. Freeze both together.
What to try next
Once a single-reward GRPO loop works, the same machinery extends naturally: multi-turn tool-use tasks where the reward is whether the tool call succeeded, agentic workflows scored on end-state, and DPO or GRPO stages layered after SFT in a single pipeline. The engineering effort moves almost entirely into the grader — which is the right place for it, because the grader is a precise statement of what you actually want the model to do.
We build and run these pipelines for clients as part of our LLM fine-tuning and customization service: task and reward design, training infrastructure, honest evaluation, and serving-ready weights on your own hardware. Contact us if you have a task with a checkable answer and want help turning it into a better model.