> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Reinforcement learning with GRPO
> Train a model against a reward function with group relative policy optimization.
Group relative policy optimization (GRPO) trains a model from rewards you compute for groups of generated completions. Updates favor completions above their group's mean reward and suppress those below it. Use it when you can score outputs but can't write ideal responses, or to improve accuracy beyond your demonstrations.
The [reinforcement learning (RL) quickstart](/docs/post-training/quickstart-rl) runs a complete recipe. This page covers GRPO on the post-training service, custom loops, tuning, async training, and troubleshooting.
## Requirements
* A [model with generator support](/docs/post-training/supported-models) and `num_generator_replicas` of one or more. Trainer-only models can't run GRPO.
* Your dataset and [reward function](/docs/post-training/reward-functions), or one of the cookbook's [four built-in datasets with rewards](/docs/post-training/reward-functions#built-in-datasets).
* For the cookbook, export `WANDB_API_KEY`, set `WANDB_MODE=offline`, or disable logging with `wandb_project=None`. `GRPOConfig` logs to Weights & Biases by default, and missing setup fails after provisioning. See [Monitoring](/docs/post-training/monitoring).
## How post-training runs GRPO
Each step runs six operations:
1. **`sample`:** Generators return `group_size` completions per prompt and each generated token's log-probability.
2. **Reward:** Your reward function scores each completion.
3. **Advantages:** Your code subtracts the group's mean reward from each completion's reward.
4. **`forward_backward`:** With `LOSS_TYPE_GRPO`, the trainer computes clipped policy-gradient loss and accumulates gradients. Samples carry completion tokens, advantages, and generator log-probabilities.
5. **`optim_step`:** The optimizer applies the gradients with the learning rate, gradient clipping, and other [optimizer settings](/reference/post-training/operations-optim-step-create) in `adam_params`.
6. **`weights_sync`:** The trainer publishes the updated weights to the generators before the next `sample`. [Async training](#async-training) describes deferred sync modes.
Rewards and advantages run in your process, along with data loading and prompt construction, so all of them can be any code you write. The other four operations run on the model resource.
## Run a GRPO recipe
Pass `GRPOConfig` to the cookbook's `train_single_turn` recipe. Its `dataset` field names a registered dataset, which supplies the prompt format and the reward. To use another schema, [register your own dataset and reward](/docs/post-training/reward-functions#register-a-dataset-and-reward). The [RL quickstart](/docs/post-training/quickstart-rl) runs GSM8K end to end:
```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn
config = GRPOConfig(
model="Qwen/Qwen3.6-35B-A3B",
dataset="gsm8k",
enable_thinking=False,
max_sampling_attempts=3,
num_steps=200,
max_model_length=1536,
max_sample_tokens=768,
)
train_single_turn(config)
```
`train_single_turn` provisions a resource with `num_generator_replicas` generators (one by default) and one session, then stops both when it returns. Set `model_resources_id` instead of `model` to [use an existing resource](/docs/post-training/model-resources).
`learning_rate` defaults to `1e-5`. Try `5e-6` for steadier training or `2e-5` for faster convergence. Change one setting at a time in small increments. To compare several rates at once, [train multiple adapters on one resource](/docs/post-training/multi-lora).
The [cookbook reference](/docs/post-training/cookbook#grpoconfig) lists every field.
## Rewards and advantages
The cookbook computes `advantage = reward - group_mean`. It leaves advantages on the reward's scale instead of dividing by the group's standard deviation as in the GRPO paper.
Groups with identical rewards carry no signal, so the cookbook builds no samples from them, though their generation is still billed. `zero_variance_skip_ratio` reports the fraction skipped.
**Group size:** `group_size` defaults to `8`. Larger groups, such as `16`, give steadier advantages and fewer zero-variance groups, with proportionally more sampling.
**Temperature:** `temperature` defaults to `0.7`. Raise it toward `1.0` for more varied completions when groups collapse, or lower it toward `0.5` to exploit the current policy.
## Write your own GRPO loop
Customize data preparation, sampling, rewards, or sample construction in your own loop. This example uses in-memory math problems and rewards a correct answer after `####`:
```python theme={null}
from post_training_cookbook.codecs import get_codec
from post_training_cookbook.grpo.samples import (
build_policy_sample,
compute_group_advantages,
)
from post_training_cookbook.samples import make_prompt
from together.post_training import (
AdamParams,
ComputeConfig,
GrpoLossParams,
LoraConfig,
LossConfig,
ModelResourcesClient,
SamplingParams,
)
MODEL = "Qwen/Qwen3.8-27B"
BATCH_SIZE = 16 # prompts per step
GROUP_SIZE = 8 # completions per prompt
problems = load_problems() # a list of (messages, answer) pairs
def reward_fn(completion: str, answer: str) -> float:
# 1.0 when the text after the last "####" is the answer, else 0.0
return 1.0 if completion.split("####")[-1].strip() == answer else 0.0
# Load the model's chat template, with thinking off to keep completions short
codec = get_codec(MODEL, enable_thinking=False)
# Cap completion length, and use the cookbook's default temperature
sampling_params = SamplingParams(max_tokens=768, temperature=0.7)
# The GRPO loss, with the service's default clip band
loss = LossConfig(
type="LOSS_TYPE_GRPO",
grpo_params=GrpoLossParams(
clip_low_threshold=0.8, clip_high_threshold=1.2
),
)
# Provision a model resource with one generator replica to sample from
with ModelResourcesClient.create(
base_model=MODEL,
compute_config=ComputeConfig(
gpu_type="B200-SXM", num_generator_replicas=1
),
) as resources:
# Start a training session that trains a LoRA adapter
with resources.create_session(
lora_config=LoraConfig(rank=16, alpha=16)
) as session:
# Each step trains on the next BATCH_SIZE problems
for step, start in enumerate(range(0, len(problems), BATCH_SIZE)):
batch = problems[start : start + BATCH_SIZE]
# Render each prompt with the chat template, ready for a reply
prompts = [
codec.encode_messages(messages, add_generation_prompt=True)
for messages, _ in batch
]
# 1. Sample GROUP_SIZE completions per prompt from the generator
results = session.generator.sample_batch(
[make_prompt(prompt) for prompt in prompts],
num_samples=GROUP_SIZE,
sampling_params=sampling_params,
)
samples, rewards = [], []
for prompt, (_, answer), result in zip(prompts, batch, results):
# Token IDs can come back as strings, so convert them
group = [
([int(t) for t in seq.tokens], list(seq.logprobs))
for seq in result.sequences
]
# 2. Score each completion with the reward function
group_rewards = [
reward_fn(codec.decode(tokens), answer)
for tokens, _ in group
]
rewards.extend(group_rewards)
# 3. Subtract the group's mean reward to get each advantage.
# A group where every completion scored the same has no
# signal, so skip it.
advantages = compute_group_advantages(group_rewards).values
if advantages is None:
continue
# Build one sample per completion. Zeros over the prompt keep
# it out of the loss, and the generator's logprobs let the
# trainer compute the clipping ratio.
prompt_zeros = [0.0] * len(prompt)
for (tokens, logprobs), advantage in zip(group, advantages):
samples.append(
build_policy_sample(
tokens=prompt + tokens,
logprobs=prompt_zeros + logprobs,
loss_mask=prompt_zeros + [1.0] * len(tokens),
advantage=advantage,
)
)
# The mean reward should rise as the policy improves
print(f"[step {step}] reward={sum(rewards) / len(rewards):.3f}")
# Skip the update when every group in the step was skipped
if not samples:
continue
# 4. Compute the clipped policy-gradient loss and its gradients
session.trainer.forward_backward(samples=samples, loss=loss)
# 5. Update the adapter's weights
session.trainer.optim_step(
adam_params=AdamParams(learning_rate=1e-5, weight_decay=0.01),
)
# 6. Send the new weights to the generator, so the next step
# samples from the updated policy
session.trainer.weights_sync()
# Save the adapter as an inference checkpoint you can deploy
checkpoint = session.create_inference_checkpoint()
print(checkpoint.id)
```
[Write a reward function](/docs/post-training/reward-functions#write-a-reward-function) covers how to choose scoring code for `reward_fn`.
`compute_group_advantages` returns `values=None` for identical rewards, so skip that group. `build_policy_sample` aligns input and target tokens and puts the advantage on every trained position.
`sample_batch` sends all prompts together, with `num_samples` requesting a whole group at once. Use `_async` variants for concurrent independent calls. Sampling overlaps other sampling and trainer operations, but synchronous `weights_sync` waits for in-flight sampling.
One `SamplingParams` applies to every prompt in a `sample_batch` call. The [API reference](/reference/post-training/operations-sample-create) lists fields and defaults. Two fields need care:
* **`stop`:** For models without stop-string support, omit it and truncate completions in your code. Unsupported stop strings fail as invalid input.
* **`seed`:** Leave it unset. It's unsupported and causes an internal error when the generator uses multiple inference engines.
To score fixed text, call `session.generator.compute_logprobs(prompt)` or `compute_logprobs_batch(prompts)`. Call `weights_sync` first if the trainer has stepped since the last sync. Entry `0` is a `0` placeholder because the first token has no preceding context. `sample` uses the same placeholder with `prompt_logprobs=True`.
The recipe adds cosine learning-rate decay, [dynamic sampling](#dynamic-sampling), [overlong handling](#overlong-responses), [length budgets](#length-budgets), validation, and periodic checkpoints.
## Dynamic sampling
Dynamic sampling, from the DAPO paper, draws fresh batches until it collects `batch_size` groups with varied rewards or reaches `max_sampling_attempts`. Set that cap above `1` to enable it. The default is `1`, and the GSM8K quickstart uses `3`.
Each attempt generates a full batch, so a step can cost up to `max_sampling_attempts` batches of generation. With no usable groups, the cookbook skips the step and logs `skipped: 0/ groups had non-zero reward variance` (single-turn) or `samples=0` (multi-turn).
If `zero_variance_skip_ratio` stays high, also adjust task difficulty and temperature.
## Clipping
Clipping limits how far an update moves the policy. For each token, the service computes the ratio of the trainer's probability to the generator's probability at sampling time. It suppresses gradients for positive-advantage tokens whose ratio exceeds `clip_high_threshold`, and for negative-advantage tokens whose ratio falls below `clip_low_threshold`.
Thresholds are absolute ratio bounds, not offsets from `1.0`, so a DAPO epsilon pair (ε\_low, ε\_high) maps to `(1 - ε_low, 1 + ε_high)`. Move either bound toward `1.0` for stronger clipping.
The cookbook leaves both thresholds unset (`None`), using the service's `[0.8, 1.2]` default. To use DAPO's asymmetric "Clip-Higher" band, set them to `0.8` and `1.28`, allowing more room to raise underweighted tokens' probabilities.
`forward_backward.metrics` reports clip rates, logged as `fb/loss/clip/high_fraction` and `fb/loss/clip/low_fraction`. See [Service-emitted metrics](/docs/post-training/monitoring#service-emitted-metrics).
## Overlong responses
The cookbook offers two independent controls for completions near the token budget:
* **Overlong penalty (on by default):** A DAPO reward penalty that ramps up linearly over the last `overlong_buffer_len` tokens of the budget. The buffer defaults to `max_sample_tokens // 8`. `overlong_penalty_factor` scales the penalty, and `0` disables it.
* **Overlong masking (off by default):** `overlong_masking=True` excludes truncated completions from training while retaining their rewards in the group baseline. Single-turn training skips steps left with fewer than `group_size` samples.
Masking doesn't turn the penalty off. To mask without penalizing, set `overlong_masking=True` and `overlong_penalty_factor=0`.
## Length budgets
The cookbook derives two budgets from `max_model_length` (default `2048`):
* **`max_sample_tokens`:** Generated tokens per completion, defaulting to `max_model_length`.
* **`max_prompt_length`:** Single-turn prompt filter, defaulting to `max_model_length - max_sample_tokens`. Longer prompts are dropped before training.
For single-turn training, set `max_sample_tokens` or `max_prompt_length` explicitly. Leaving both unset makes the prompt limit `0`, dropping every prompt. With `num_steps` set, the run fails after provisioning and billing begin. Otherwise it exits before provisioning with no training needed.
Keep `max_sample_tokens` below `max_model_length` to leave room for the prompt. A `truncated_ratio` above about 30% indicates a tight budget or runaway-length output.
GLM-5.3 always thinks, at maximum reasoning effort by default. Allow a generous completion budget or set `thinking_effort="high"` or `"low"` to avoid truncating reasoning. Its codec rejects `enable_thinking=False`.
## Variants
The cookbook keeps group sampling and advantages unchanged when switching objectives. Configure variants with these `GRPOConfig` fields:
* **Policy loss:** `loss` defaults to `"grpo"`. The alternatives are `"ppo"`, `"cispo"`, `"dppo"`, `"dro"`, and `"importance_sampling"`, described in [Built-in loss functions](#built-in-loss-functions). The alternatives sum token losses rather than averaging them, so their loss grows with trained-token count and may require learning-rate retuning.
* **Clip band:** `clip_low_threshold` and `clip_high_threshold` apply to `"grpo"`, `"ppo"`, and `"cispo"`. Leave them `None` for service defaults: `[0.8, 1.2]` for GRPO/PPO and `[0, 4]` for CISPO, or set your own bounds.
* **Metrics:** GRPO/PPO report `fb/loss/clip/*` and `fb/loss/ratio/mean`. Other objectives use keys such as `fb/ratio_mean`. CISPO reports `fb/clip_fraction`, and scales gradients with clipping instead of zeroing them.
* **DRO penalty:** `"dro"` requires `dro_beta` for its quadratic log-ratio penalty.
* **Loss aggregation:** `loss_agg_type` defaults to `GRPO_LOSS_AGGREGATION_TYPE_FIXED_HORIZON`, dividing the masked sum by all batch tokens, including padding. `TOKEN_MEAN` divides by trained tokens. `SEQUENCE_MEAN` averages within each sequence, then across sequences, giving each equal weight. Keep aggregation fixed when comparing runs because it changes the effective learning rate.
* **GSPO:** Set `ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE"` for a sequence-level importance ratio and clipping over the whole completion.
`loss_agg_type` and `ratio_type` apply only to `"grpo"`. The config rejects changes to these, clip bounds, or `dro_beta` for objectives that don't use them.
For a Kullback-Leibler (KL) reference-policy penalty, use [your own loop](#write-your-own-grpo-loop) with `GrpoLossParams.beta > 0` and a `reference_logprobs` row per sample. The cookbook config has no KL setting. For other objectives, [write a custom loss function](/docs/post-training/custom-loss-functions).
### Built-in loss functions
In your own loop, pass a `LossConfig` to `forward_backward`, selecting the objective with `type` and configuring its matching `*_params`. Mismatched params fail client-side. Each objective reads the `loss_fn_inputs` rows below. See the [API reference](/reference/post-training/operations-forward-backward-create) for params and defaults.
| `type` | Params | Sample rows required | Use |
| - | - | - | - |
| `LOSS_TYPE_CROSS_ENTROPY` | `cross_entropy_params` | `target_tokens`, `weights` | [Supervised fine-tuning (SFT)](/docs/post-training/sft). Weighted next-token cross-entropy. `weights` may be fractional. |
| `LOSS_TYPE_GRPO` | `grpo_params` | `target_tokens`, `advantages`, `logprobs`, plus `reference_logprobs` when `beta > 0` | GRPO, and GSPO with `ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE"`. |
| `LOSS_TYPE_PPO` | `ppo_params` | `target_tokens`, `advantages`, `logprobs` | Clipped surrogate with a token-level ratio and no KL term. The default clip range is `[0.8, 1.2]`. |
| `LOSS_TYPE_CISPO` | `cispo_params` | `target_tokens`, `advantages`, `logprobs` | Clipped ratio applied as a detached coefficient, so no token's gradient is zeroed. The default clip range is `[0, 4]`. |
| `LOSS_TYPE_DRO` | `dro_params` | `target_tokens`, `advantages`, `logprobs` | Policy gradient plus a quadratic log-ratio penalty. `beta` is required. |
| `LOSS_TYPE_IMPORTANCE_SAMPLING` | None | `target_tokens`, `advantages`, `logprobs` | Unclipped `-(ratio * advantage)`. |
| `LOSS_TYPE_DPPO` | `dppo_params` | `target_tokens`, `advantages`, `logprobs` | Drops tokens whose probability moved past `delta_low` or `delta_high` in probability space, and trains the rest with the plain importance-sampling gradient. Both thresholds default to `0.15`. |
The advantage-based objectives accept an optional binary `weights` mask. Use `1` on completion tokens and `0` elsewhere. Omitting it trains every position, including the prompt. Only cross-entropy accepts fractional weights.
## Async training
Lockstep is the default: rollouts finish before training, leaving generators idle during training and the trainer idle during sampling. Async overlaps collection and training.
The cookbook supports async in `train_multi_turn`, where long, variable-length episodes create larger idle gaps. `train_single_turn` rejects `max_steps_off_policy`.
Three settings control async training:
* **`max_steps_off_policy`:** `None` means lockstep. An integer enables continuous rollout workers and sets the maximum group age in optimizer steps, measured from rollout start. `0` admits only groups started under the current policy.
* **`groups_in_flight`:** Concurrent rollout groups, defaulting to `batch_size`.
* **`weight_sync_type`:** `WEIGHT_SYNC_TYPE_SYNCHRONOUS` publishes inline by default. `WEIGHT_SYNC_TYPE_BACKGROUND_PUBLISH` returns immediately after handing publication to a server worker. `WEIGHT_SYNC_TYPE_PIPELINE` also defers publication and lets in-flight generation finish under updated weights.
Async collection supports all sync modes. In the cookbook, deferred modes require `max_steps_off_policy` to bound and measure drift.
`WEIGHT_SYNC_TYPE_PIPELINE` requires full-weight training (`use_lora=False`). The server rejects it on adapter sessions, and the cookbook checks this before provisioning.
Pass this config, your tasks, and an environment factory to `train_multi_turn`, as in [Multi-turn environments](/docs/post-training/multi-turn-environments#run-multi-turn-training):
```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
config = GRPOConfig(
model="Qwen/Qwen3.6-35B-A3B",
max_turns=6,
batch_size=4,
episodes_per_task=8,
max_steps_off_policy=1,
groups_in_flight=8,
num_generator_replicas=2,
)
```
Start with `max_steps_off_policy=1`. At `0`, arriving groups are often already one version behind and get dropped.
Watch `async/train_wait_s` and `async/dropped_stale_groups` in the [async collection metrics](/docs/post-training/monitoring#async-collection-metrics). Stale groups are counted separately and don't inflate `zero_variance_skip_ratio`. Run `recipes/grpo/tool_use_game_async.py` for a complete example.
## Troubleshooting
* **`zero_variance_skip_ratio` stays above 50%:** Check `identical_nonzero_reward_skip_ratio` (single-turn only) to distinguish identical nonzero rewards from all-zero rewards. For easy tasks, use harder data or raise `temperature`. For hard tasks, use easier data or a stronger model. Increasing `group_size` to `16` or `max_sampling_attempts` above `1` also recovers more groups, at the cost of extra sampling.
* **Checkpoint creation fails with `You need to complete at least one training step before you can create an inference checkpoint`:** Every step was skipped, although sampling was billed. Check `skipped:` (single-turn) or `samples=0` (multi-turn). Adjust difficulty for identical rewards, or the [length budget](#length-budgets) if [overlong masking](#overlong-responses) left too few samples.
* **`fb/loss/ratio/mean` is far from `1.0` on the first step:** The trainer and generator initially share weights, so the ratio should start near `1.0`. In a hand-built loop, check `logprobs` alignment with `target_tokens`: [drop the first value of each per-token array](/docs/post-training/concepts#build-samples-by-hand). Misaligned `advantages` don't affect this metric.
* **Out of memory or timeouts:** Reduce `batch_size`, `max_model_length`, or `group_size`, or raise `timeout` on the config.
* **The session isn't active:** It expired after an hour without operations, was stopped, or hit an error. See [Manage model resources troubleshooting](/docs/post-training/model-resources#troubleshooting).
## Next steps
Bring your own dataset and reward to the loop.
Train tool-use agents over whole episodes.
Every metric the loop logs and how to act on it.
Every `GRPOConfig` field and its default.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.