Reinforcement learning with GRPO

Train a model against a reward function with group relative policy optimization.

Group relative policy optimization (GRPO) trains a model from rewards you compute for groups of generated completions. Updates favor completions above their group’s mean reward and suppress those below it. Use it when you can score outputs but can’t write ideal responses, or to improve accuracy beyond your demonstrations.

The reinforcement learning (RL) quickstart runs a complete recipe. This page covers GRPO on the post-training service, custom loops, tuning, async training, and troubleshooting.

Requirements

• A model with generator support and num_generator_replicas of one or more. Trainer-only models can’t run GRPO.
• Your dataset and reward function, or one of the cookbook’s four built-in datasets with rewards.
• For the cookbook, export WANDB_API_KEY, set WANDB_MODE=offline, or disable logging with wandb_project=None. GRPOConfig logs to Weights & Biases by default, and missing setup fails after provisioning. See Monitoring.

How post-training runs GRPO

Each step runs six operations:

1. sample: Generators return group_size completions per prompt and each generated token’s log-probability.
2. Reward: Your reward function scores each completion.
3. Advantages: Your code subtracts the group’s mean reward from each completion’s reward.
4. forward_backward: With LOSS_TYPE_GRPO, the trainer computes clipped policy-gradient loss and accumulates gradients. Samples carry completion tokens, advantages, and generator log-probabilities.
5. optim_step: The optimizer applies the gradients with the learning rate, gradient clipping, and other optimizer settings in adam_params.
6. weights_sync: The trainer publishes the updated weights to the generators before the next sample. Async training describes deferred sync modes.

Rewards and advantages run in your process, along with data loading and prompt construction, so all of them can be any code you write. The other four operations run on the model resource.

Run a GRPO recipe

Pass GRPOConfig to the cookbook’s train_single_turn recipe. Its dataset field names a registered dataset, which supplies the prompt format and the reward. To use another schema, register your own dataset and reward. The RL quickstart runs GSM8K end to end:

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    max_sampling_attempts=3,
    num_steps=200,
    max_model_length=1536,
    max_sample_tokens=768,
)

train_single_turn(config)

train_single_turn provisions a resource with num_generator_replicas generators (one by default) and one session, then stops both when it returns. Set model_resources_id instead of model to use an existing resource.

learning_rate defaults to 1e-5. Try 5e-6 for steadier training or 2e-5 for faster convergence. Change one setting at a time in small increments. To compare several rates at once, train multiple adapters on one resource.

The cookbook reference lists every field.

Rewards and advantages

The cookbook computes advantage = reward - group_mean. It leaves advantages on the reward’s scale instead of dividing by the group’s standard deviation as in the GRPO paper.

Groups with identical rewards carry no signal, so the cookbook builds no samples from them, though their generation is still billed. zero_variance_skip_ratio reports the fraction skipped.

Group size: group_size defaults to 8. Larger groups, such as 16, give steadier advantages and fewer zero-variance groups, with proportionally more sampling.

Temperature: temperature defaults to 0.7. Raise it toward 1.0 for more varied completions when groups collapse, or lower it toward 0.5 to exploit the current policy.

Write your own GRPO loop

Customize data preparation, sampling, rewards, or sample construction in your own loop. This example uses in-memory math problems and rewards a correct answer after ####:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.grpo.samples import (
    build_policy_sample,
    compute_group_advantages,
)
from post_training_cookbook.samples import make_prompt
from together.post_training import (
    AdamParams,
    ComputeConfig,
    GrpoLossParams,
    LoraConfig,
    LossConfig,
    ModelResourcesClient,
    SamplingParams,
)

MODEL = "Qwen/Qwen3.8-27B"
BATCH_SIZE = 16  # prompts per step
GROUP_SIZE = 8  # completions per prompt
problems = load_problems()  # a list of (messages, answer) pairs


def reward_fn(completion: str, answer: str) -> float:
    # 1.0 when the text after the last "####" is the answer, else 0.0
    return 1.0 if completion.split("####")[-1].strip() == answer else 0.0


# Load the model's chat template, with thinking off to keep completions short
codec = get_codec(MODEL, enable_thinking=False)

# Cap completion length, and use the cookbook's default temperature
sampling_params = SamplingParams(max_tokens=768, temperature=0.7)

# The GRPO loss, with the service's default clip band
loss = LossConfig(
    type="LOSS_TYPE_GRPO",
    grpo_params=GrpoLossParams(
        clip_low_threshold=0.8, clip_high_threshold=1.2
    ),
)

# Provision a model resource with one generator replica to sample from
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=1
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Each step trains on the next BATCH_SIZE problems
        for step, start in enumerate(range(0, len(problems), BATCH_SIZE)):
            batch = problems[start : start + BATCH_SIZE]

            # Render each prompt with the chat template, ready for a reply
            prompts = [
                codec.encode_messages(messages, add_generation_prompt=True)
                for messages, _ in batch
            ]

            # 1. Sample GROUP_SIZE completions per prompt from the generator
            results = session.generator.sample_batch(
                [make_prompt(prompt) for prompt in prompts],
                num_samples=GROUP_SIZE,
                sampling_params=sampling_params,
            )

            samples, rewards = [], []
            for prompt, (_, answer), result in zip(prompts, batch, results):
                # Token IDs can come back as strings, so convert them
                group = [
                    ([int(t) for t in seq.tokens], list(seq.logprobs))
                    for seq in result.sequences
                ]

                # 2. Score each completion with the reward function
                group_rewards = [
                    reward_fn(codec.decode(tokens), answer)
                    for tokens, _ in group
                ]
                rewards.extend(group_rewards)

                # 3. Subtract the group's mean reward to get each advantage.
                # A group where every completion scored the same has no
                # signal, so skip it.
                advantages = compute_group_advantages(group_rewards).values
                if advantages is None:
                    continue

                # Build one sample per completion. Zeros over the prompt keep
                # it out of the loss, and the generator's logprobs let the
                # trainer compute the clipping ratio.
                prompt_zeros = [0.0] * len(prompt)
                for (tokens, logprobs), advantage in zip(group, advantages):
                    samples.append(
                        build_policy_sample(
                            tokens=prompt + tokens,
                            logprobs=prompt_zeros + logprobs,
                            loss_mask=prompt_zeros + [1.0] * len(tokens),
                            advantage=advantage,
                        )
                    )

            # The mean reward should rise as the policy improves
            print(f"[step {step}] reward={sum(rewards) / len(rewards):.3f}")

            # Skip the update when every group in the step was skipped
            if not samples:
                continue

            # 4. Compute the clipped policy-gradient loss and its gradients
            session.trainer.forward_backward(samples=samples, loss=loss)

            # 5. Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=1e-5, weight_decay=0.01),
            )

            # 6. Send the new weights to the generator, so the next step
            # samples from the updated policy
            session.trainer.weights_sync()

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

Write a reward function covers how to choose scoring code for reward_fn.

compute_group_advantages returns values=None for identical rewards, so skip that group. build_policy_sample aligns input and target tokens and puts the advantage on every trained position.

sample_batch sends all prompts together, with num_samples requesting a whole group at once. Use _async variants for concurrent independent calls. Sampling overlaps other sampling and trainer operations, but synchronous weights_sync waits for in-flight sampling.

One SamplingParams applies to every prompt in a sample_batch call. The API reference lists fields and defaults. Two fields need care:

• stop: For models without stop-string support, omit it and truncate completions in your code. Unsupported stop strings fail as invalid input.
• seed: Leave it unset. It’s unsupported and causes an internal error when the generator uses multiple inference engines.

To score fixed text, call session.generator.compute_logprobs(prompt) or compute_logprobs_batch(prompts). Call weights_sync first if the trainer has stepped since the last sync. Entry 0 is a 0 placeholder because the first token has no preceding context. sample uses the same placeholder with prompt_logprobs=True.

The recipe adds cosine learning-rate decay, dynamic sampling, overlong handling, length budgets, validation, and periodic checkpoints.

Dynamic sampling

Dynamic sampling, from the DAPO paper, draws fresh batches until it collects batch_size groups with varied rewards or reaches max_sampling_attempts. Set that cap above 1 to enable it. The default is 1, and the GSM8K quickstart uses 3.

Each attempt generates a full batch, so a step can cost up to max_sampling_attempts batches of generation. With no usable groups, the cookbook skips the step and logs skipped: 0/<groups> groups had non-zero reward variance (single-turn) or samples=0 (multi-turn).

If zero_variance_skip_ratio stays high, also adjust task difficulty and temperature.

Clipping

Clipping limits how far an update moves the policy. For each token, the service computes the ratio of the trainer’s probability to the generator’s probability at sampling time. It suppresses gradients for positive-advantage tokens whose ratio exceeds clip_high_threshold, and for negative-advantage tokens whose ratio falls below clip_low_threshold.

Thresholds are absolute ratio bounds, not offsets from 1.0, so a DAPO epsilon pair (ε_low, ε_high) maps to (1 - ε_low, 1 + ε_high). Move either bound toward 1.0 for stronger clipping.

The cookbook leaves both thresholds unset (None), using the service’s [0.8, 1.2] default. To use DAPO’s asymmetric “Clip-Higher” band, set them to 0.8 and 1.28, allowing more room to raise underweighted tokens’ probabilities.

forward_backward.metrics reports clip rates, logged as fb/loss/clip/high_fraction and fb/loss/clip/low_fraction. See Service-emitted metrics.

Overlong responses

The cookbook offers two independent controls for completions near the token budget:

• Overlong penalty (on by default): A DAPO reward penalty that ramps up linearly over the last overlong_buffer_len tokens of the budget. The buffer defaults to max_sample_tokens // 8. overlong_penalty_factor scales the penalty, and 0 disables it.
• Overlong masking (off by default): overlong_masking=True excludes truncated completions from training while retaining their rewards in the group baseline. Single-turn training skips steps left with fewer than group_size samples.

Masking doesn’t turn the penalty off. To mask without penalizing, set overlong_masking=True and overlong_penalty_factor=0.

Length budgets

The cookbook derives two budgets from max_model_length (default 2048):

• max_sample_tokens: Generated tokens per completion, defaulting to max_model_length.
• max_prompt_length: Single-turn prompt filter, defaulting to max_model_length - max_sample_tokens. Longer prompts are dropped before training.

For single-turn training, set max_sample_tokens or max_prompt_length explicitly. Leaving both unset makes the prompt limit 0, dropping every prompt. With num_steps set, the run fails after provisioning and billing begin. Otherwise it exits before provisioning with no training needed.

Keep max_sample_tokens below max_model_length to leave room for the prompt. A truncated_ratio above about 30% indicates a tight budget or runaway-length output.

GLM-5.3 always thinks, at maximum reasoning effort by default. Allow a generous completion budget or set thinking_effort="high" or "low" to avoid truncating reasoning. Its codec rejects enable_thinking=False.

Variants

The cookbook keeps group sampling and advantages unchanged when switching objectives. Configure variants with these GRPOConfig fields:

• Policy loss: loss defaults to "grpo". The alternatives are "ppo", "cispo", "dppo", "dro", and "importance_sampling", described in Built-in loss functions. The alternatives sum token losses rather than averaging them, so their loss grows with trained-token count and may require learning-rate retuning.
• Clip band: clip_low_threshold and clip_high_threshold apply to "grpo", "ppo", and "cispo". Leave them None for service defaults: [0.8, 1.2] for GRPO/PPO and [0, 4] for CISPO, or set your own bounds.
• Metrics: GRPO/PPO report fb/loss/clip/* and fb/loss/ratio/mean. Other objectives use keys such as fb/ratio_mean. CISPO reports fb/clip_fraction, and scales gradients with clipping instead of zeroing them.
• DRO penalty: "dro" requires dro_beta for its quadratic log-ratio penalty.
• Loss aggregation: loss_agg_type defaults to GRPO_LOSS_AGGREGATION_TYPE_FIXED_HORIZON, dividing the masked sum by all batch tokens, including padding. TOKEN_MEAN divides by trained tokens. SEQUENCE_MEAN averages within each sequence, then across sequences, giving each equal weight. Keep aggregation fixed when comparing runs because it changes the effective learning rate.
• GSPO: Set ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE" for a sequence-level importance ratio and clipping over the whole completion.

loss_agg_type and ratio_type apply only to "grpo". The config rejects changes to these, clip bounds, or dro_beta for objectives that don’t use them.

For a Kullback-Leibler (KL) reference-policy penalty, use your own loop with GrpoLossParams.beta > 0 and a reference_logprobs row per sample. The cookbook config has no KL setting. For other objectives, write a custom loss function.

Built-in loss functions

In your own loop, pass a LossConfig to forward_backward, selecting the objective with type and configuring its matching *_params. Mismatched params fail client-side. Each objective reads the loss_fn_inputs rows below. See the API reference for params and defaults.

type	Params	Sample rows required	Use
LOSS_TYPE_CROSS_ENTROPY	cross_entropy_params	target_tokens, weights	Supervised fine-tuning (SFT). Weighted next-token cross-entropy. weights may be fractional.
LOSS_TYPE_GRPO	grpo_params	target_tokens, advantages, logprobs, plus reference_logprobs when beta > 0	GRPO, and GSPO with ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE".
LOSS_TYPE_PPO	ppo_params	target_tokens, advantages, logprobs	Clipped surrogate with a token-level ratio and no KL term. The default clip range is [0.8, 1.2].
LOSS_TYPE_CISPO	cispo_params	target_tokens, advantages, logprobs	Clipped ratio applied as a detached coefficient, so no token’s gradient is zeroed. The default clip range is [0, 4].
LOSS_TYPE_DRO	dro_params	target_tokens, advantages, logprobs	Policy gradient plus a quadratic log-ratio penalty. beta is required.
LOSS_TYPE_IMPORTANCE_SAMPLING	None	target_tokens, advantages, logprobs	Unclipped -(ratio * advantage).
LOSS_TYPE_DPPO	dppo_params	target_tokens, advantages, logprobs	Drops tokens whose probability moved past delta_low or delta_high in probability space, and trains the rest with the plain importance-sampling gradient. Both thresholds default to 0.15.

The advantage-based objectives accept an optional binary weights mask. Use 1 on completion tokens and 0 elsewhere. Omitting it trains every position, including the prompt. Only cross-entropy accepts fractional weights.

Async training

Lockstep is the default: rollouts finish before training, leaving generators idle during training and the trainer idle during sampling. Async overlaps collection and training.

The cookbook supports async in train_multi_turn, where long, variable-length episodes create larger idle gaps. train_single_turn rejects max_steps_off_policy.

Three settings control async training:

• max_steps_off_policy: None means lockstep. An integer enables continuous rollout workers and sets the maximum group age in optimizer steps, measured from rollout start. 0 admits only groups started under the current policy.
• groups_in_flight: Concurrent rollout groups, defaulting to batch_size.
• weight_sync_type: WEIGHT_SYNC_TYPE_SYNCHRONOUS publishes inline by default. WEIGHT_SYNC_TYPE_BACKGROUND_PUBLISH returns immediately after handing publication to a server worker. WEIGHT_SYNC_TYPE_PIPELINE also defers publication and lets in-flight generation finish under updated weights.

Async collection supports all sync modes. In the cookbook, deferred modes require max_steps_off_policy to bound and measure drift.

WEIGHT_SYNC_TYPE_PIPELINE requires full-weight training (use_lora=False). The server rejects it on adapter sessions, and the cookbook checks this before provisioning.

Pass this config, your tasks, and an environment factory to train_multi_turn, as in Multi-turn environments:

from post_training_cookbook.grpo.config import GRPOConfig

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    max_turns=6,
    batch_size=4,
    episodes_per_task=8,
    max_steps_off_policy=1,
    groups_in_flight=8,
    num_generator_replicas=2,
)

Start with max_steps_off_policy=1. At 0, arriving groups are often already one version behind and get dropped.

Watch async/train_wait_s and async/dropped_stale_groups in the async collection metrics. Stale groups are counted separately and don’t inflate zero_variance_skip_ratio. Run recipes/grpo/tool_use_game_async.py for a complete example.

Troubleshooting

• zero_variance_skip_ratio stays above 50%: Check identical_nonzero_reward_skip_ratio (single-turn only) to distinguish identical nonzero rewards from all-zero rewards. For easy tasks, use harder data or raise temperature. For hard tasks, use easier data or a stronger model. Increasing group_size to 16 or max_sampling_attempts above 1 also recovers more groups, at the cost of extra sampling.
• Checkpoint creation fails with You need to complete at least one training step before you can create an inference checkpoint: Every step was skipped, although sampling was billed. Check skipped: (single-turn) or samples=0 (multi-turn). Adjust difficulty for identical rewards, or the length budget if overlong masking left too few samples.
• fb/loss/ratio/mean is far from 1.0 on the first step: The trainer and generator initially share weights, so the ratio should start near 1.0. In a hand-built loop, check logprobs alignment with target_tokens: drop the first value of each per-token array. Misaligned advantages don’t affect this metric.
• Out of memory or timeouts: Reduce batch_size, max_model_length, or group_size, or raise timeout on the config.
• The session isn’t active: It expired after an hour without operations, was stopped, or hit an error. See Manage model resources troubleshooting.

Next steps

Reward functions

Bring your own dataset and reward to the loop.

Multi-turn environments

Train tool-use agents over whole episodes.

Monitoring

Every metric the loop logs and how to act on it.

Cookbook reference

Every GRPOConfig field and its default.