Direct preference optimization

Train on preference pairs with a custom loss function that runs in your own process.

Direct preference optimization (DPO) raises the likelihood of a chosen response relative to a rejected response to the same prompt, measured against a frozen snapshot of the model from before training. It trains on existing responses without generating new ones. Use it for preference data from human ratings, reward-model scores, or comparisons between systems.

On the post-training service, DPO runs as a custom loss function that your own process computes. Fine-tuning jobs also offer DPO, with a fixed data format and no code to write. To choose between them, see Post-training vs. standard fine-tuning.

How post-training runs DPO

DPO uses a trainer-only resource (num_generator_replicas=0). Before any updates, the cookbook scores all training and validation pairs with forward to snapshot the reference policy. Each training step then runs:

1. forward: The trainer returns per-target-token log-probabilities without accumulating gradients.
2. Local loss: Your process computes the loss in PyTorch from live and reference log-probabilities, then calls .backward() to get per-token log-probability gradients.
3. custom_forward_backward: Your process sends those gradients to the trainer, which repeats the forward pass and backpropagates through the model to accumulate parameter gradients.
4. optim_step: The optimizer applies the accumulated gradients.

Write a custom loss function covers this pattern for other objectives and the requirements for each pass.

Run the DPO recipe

The cookbook’s DPO loop trains on HelpSteer3 preference pairs:

from post_training_cookbook.dpo.config import DPOConfig
from post_training_cookbook.dpo.train import train

config = DPOConfig(
    model="Qwen/Qwen3.8-27B",
    enable_thinking=False,
    dataset_name="nvidia/HelpSteer3",
    dataset_split="train",
    dataset_subsample=1600,
    batch_size=8,
    learning_rate=5e-6,
    beta=0.1,
    max_length=4096,
    max_steps=200,
    precompute_ref_batch_size=512,
)

train(config)

Run uv run python recipes/dpo/helpsteer3.py. train provisions a trainer-only resource and one session, then stops both when it returns. To use an existing resource, attach through the SDK, because DPOConfig has no model_resources_id field.

Settings worth knowing:

• beta: The temperature on the implicit reward. Higher values keep the policy closer to the reference.
• enable_thinking: Set it to False for Qwen preference data without reasoning, such as HelpSteer3. The catalog’s Qwen models default to thinking on, making these pairs unrenderable, so train raises ValueError before provisioning. GLM-5.3 accepts these pairs with thinking on and rejects False, so leave the setting unset for it.
• label_smoothing: Conservative-DPO smoothing. Off by default.
• learning_rate: The default is 5e-6 with cosine decay, lower than for supervised fine-tuning (SFT).
• precompute_ref_batch_size: Samples per forward pass for reference precomputation and validation. Lower it if long pairs cause out-of-memory errors.
• val_interval: The number of steps between validation runs on held-out pairs.

The cookbook reference lists every DPOConfig field.

Prepare preference pairs

You choose your dataset schema and prepare tokenized samples in your code. The cookbook’s build_preference_pair takes two full conversations with identical history, ending in a chosen and a rejected assistant response. It applies the model’s chat template and returns two SFT-style Sample objects, masking only the final responses:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.dpo.samples import build_preference_pair

codec = get_codec("zai-org/GLM-5.3")

context = [{"role": "user", "content": "Explain recursion in one sentence."}]
chosen = context + [
    {
        "role": "assistant",
        "content": "Recursion is when a function solves a problem by calling itself on a smaller version of it.",
    }
]
rejected = context + [
    {"role": "assistant", "content": "Recursion is a programming concept."}
]

pair = build_preference_pair(
    chosen_messages=chosen,
    rejected_messages=rejected,
    codec=codec,
    max_length=4096,
)
if pair is not None:
    chosen_sample, rejected_sample = pair

Skip pairs that return None, which means truncation to max_length left at least one side with no trained token. Unrenderable chat templates raise ValueError. For Qwen data without reasoning, pass enable_thinking=False to get_codec.

The cookbook interleaves samples as [chosen_0, rejected_0, chosen_1, rejected_1, ...]: chosen responses at even indexes, rejected responses at odd indexes.

Use your own preference dataset

Select data for the cookbook’s Hugging Face datasets loader with dataset_name, dataset_config, and dataset_split. Pass train a to_pair function that returns (chosen_messages, rejected_messages) per row, or None to skip it. The default, helpsteer3_pair in post_training_cookbook.dpo.train, chooses by HelpSteer3’s overall_preference and skips ties. This example maps prompt, chosen, and rejected text columns:

def preference_pair(row):
    context = [{"role": "user", "content": row["prompt"]}]
    chosen = context + [{"role": "assistant", "content": row["chosen"]}]
    rejected = context + [{"role": "assistant", "content": row["rejected"]}]
    return chosen, rejected


train(config, to_pair=preference_pair)

If your dataset has no "validation" split, set val_split to a held-out split it has, or disable validation with val_interval=0. Reusing the training split inflates val/preference_accuracy.

Write your own DPO loop

Customize data loading, preprocessing, tokenization, or the objective in your own loop. This example renders in-memory preference pairs with the cookbook’s chat-template codec:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.custom_loss import (
    compute_custom_forward_backward,
    compute_logprobs,
)
from post_training_cookbook.dpo.loss import make_dpo_loss
from post_training_cookbook.dpo.samples import build_preference_pair
from post_training_cookbook.samples import extract_loss_mask
from together.post_training import (
    AdamParams,
    ComputeConfig,
    LoraConfig,
    ModelResourcesClient,
)

MODEL = "Qwen/Qwen3.8-27B"
PAIRS_PER_STEP = 8
REF_CHUNK_SIZE = 64
pairs = load_pairs()  # a list of (chosen_messages, rejected_messages)

# Load the model's chat template, with thinking off for data without reasoning
codec = get_codec(MODEL, enable_thinking=False)

# Render each pair into a chosen sample and a rejected sample, interleaved
# so even indexes hold chosen responses and odd indexes hold rejected ones
samples = []
for chosen, rejected in pairs:
    pair = build_preference_pair(
        chosen_messages=chosen,
        rejected_messages=rejected,
        codec=codec,
        max_length=4096,
    )
    # Skip pairs that truncation left with no response tokens
    if pair is not None:
        samples.extend(pair)

# The loss mask on each sample covers only its final response
masks = [extract_loss_mask(sample) for sample in samples]

# Provision a trainer-only model resource (DPO never samples)
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=0
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Before any update, score every sample to snapshot the reference
        # policy. Keep each sample's summed logprob over its response.
        ref_sums = []
        for start in range(0, len(samples), REF_CHUNK_SIZE):
            chunk = samples[start : start + REF_CHUNK_SIZE]
            chunk_masks = masks[start : start + REF_CHUNK_SIZE]
            for logprobs, mask in zip(
                compute_logprobs(session, chunk), chunk_masks
            ):
                ref_sums.append(sum(lp * m for lp, m in zip(logprobs, mask)))

        # Each step trains on the next PAIRS_PER_STEP pairs. The step size is
        # even, so a pair's two samples always land in the same batch.
        batch_size = 2 * PAIRS_PER_STEP
        for step, start in enumerate(range(0, len(samples), batch_size)):
            batch = samples[start : start + batch_size]

            # Bind the DPO loss to this batch's reference sums and masks
            loss_fn = make_dpo_loss(
                ref_sums[start : start + batch_size],
                masks[start : start + batch_size],
                beta=0.1,
            )

            # Score the batch, compute the loss and its gradients in PyTorch,
            # and send the gradients to the trainer to accumulate
            result = compute_custom_forward_backward(session, batch, loss_fn)

            # Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=5e-6, weight_decay=0.1),
            )

            # The service doesn't report a loss for a custom loss function,
            # so this is the only record of it
            print(f"[step {step}] loss={result.loss:.4f}")

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

make_dpo_loss binds the loss to a batch’s reference sums and response masks. It sums masked token log-probabilities, computes the chosen-minus-rejected margin relative to the reference, and averages -log σ(β · margin) across pairs. Pass label_smoothing for conservative DPO.

compute_custom_forward_backward runs the two trainer passes and local PyTorch autograd. Log its result.loss yourself, because the service doesn’t report a loss for a custom loss function.

compute_logprobs scores its whole batch in one forward pass, so the loop chunks the reference snapshot by REF_CHUNK_SIZE. Lower it if long pairs run out of memory. The recipe uses precompute_ref_batch_size and adds a cosine learning-rate schedule, validation, and periodic checkpoints.

Validate with preference accuracy

DPO validates with implicit-reward preference accuracy, rather than token-level accuracy. Every val_interval steps, the recipe measures the fraction of held-out pairs whose reward shift favors the chosen response. Watch val/preference_accuracy and val/mean_margin rise. At step zero, accuracy is 0, because the policy equals the reference and ties count as wrong. See DPO metrics.

Next steps

Supervised fine-tuning

Warm up a model on demonstrations before preference training.

Monitoring

Read the DPO validation metrics.

Write a custom loss function

Adapt the DPO pattern to your own objective.

Deploy a checkpoint

Deploy the trained model.