> ## Documentation Index > Fetch the complete documentation index at: https://docs.together.ai/llms.txt > Use this file to discover all available pages before exploring further. # Direct preference optimization > Train on preference pairs with a custom loss function that runs in your own process. Direct preference optimization (DPO) raises the likelihood of a chosen response relative to a rejected response to the same prompt, measured against a frozen snapshot of the model from before training. It trains on existing responses without generating new ones. Use it for preference data from human ratings, reward-model scores, or comparisons between systems. On the post-training service, DPO runs as a [custom loss function](/docs/post-training/custom-loss-functions) that your own process computes. [Fine-tuning jobs](/docs/fine-tuning/preference-tuning) also offer DPO, with a fixed data format and no code to write. To choose between them, see [Post-training vs. standard fine-tuning](/docs/post-training/overview#post-training-vs-standard-fine-tuning). ## How post-training runs DPO DPO uses a trainer-only resource (`num_generator_replicas=0`). Before any updates, the cookbook scores all training and validation pairs with `forward` to snapshot the reference policy. Each training step then runs: 1. **`forward`:** The trainer returns per-target-token log-probabilities without accumulating gradients. 2. **Local loss:** Your process computes the loss in PyTorch from live and reference log-probabilities, then calls `.backward()` to get per-token log-probability gradients. 3. **`custom_forward_backward`:** Your process sends those gradients to the trainer, which repeats the forward pass and backpropagates through the model to accumulate parameter gradients. 4. **`optim_step`:** The optimizer applies the accumulated gradients. [Write a custom loss function](/docs/post-training/custom-loss-functions) covers this pattern for other objectives and the requirements for each pass. ## Run the DPO recipe The cookbook's DPO loop trains on [HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) preference pairs: ```python theme={null} from post_training_cookbook.dpo.config import DPOConfig from post_training_cookbook.dpo.train import train config = DPOConfig( model="Qwen/Qwen3.8-27B", enable_thinking=False, dataset_name="nvidia/HelpSteer3", dataset_split="train", dataset_subsample=1600, batch_size=8, learning_rate=5e-6, beta=0.1, max_length=4096, max_steps=200, precompute_ref_batch_size=512, ) train(config) ``` Run `uv run python recipes/dpo/helpsteer3.py`. `train` provisions a trainer-only resource and one session, then stops both when it returns. To use an existing resource, [attach through the SDK](/docs/post-training/model-resources#attach-to-an-existing-resource), because `DPOConfig` has no `model_resources_id` field. Settings worth knowing: * **`beta`:** The temperature on the implicit reward. Higher values keep the policy closer to the reference. * **`enable_thinking`:** Set it to `False` for Qwen preference data without reasoning, such as HelpSteer3. The catalog's Qwen models default to thinking on, making these pairs unrenderable, so `train` raises `ValueError` before provisioning. GLM-5.3 accepts these pairs with thinking on and rejects `False`, so leave the setting unset for it. * **`label_smoothing`:** Conservative-DPO smoothing. Off by default. * **`learning_rate`:** The default is `5e-6` with cosine decay, lower than for [supervised fine-tuning (SFT)](/docs/post-training/sft). * **`precompute_ref_batch_size`:** Samples per forward pass for reference precomputation and validation. Lower it if long pairs cause out-of-memory errors. * **`val_interval`:** The number of steps between validation runs on held-out pairs. The [cookbook reference](/docs/post-training/cookbook#dpoconfig) lists every `DPOConfig` field. ## Prepare preference pairs You choose your dataset schema and prepare tokenized samples in your code. The cookbook's `build_preference_pair` takes two full conversations with identical history, ending in a chosen and a rejected assistant response. It applies the model's chat template and returns two SFT-style `Sample` objects, masking only the final responses: ```python theme={null} from post_training_cookbook.codecs import get_codec from post_training_cookbook.dpo.samples import build_preference_pair codec = get_codec("zai-org/GLM-5.3") context = [{"role": "user", "content": "Explain recursion in one sentence."}] chosen = context + [ { "role": "assistant", "content": "Recursion is when a function solves a problem by calling itself on a smaller version of it.", } ] rejected = context + [ {"role": "assistant", "content": "Recursion is a programming concept."} ] pair = build_preference_pair( chosen_messages=chosen, rejected_messages=rejected, codec=codec, max_length=4096, ) if pair is not None: chosen_sample, rejected_sample = pair ``` Skip pairs that return `None`, which means truncation to `max_length` left at least one side with no trained token. Unrenderable chat templates raise `ValueError`. For Qwen data without reasoning, pass `enable_thinking=False` to `get_codec`. The cookbook interleaves samples as `[chosen_0, rejected_0, chosen_1, rejected_1, ...]`: chosen responses at even indexes, rejected responses at odd indexes. ### Use your own preference dataset Select data for the cookbook's Hugging Face `datasets` loader with `dataset_name`, `dataset_config`, and `dataset_split`. Pass `train` a `to_pair` function that returns `(chosen_messages, rejected_messages)` per row, or `None` to skip it. The default, `helpsteer3_pair` in `post_training_cookbook.dpo.train`, chooses by HelpSteer3's `overall_preference` and skips ties. This example maps `prompt`, `chosen`, and `rejected` text columns: ```python theme={null} def preference_pair(row): context = [{"role": "user", "content": row["prompt"]}] chosen = context + [{"role": "assistant", "content": row["chosen"]}] rejected = context + [{"role": "assistant", "content": row["rejected"]}] return chosen, rejected train(config, to_pair=preference_pair) ``` If your dataset has no `"validation"` split, set `val_split` to a held-out split it has, or disable validation with `val_interval=0`. Reusing the training split inflates `val/preference_accuracy`. ## Write your own DPO loop Customize data loading, preprocessing, tokenization, or the objective in your own loop. This example renders in-memory preference pairs with the cookbook's chat-template codec: ```python theme={null} from post_training_cookbook.codecs import get_codec from post_training_cookbook.custom_loss import ( compute_custom_forward_backward, compute_logprobs, ) from post_training_cookbook.dpo.loss import make_dpo_loss from post_training_cookbook.dpo.samples import build_preference_pair from post_training_cookbook.samples import extract_loss_mask from together.post_training import ( AdamParams, ComputeConfig, LoraConfig, ModelResourcesClient, ) MODEL = "Qwen/Qwen3.8-27B" PAIRS_PER_STEP = 8 REF_CHUNK_SIZE = 64 pairs = load_pairs() # a list of (chosen_messages, rejected_messages) # Load the model's chat template, with thinking off for data without reasoning codec = get_codec(MODEL, enable_thinking=False) # Render each pair into a chosen sample and a rejected sample, interleaved # so even indexes hold chosen responses and odd indexes hold rejected ones samples = [] for chosen, rejected in pairs: pair = build_preference_pair( chosen_messages=chosen, rejected_messages=rejected, codec=codec, max_length=4096, ) # Skip pairs that truncation left with no response tokens if pair is not None: samples.extend(pair) # The loss mask on each sample covers only its final response masks = [extract_loss_mask(sample) for sample in samples] # Provision a trainer-only model resource (DPO never samples) with ModelResourcesClient.create( base_model=MODEL, compute_config=ComputeConfig( gpu_type="B200-SXM", num_generator_replicas=0 ), ) as resources: # Start a training session that trains a LoRA adapter with resources.create_session( lora_config=LoraConfig(rank=16, alpha=16) ) as session: # Before any update, score every sample to snapshot the reference # policy. Keep each sample's summed logprob over its response. ref_sums = [] for start in range(0, len(samples), REF_CHUNK_SIZE): chunk = samples[start : start + REF_CHUNK_SIZE] chunk_masks = masks[start : start + REF_CHUNK_SIZE] for logprobs, mask in zip( compute_logprobs(session, chunk), chunk_masks ): ref_sums.append(sum(lp * m for lp, m in zip(logprobs, mask))) # Each step trains on the next PAIRS_PER_STEP pairs. The step size is # even, so a pair's two samples always land in the same batch. batch_size = 2 * PAIRS_PER_STEP for step, start in enumerate(range(0, len(samples), batch_size)): batch = samples[start : start + batch_size] # Bind the DPO loss to this batch's reference sums and masks loss_fn = make_dpo_loss( ref_sums[start : start + batch_size], masks[start : start + batch_size], beta=0.1, ) # Score the batch, compute the loss and its gradients in PyTorch, # and send the gradients to the trainer to accumulate result = compute_custom_forward_backward(session, batch, loss_fn) # Update the adapter's weights session.trainer.optim_step( adam_params=AdamParams(learning_rate=5e-6, weight_decay=0.1), ) # The service doesn't report a loss for a custom loss function, # so this is the only record of it print(f"[step {step}] loss={result.loss:.4f}") # Save the adapter as an inference checkpoint you can deploy checkpoint = session.create_inference_checkpoint() print(checkpoint.id) ``` `make_dpo_loss` binds the loss to a batch's reference sums and response masks. It sums masked token log-probabilities, computes the chosen-minus-rejected margin relative to the reference, and averages `-log σ(β · margin)` across pairs. Pass `label_smoothing` for conservative DPO. `compute_custom_forward_backward` runs the two trainer passes and local PyTorch autograd. Log its `result.loss` yourself, because the service doesn't report a loss for a custom loss function. `compute_logprobs` scores its whole batch in one forward pass, so the loop chunks the reference snapshot by `REF_CHUNK_SIZE`. Lower it if long pairs run out of memory. The recipe uses `precompute_ref_batch_size` and adds a cosine learning-rate schedule, validation, and periodic checkpoints. ## Validate with preference accuracy DPO validates with implicit-reward preference accuracy, rather than token-level accuracy. Every `val_interval` steps, the recipe measures the fraction of held-out pairs whose reward shift favors the chosen response. Watch `val/preference_accuracy` and `val/mean_margin` rise. At step zero, accuracy is `0`, because the policy equals the reference and ties count as wrong. See [DPO metrics](/docs/post-training/monitoring#dpo-metrics). ## Next steps Warm up a model on demonstrations before preference training. Read the DPO validation metrics. Adapt the DPO pattern to your own objective. Deploy the trained model. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.