Supervised fine-tuning

Train a model on example conversations with supervised fine-tuning.

Supervised fine-tuning (SFT) trains a model to imitate example responses. Use it when you have high-quality demonstrations of the behavior you want: human-written answers, a target output format, or conversations from a system that already works. The SFT quickstart walks through a complete run. This page explains SFT on the post-training service, including data preparation, custom training loops, and continuing an adapter with GRPO.

Fine-tuning jobs also run SFT, with a fixed data format and no training loop to write. To choose between them, see Post-training vs. standard fine-tuning.

How post-training runs SFT

SFT runs on a trainer-only model resource. Nothing samples from the model, so the resource runs with num_generator_replicas=0. Each step runs two operations:

1. forward_backward with LOSS_TYPE_CROSS_ENTROPY, which computes the weighted next-token cross-entropy over the batch and accumulates gradients.
2. optim_step, which applies the accumulated gradients with the step’s learning rate.

The next step trains directly on the updated weights.

Run an SFT recipe

The cookbook’s SFTConfig and train function handle dataset loading, tokenization, loss masks, the training loop, and checkpoints. This example trains Qwen3.8 27B on the No Robots dataset, with every other setting at its default:

from post_training_cookbook.sft.config import SFTConfig
from post_training_cookbook.sft.train import train

config = SFTConfig(
    model="Qwen/Qwen3.8-27B",
    enable_thinking=False,
    dataset_name="HuggingFaceH4/no_robots",
    max_steps=200,
    wandb_project="sft-no-robots",
)

train(config)

train provisions a trainer-only model resource for model, attaches one session to it, and stops both when it returns. To train on a resource that’s already running, set model_resources_id instead of model. See Manage model resources.

The config sets enable_thinking=False because No Robots conversations contain no reasoning, as Data for reasoning models explains.

The config also sets wandb_project, so train logs to Weights & Biases. Export WANDB_API_KEY before you run it, or set WANDB_MODE=offline to log locally. Without either, the run fails after it provisions the model resource, or, in an interactive terminal, waits at a W&B login prompt while the resource bills.

Settings worth adjusting first:

• num_epochs and max_steps: train makes num_epochs passes over the dataset, one by default, and reshuffles the rows before each pass. max_steps caps the total number of steps, which keeps a test run short. Its default, None, sets no cap, so a config without it trains on every full batch: about 1,187 steps for No Robots at a batch_size of 8. The learning rate decays linearly toward zero over the run.
• learning_rate: The right rate depends on the model and the task. For a low-rank adapter (LoRA) run, 1e-4 to 5e-4 is typical. Full-weight training needs a lower rate. The cookbook’s No Robots recipes use 2e-5 for full-weight training and 1e-4 for LoRA on the same model.
• train_on_what: TRAIN_ON_ALL trains on every assistant turn. TRAIN_ON_LAST trains on only the final one, which suits multi-turn datasets where only the last response matters.
• batch_size: Larger batches (16 to 32) produce smoother loss curves.
• use_lora=False: Trains the full weights instead of an adapter. Not every model supports full-weight training, so check Supported models first. See LoRA vs. full-weight training.

The cookbook reference lists every SFTConfig field.

Prepare your data

The post-training service receives tokenized Sample objects containing input tokens, next-token targets, and loss weights. Your code controls data loading, preprocessing, tokenization, and which tokens train, so any dataset schema works.

For the cookbook recipe, select your data with dataset_name, dataset_config, and dataset_split. Its default mapping reads a messages column, a list of {"role": ..., "content": ...} dicts. For other schemas, pass to_messages to map each row to its messages. Put any instruction you want the model to learn, such as an answer format, in that mapping:

def qa_messages(row):
    return [
        {"role": "user", "content": row["question"]},
        {"role": "assistant", "content": row["answer"]},
    ]


train(config, to_messages=qa_messages)

The recipe loads data with Hugging Face datasets, so local files work too. For example, save one row per line in ./my_data/train.jsonl and set dataset_name="./my_data" with dataset_split="train". Relative paths resolve from your working directory. train checks the first row with to_messages before provisioning a model resource or session, so schema errors fail before billing.

Data for reasoning models

When the cookbook renders your conversations, match the model’s chat template and thinking settings to your data:

• Qwen data without reasoning: Set enable_thinking=False, as the No Robots example does. Thinking is on by default for the catalog’s Qwen models. Leaving it on makes prompt and full-conversation rendering disagree, so the recipe skips the rows and fails after provisioning the resource. The GLM-5.3 codec accepts these rows with thinking on and rejects enable_thinking=False.
• Data with reasoning: Put assistant reasoning in reasoning_content alongside content. Qwen3.8 27B’s template reads reasoning only from that field, so the recipe skips rows with <think> tags inside content. preserve_thinking controls whether earlier turns retain reasoning. See Thinking settings.
• Consecutive GLM assistant turns: Merge or split them before training. GLM closes each turn with the next message’s role marker, so consecutive assistant turns have no stop token and the recipe skips those rows with a warning.

Loss masks

Choose which tokens contribute to cross-entropy with per-target float32 weights: 0 excludes a token, 1 trains it fully, and fractional values reduce its contribution.

The cookbook’s render_conversation uses 0 for prompt content and role headers, including the assistant header, and 1 for selected assistant content and end-of-turn tokens. Training on end-of-turn tokens teaches the model when to stop. train_on_what selects all assistant turns or only the last one. For custom weighting, supply your own mask to build_cross_entropy_sample. Data and samples explains token alignment.

Write your own SFT loop

To customize data loading, preprocessing, tokenization, batching, or loss weights, write your own training loop and submit samples to the post-training service. This example renders an in-memory list of conversations with the cookbook’s chat-template codec. Replace that preparation with your own sample construction as needed:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.samples import (
    build_cross_entropy_sample,
    is_trainable,
)
from post_training_cookbook.sft.train import TRAIN_ON_ALL, render_conversation
from together.post_training import (
    AdamParams,
    ComputeConfig,
    LoraConfig,
    LossConfig,
    ModelResourcesClient,
)

MODEL = "Qwen/Qwen3.8-27B"
BATCH_SIZE = 8
conversations = load_conversations()  # a list of message lists

# Load the model's chat template, with thinking off for data without reasoning
codec = get_codec(MODEL, enable_thinking=False)

# Provision a trainer-only model resource (SFT never samples)
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=0
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Each step trains on the next BATCH_SIZE conversations
        for step, start in enumerate(range(0, len(conversations), BATCH_SIZE)):

            # Render each conversation into tokens and a loss mask
            # that trains only the assistant turns
            samples = []
            for messages in conversations[start : start + BATCH_SIZE]:
                for seq in render_conversation(
                    messages,
                    codec=codec,
                    train_on_what=TRAIN_ON_ALL,
                    max_length=4096,
                ):
                    # Skip sequences that truncation left with no trained token
                    if not is_trainable(seq.loss_mask):
                        continue
                    samples.append(
                        build_cross_entropy_sample(
                            tokens=seq.tokens, loss_mask=seq.loss_mask
                        )
                    )

            # Compute the cross-entropy loss and accumulate gradients
            result = session.trainer.forward_backward(
                samples=samples,
                loss=LossConfig(type="LOSS_TYPE_CROSS_ENTROPY"),
            )

            # Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=1e-4, weight_decay=0.1),
            )

            # The loss is summed over the trained tokens, not averaged
            print(f"[step {step}] loss_sum={result.loss:.1f}")

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

render_conversation returns a list of sequences, not a single one, because it splits a conversation wherever the chat template strips earlier reasoning.

build_cross_entropy_sample shifts the tokens into input and target positions for you. For cross-entropy, result.loss is the sum over all trained positions, not a mean. Divide it by the batch’s total weight to get the per-token mean the recipes log.

To train with a variant of cross-entropy, such as one with an added regularization term, write a custom loss function. Each step then uses forward and custom_forward_backward instead of forward_backward.

Chain SFT and GRPO

To continue an SFT adapter with GRPO, hand it over with a training checkpoint. Combine methods explains when this helps. The handoff takes three settings:

1. Set training_checkpoint_interval so the SFT run writes a training checkpoint at its last step. The cookbook writes one only when the interval fires, and train returns the latest checkpoint ID (or None).
2. Start the GRPO run with resume_from_checkpoint_id set to that ID and load_optimizer=False. The new session loads the adapter weights with a fresh optimizer and its step counter at zero.
3. Use the same LoRA rank and alpha in both phases. A checkpoint with a different rank fails to load, and a different alpha silently changes how strongly the adapter applies.

Because SFT is trainer-only and GRPO needs a generator, the two phases run in separate sessions, typically on a new model resource for the same base model:

from dataclasses import replace

from post_training_cookbook.datasets.registry import DATASETS
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn
from post_training_cookbook.sft.config import SFTConfig
from post_training_cookbook.sft.train import train as sft_train

MODEL = "Qwen/Qwen3.8-27B"


def gsm8k_messages(row):
    # Pose each question the way the GRPO phase does, so SFT teaches
    # the "#### <answer>" format that the GRPO reward reads.
    return [
        {"role": "user", "content": DATASETS["gsm8k"].format_question(row)},
        {"role": "assistant", "content": row["answer"]},
    ]


sft_config = SFTConfig(
    model=MODEL,
    dataset_name="openai/gsm8k",
    dataset_config="main",
    max_steps=20,
    enable_thinking=False,
    lora_rank=16,
    lora_alpha=16,
    training_checkpoint_interval=20,
)
grpo_config = GRPOConfig(
    model=MODEL,
    dataset="gsm8k",
    enable_thinking=False,
    num_steps=20,
    max_model_length=1536,
    max_prompt_length=768,
    max_sample_tokens=768,
    lora_rank=16,
    lora_alpha=16,
)

checkpoint_id = sft_train(sft_config, to_messages=gsm8k_messages)
train_single_turn(
    replace(
        grpo_config,
        resume_from_checkpoint_id=checkpoint_id,
        load_optimizer=False,
    )
)

GSM8K rows have question and answer columns, not messages, so the SFT phase maps each row with to_messages. The mapping poses each question in the GRPO prompt format, so SFT teaches the answer format that GRPO then scores. The cookbook’s recipes/sft_then_grpo.py is the complete worked example.

Unlike SFTConfig, GRPOConfig logs to Weights & Biases by default, so export WANDB_API_KEY or WANDB_MODE=offline, or pass wandb_project=None to GRPOConfig, before you run this example.

Next steps

Direct preference optimization

Train on preference pairs with a custom loss function.

Reinforcement learning with GRPO

Continue the adapter with reinforcement learning.

Cookbook reference

Every SFTConfig field and its default.

Deploy a checkpoint

Deploy the trained model on dedicated model inference.