Preemptible compute

Run interruptible workloads on discounted preemptible GPU nodes in your cluster.

Preemptible compute is in public preview for Kubernetes and Slurm clusters. There is no minimum-lifetime guarantee for preemptible nodes during preview, so design workloads that can survive losing nodes at any time.

Together AI GPU clusters offer two compute types:

• Standard nodes (the default) are provisioned up front (synchronously) when you create or scale a cluster, and are never preempted.
• Preemptible nodes fill in over time (asynchronously). You set a target, and Together provisions toward it as spare capacity becomes available. The target is not guaranteed, and preemptible nodes can be preempted at any time.

Preemptible nodes are discounted relative to on-demand at a flat rate (not a bid), and usage is metered and billed sub-hourly (every one to two minutes).

There is no separate preemptible cluster type. You add preemptible capacity to a cluster, at create time or later. Each cluster has a standard node count and a preemptible node count, and preemptible nodes join the same cluster as the standard nodes. Every cluster requires at least one standard node (num_gpus), and you cannot convert a node between standard and preemptible in place.

On Kubernetes clusters, preemptible nodes carry the label together.ai/compute-class=preemptible, and all other nodes carry together.ai/compute-class=non-preemptible. Node names encode the compute type: gpu-dp is standard, gpu-preemptible-dp is preemptible.

On Slurm clusters, preemptible nodes join the same partitions as standard nodes and have the same slinky feature, so Slurm can’t tell them apart. Any job can land on a preemptible node. If adding preemptible capacity to a Slurm cluster returns a 400 with preemptible compute is not supported for Slurm clusters, contact your account team to enable it for your organization.

How preemption works

When Together reclaims a preemptible node, it drains the node for at most five minutes before removing it. Together then provisions replacement preemptible nodes toward your target as capacity becomes available, so you don’t need to re-request capacity.

Here’s how preemption works for each cluster type.

Kubernetes

1. T+0: The node is cordoned, a TogetherPreemptionNotified Kubernetes event (type: Warning) is emitted on the node, and pods on the node receive SIGTERM.
2. Pods that set terminationGracePeriodSeconds (capped at 300 seconds) get up to the full window to checkpoint and exit. The node is reclaimed as soon as your pods exit, so checkpoint and exit promptly instead of sleeping through the window.
3. T+5:00: The node is removed, regardless of pod status. Five minutes is a hard maximum, not a guarantee that pods finish.

Slurm

1. T+0: Slurm drains the node. No new jobs start on it, and jobs already running keep running.
2. Jobs that finish within the window exit normally.
3. T+5:00: The node is removed, and any jobs still running on it are killed. Slurm requeues the killed jobs that were submitted with --requeue.

Checkpoint frequently: Write periodic checkpoints to shared storage (/home on Slurm clusters, a shared volume on Kubernetes) so your job can resume from where it left off after a requeue.

Request preemptible capacity

Set a preemptible GPU target alongside the standard GPU count when you create or update a cluster. The clusters API uses these fields for the target and its progress:

Field	Request or response	Meaning
num_preemptible_gpus	Request (create and update)	The preemptible GPU target. Must be a multiple of 8. project_id is required when the target is greater than 0. Omit on update to keep the current value.
desired_preemptible_gpus	Response (read-only)	The current target.
allocated_preemptible_gpus	Response (read-only)	The preemptible GPUs currently live in the cluster. Rises from 0 toward the target as nodes are provisioned, and can sit below the target when capacity is tight.

Console

To add preemptible capacity when you create a cluster:

1. In the Together AI console, select Create cluster.
2. Choose a GPU type, then set the standard GPU count under Select GPU Count.
3. Under Add preemptible compute, set Preemptible GPUs in multiples of 8 (whole nodes). The form caps the count at the number of preemptible GPUs currently available for that GPU type. If none are available, try a different GPU type.
4. Finish the rest of the form, select Create Cluster, then select Confirm & Create.

To change the target on an existing cluster, open the cluster, select Scale, and choose a new value for Scale Preemptible GPUs.

The cluster’s details page shows its Preemptible Nodes and Preemptible rate.

CLI

To create a cluster with preemptible capacity, pass --num-preemptible-gpus. Preemptible capacity requires a project, so also pass the global --project flag or set TOGETHER_PROJECT_ID:

tg clusters create \
  --name my-cluster \
  --region us-central-8 \
  --gpu-type H100_SXM \
  --cluster-type KUBERNETES \
  --num-gpus 8 \
  --billing-type ON_DEMAND \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --num-preemptible-gpus 8 \
  --project <PROJECT_ID> \
  --non-interactive --json

To change the target on an existing cluster:

tg clusters update <CLUSTER_ID> --num-preemptible-gpus 16

To check progress toward the target, compare it with the allocated count:

tg clusters retrieve <CLUSTER_ID> --json \
  | jq '{status, desired_preemptible_gpus, allocated_preemptible_gpus}'

Schedule workloads on preemptible nodes

On Kubernetes clusters, target preemptible nodes explicitly for interruptible workers, and keep coordinators and serving replicas on standard nodes. Slurm clusters can’t target them.

Kubernetes

# Interruptible worker: runs on preemptible nodes
nodeSelector:
  together.ai/compute-class: preemptible

Pin coordinators and serving replicas to standard nodes:

# Coordinator: runs on standard nodes
nodeSelector:
  together.ai/compute-class: non-preemptible

Slurm

Slurm can’t target preemptible nodes or keep a job off them, so any job on a cluster with preemptible capacity can lose a node. Add --requeue to jobs that resume from a checkpoint, so Slurm resubmits them if a node is reclaimed while they run:

#!/bin/bash
#SBATCH --job-name=my-training-run
#SBATCH --output=train-%j.out
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8
#SBATCH --requeue

CKPT_DIR=/home/$USER/checkpoints/$SLURM_JOB_NAME
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)

srun torchrun \
  --nnodes=$SLURM_JOB_NUM_NODES \
  --nproc_per_node=8 \
  --rdzv_id=$SLURM_JOB_ID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:29500 \
  train.py \
    --checkpoint-dir $CKPT_DIR \
    --resume-from-latest

Key directives

Directive	Purpose
--requeue	Resubmit the job automatically when its preemptible node is removed. The restarted job sees SLURM_RESTART_COUNT incremented, so your script can detect a restart and resume from its last checkpoint.
--nodes=MIN-MAX	Let the job start, or restart after a requeue, without waiting for every node when preemptible capacity is short (for example, --nodes=2-4). Slurm starts the job as soon as MIN nodes are free, on as many nodes in the range as are available. A running job doesn’t grow when more nodes appear later, so use a range only if your workload can resume on a different number of nodes.

Detect preemption

Kubernetes

Preemption surfaces through three channels:

1. In-pod (recommended): A SIGTERM handler or preStop hook reacts automatically, with no polling.

2. Kubernetes events: Watch for TogetherPreemptionNotified events:

   kubectl get events -A --field-selector reason=TogetherPreemptionNotified --watch
   

3. Together API (no kubeconfig needed): Reading a cluster returns a node_lifecycle_events array (72-hour retention, deduplicated by node and reason). Filter for reason == "TogetherPreemptionNotified". Related reasons include TogetherNodeAdded and TogetherScaledDown.

   curl -s -H "Authorization: Bearer $TOGETHER_API_KEY" \
     "https://api.together.ai/v1/compute/clusters/<CLUSTER_ID>" \
     | jq '.node_lifecycle_events[] | select(.reason == "TogetherPreemptionNotified")'
   

In the console, the cluster’s details page shows a warning banner naming the nodes being preempted, and preemption events appear in its Event timeline.

Slurm

On Slurm clusters, TogetherPreemptionNotified events appear in the cluster’s node_lifecycle_events array in the Together API and in the console’s Event timeline, the same as on Kubernetes.

Inside a job, check SLURM_RESTART_COUNT at startup. Slurm sets it when it requeues a job, so a value above 0 means the job is restarting, for example after its node was reclaimed, and should resume from its latest checkpoint.

Handle preemption

Kubernetes

Pod-level: checkpoint on SIGTERM. This pattern is per-workload and requires no extra infrastructure. Claim the full grace window with terminationGracePeriodSeconds and checkpoint when the signal arrives:

apiVersion: v1
kind: Pod
metadata:
  name: training-worker
spec:
  terminationGracePeriodSeconds: 300 # claim the full 5-minute window
  nodeSelector:
    together.ai/compute-class: preemptible
  containers:
    - name: worker
      image: my-training-image
      command: ["/bin/sh", "-c"]
      args:
        - |
          trap 'echo "preempted: checkpointing"; ./checkpoint.sh; exit 0' TERM
          exec_training &
          wait
      lifecycle:
        preStop: # fires before SIGTERM reaches the container
          exec:
            command: ["/bin/sh", "-c", "./checkpoint.sh"]

The preStop hook runs first, then SIGTERM, and both count against the same grace period. Use one or the other as the checkpoint trigger, not both doing duplicate work. The preStop variant matters for containers whose main process can’t trap signals, or where PID 1 swallows them.

Cluster-level: An operator watching preemption events. For a coordinated, cluster-wide response (multi-node training, custom schedulers), run a controller that watches for TogetherPreemptionNotified events and reacts, for example by draining a job queue, triggering a coordinated checkpoint, or removing the node from a Ray or torchrun worker pool:

from kubernetes import client, config, watch

config.load_incluster_config()  # or load_kube_config() outside the cluster
v1 = client.CoreV1Api()
w = watch.Watch()

for event in w.stream(
    v1.list_event_for_all_namespaces,
    field_selector="reason=TogetherPreemptionNotified",
):
    node = event["object"].involved_object.name
    print(f"Node {node} preempted; drain window open (~5 min)")
    # React here, e.g.:
    #  - signal the training coordinator to checkpoint and rebalance
    #  - remove the node from your scheduler / worker pool
    #  - requeue pending work off the doomed node

Slurm

A job still running when its node is removed loses everything since its last checkpoint. Write checkpoints to shared storage under /home at a regular interval, and resume from the latest one when Slurm requeues the job. Checkpoint often enough that losing one interval of work is acceptable.

The following job script resumes from the latest checkpoint whenever SLURM_RESTART_COUNT shows that Slurm requeued it. train.py stands in for your training script, which saves a checkpoint to --checkpoint-dir every --checkpoint-interval steps and loads the latest one when passed --resume-from-latest:

#!/bin/bash
#SBATCH --job-name=preemptible-train
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8
#SBATCH --requeue

CKPT_DIR=/home/$USER/checkpoints/$SLURM_JOB_NAME
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)

if [ "${SLURM_RESTART_COUNT:-0}" -gt 0 ]; then
  echo "Requeued (restart $SLURM_RESTART_COUNT), resuming from the latest checkpoint"
  RESUME_FLAG="--resume-from-latest"
else
  RESUME_FLAG=""
fi

srun torchrun \
  --nnodes=$SLURM_JOB_NUM_NODES \
  --nproc_per_node=8 \
  --rdzv_id=$SLURM_JOB_ID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:29500 \
  train.py \
    --checkpoint-dir $CKPT_DIR \
    --checkpoint-interval 500 \
    $RESUME_FLAG

When to use preemptible

Use preemptible compute for:

• Checkpointed training, fine-tuning, evals, distillation, batch inference, and hyperparameter sweeps. Ray Train and PyTorch Lightning restart from checkpoints out of the box.
• Bursty research: raise the target for a sweep and drop it after. The cluster stays up.

Don’t use preemptible compute for:

• The only copy of a multi-day run with no checkpoints.
• User-facing serving replicas with no fallback capacity.
• Workloads with strict SLOs.

As a sizing reference, a checkpoint of a roughly 10B-parameter model completes in about 90 seconds, leaving comfortable headroom within the five-minute window. Checkpoint often, and treat preemptible nodes as ephemeral.