> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Health checks

> Monitor GPU node health with active diagnostic tests and continuous passive monitoring.

Health checks detect hardware and software issues on your GPU nodes before they impact running workloads. Together supports two types of health checks: [active checks](#active-health-checks) that run on-demand diagnostic tests, and [passive checks](#passive-health-checks) that monitor nodes continuously in the background. Issues detected by either type can trigger [node repair](/docs/node-repair) recommendations.

## Active health checks

Active health checks let you run targeted diagnostic tests on your GPU nodes to validate GPUs, InfiniBand networking, storage, and other components. These tests require full GPU utilization and run on demand.

### How to run health checks

#### Quick steps

1. Navigate to your cluster in the [Together web interface](https://api.together.ai/clusters).
2. Go to the **Cluster Details** tab and select the **Health Checks** sub-tab.
3. Select the **Run a health check** button (top right).
4. In the **Run Health Checks** dialog, select one or more tests to run:
   * **DCGM Diag:** NVIDIA GPU diagnostics.
   * **GPU Burn:** GPU stress test.
   * **Single-Node NCCL:** Single-node GPU communication test.
   * **NVBandwidth: CPU to GPU Bandwidth:** PCIe bandwidth test.
   * **NVBandwidth: GPU to CPU Bandwidth:** PCIe bandwidth test.
   * **NVBandwidth: GPU-CPU Latency:** PCIe latency test.
   * **InfiniBand Write Bandwidth:** InfiniBand network performance test.
   * **Storage Performance (Preview):** Storage throughput and data-integrity test.
5. Select **Next: Select Nodes**.
6. Choose which nodes to test.
7. (Optional) Configure test parameters like duration or diagnostic level.
8. Select **Run** to start the health checks.

<Note>
  **Active tests:** These health checks require full GPU utilization from the node and impact any running workloads during the test.
</Note>

### Available tests

Each health check validates different aspects of your GPU infrastructure:

#### GPU diagnostics

**DCGM Diag**

* Runs NVIDIA Data Center GPU Manager diagnostics.
* Validates GPU compute capability, memory integrity, and thermal performance.
* **Configurable:** Diagnostic level (1-3, where 3 is most comprehensive).
* **Use for:** Comprehensive GPU health validation.
* **Learn more:** [NVIDIA DCGM documentation](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html).

**GPU Burn**

* Stress tests GPUs with intensive compute workloads.
* Validates stability under sustained high utilization.
* **Configurable:** Test duration.
* **Use for:** Identifying thermal issues, power problems, or instability.
* **Learn more:** [GPU Burn on GitHub](https://github.com/wilicc/gpu-burn).

#### Network performance

**Single-Node NCCL**

* Tests NVIDIA Collective Communications Library on a single node.
* Validates GPU-to-GPU communication within the node.
* **Use for:** Multi-GPU training readiness.
* **Learn more:** [NVIDIA NCCL documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html).

**InfiniBand Write Bandwidth**

* Measures InfiniBand network write throughput.
* Validates high-speed interconnect performance.
* **Use for:** Distributed training and multi-node workloads.

#### PCIe performance

**NVBandwidth Tests**

* **CPU to GPU Bandwidth:** Host-to-device transfer rates.
* **GPU to CPU Bandwidth:** Device-to-host transfer rates.
* **GPU-CPU Latency:** Data transfer latency.
* **Use for:** Identifying PCIe bottlenecks or degraded lanes.
* **Learn more:** [NVIDIA nvbandwidth documentation](https://github.com/NVIDIA/nvbandwidth).

#### Storage

**Storage Performance**

* Runs storage benchmarks against shared and local storage tiers available to the cluster.
* Validates data integrity with a checksummed write and read-back, and measures sequential read and write throughput.
* **Use for:** Detecting slow or degraded storage before it bottlenecks data loading or checkpointing.

<Note>
  **Preview:** The Storage Performance check is in preview. Reach out to [support](mailto:support@together.ai) for access.
</Note>

#### Training performance

**TorchTitan Training**

* Runs a short [TorchTitan](https://github.com/pytorch/torchtitan) training benchmark on a single node or across multiple nodes.
* Measures the steady-state median model FLOPs utilization (MFU), skipping the warmup step.
* Validates that a node sustains the expected training throughput for its GPU type, catching grossly degraded nodes that still pass point-in-time diagnostics.
* **Requires:** NVIDIA B200 (Blackwell) GPU nodes. The benchmark runs a CUDA and PyTorch training stack (PyTorch with FlashAttention 4) built for the B200 architecture (compute capability `sm_100`), so it does not run on other GPU types.
* **Runtime:** About 15 minutes total, roughly 9 minutes to initialize and 6 minutes to run.
* **Use for:** End-to-end training-readiness validation.

<Note>
  **Preview:** The TorchTitan training check is in preview. It currently runs on demand on B200 nodes only, is not part of automatic acceptance testing, and its behavior and thresholds may change.
</Note>

### Understanding test results

Health check results are displayed in the Health Checks table:

* **Status:** Passed (green) or Failed (red) indicator.
* **Last Run:** Timestamp of test execution.
* **Node Tested:** Which nodes were included in the test.
* **Details:** Select **View details** to see:
  * Full test output.
  * Detailed metrics and measurements.
  * Workflow CR (Custom Resource) with complete results.
  * Pass/fail criteria details.

### Pass/fail thresholds

Performance-based health checks compare the measured result against a reference value. For bandwidth tests, a test **passes** when the measured value is greater than or equal to the threshold. For latency tests, lower is better, so a test passes when the measured value is less than or equal to the threshold. The following thresholds are the defaults applied during health checks and automatic acceptance testing.

| Test                               | Metric                           | Pass threshold | Test configuration                                                             |
| ---------------------------------- | -------------------------------- | -------------- | ------------------------------------------------------------------------------ |
| Single-Node NCCL                   | Average bus bandwidth            | ≥ 300 GB/s     | `all_reduce_perf` across 8 GPUs, 32 GiB message size.                          |
| Multi-Node NCCL                    | Average bus bandwidth            | ≥ 330 GB/s     | `all_reduce_perf` across all GPUs on the selected nodes.                       |
| InfiniBand Write Bandwidth         | Reported write bandwidth         | ≥ 320 Gb/s     | `ib_write_bw` per device, 8 MiB message size, 2-second duration.               |
| NVBandwidth: CPU to GPU Bandwidth  | Per-GPU host-to-device bandwidth | ≥ 30 GB/s      | `host_to_device_memcpy_ce`, averaged across 8 GPUs.                            |
| NVBandwidth: GPU to CPU Bandwidth  | Per-GPU device-to-host bandwidth | ≥ 30 GB/s      | `device_to_host_memcpy_ce`, averaged across 8 GPUs.                            |
| NVBandwidth: GPU-CPU Latency       | Per-GPU host-device latency      | ≤ 2000 ns      | `host_device_latency_sm`, averaged across 8 GPUs.                              |
| Storage (shared): Sequential Read  | Bandwidth                        | ≥ 10 GiB/s     | `fio` sequential read, 1 MiB blocks across 64 jobs at iodepth 32.              |
| Storage (shared): Sequential Write | Bandwidth                        | ≥ 5 GiB/s      | `fio` sequential write, 1 MiB blocks across 64 jobs at iodepth 32.             |
| Storage (local): Sequential Read   | Bandwidth                        | ≥ 2 GiB/s      | `fio` sequential read, 1 MiB blocks across 16 jobs at iodepth 16.              |
| Storage (local): Sequential Write  | Bandwidth                        | ≥ 1 GiB/s      | `fio` sequential write, 1 MiB blocks across 16 jobs at iodepth 16.             |
| TorchTitan Training                | Steady-state median MFU          | ≥ 30%          | Llama 3 8B on a single B200 node, 40 training steps (the single-node default). |

<Note>
  **Units differ by test:** NCCL and NVBandwidth bandwidth thresholds are reported in gigabytes per second (GB/s), while InfiniBand write bandwidth is reported in gigabits per second (Gb/s). The two are not directly comparable (320 Gb/s is 40 GB/s). Latency is reported in nanoseconds (ns), and TorchTitan MFU is reported as a percentage.
</Note>

The TorchTitan pass threshold and reference MFU depend on the model and node topology. The 30% default is a loose floor for the Llama 3 8B single-node case (reference MFU is about 36.5%). Larger models and multi-node runs use different reference values, which are set per run. For example, Llama 3 70B reaches about 38% MFU on a 4-node cluster.

These thresholds are tuned for current GPU node types and may be adjusted over time as hardware and reference baselines change.

### Automatic acceptance testing

When you provision a new GPU cluster, Together automatically runs acceptance tests on each node before making it available for your workloads. This ensures that all nodes meet quality standards before joining your cluster.

#### During cluster provisioning

The cluster provisioning process includes an automatic testing phase:

**Phase: Running Tests**

During this phase, each node undergoes single-node acceptance tests:

* **DCGM Diag Level 2:** Comprehensive GPU diagnostics.
* **5-minute GPU Burn:** Sustained GPU stress test.
* **Single-Node NCCL:** GPU-to-GPU communication validation.
* **Multi-Node NCCL:** GPU-to-GPU communication validation across node GPUs.
* **Storage Performance:** Storage performance validation for storage volumes attached to the cluster.

You'll see the cluster status as:

* **Running Tests:** Acceptance tests are in progress.
* **Tests Failed:** One or more acceptance tests did not pass.
* **Running:** Tests passed and the cluster is ready.

#### Viewing acceptance test results

If acceptance tests fail during provisioning:

1. Navigate to your cluster in the Together Cloud UI.
2. Go to the **Cluster Details** tab.
3. Select the **Health Checks** sub-tab.
4. Find the acceptance test runs for the affected nodes.
5. Select **View details** to see:
   * Which specific test failed (DCGM Diag, GPU Burn, NCCL, or Storage Performance).
   * Detailed error messages and logs.
   * Performance metrics from the tests.

<Note>
  **Automatic remediation:** If acceptance tests fail, Together's infrastructure team is automatically notified and investigates. Nodes that fail acceptance tests are not added to your cluster until the issue is resolved.
</Note>

#### Why acceptance testing matters

Automatic acceptance testing provides several benefits:

* **Quality assurance:** Every node is validated before you can use it.
* **Early detection:** Hardware or configuration issues are caught immediately.
* **Reduced downtime:** Problems are fixed before they impact your workloads.
* **Consistent performance:** All nodes meet the same performance standards.

<Tip>
  **Provisioning time:** Acceptance tests typically add 5-10 minutes to cluster provisioning time, but this ensures you receive fully validated, production-ready nodes.
</Tip>

### When to run active health checks

**Proactive testing:**

* Before deploying critical workloads.
* After cluster scaling events.
* On a regular schedule (weekly or monthly).
* After maintenance windows.

**Reactive testing:**

* When experiencing unexplained job failures.
* Before triggering node repair actions.
* When investigating performance degradation.
* After node repairs to validate fixes.

**Specific issue investigation:**

* **Training instability:** Run GPU Burn and DCGM Diag.
* **Slow data loading:** Run Storage Performance and NVBandwidth tests.
* **Multi-GPU failures:** Run Single-Node NCCL.
* **Distributed training issues:** Run InfiniBand tests.
* **Low training throughput:** Run TorchTitan Training.

### Best practices

1. **Schedule workload-free windows:** Health checks require full GPU utilization.
2. **Start with DCGM Diag:** It provides a comprehensive overview of GPU health.
3. **Run baseline tests:** Test new nodes immediately to establish a performance baseline.
4. **Document results:** Keep records of passed tests for comparison.
5. **Test after repair:** Always validate node health after repair actions.
6. **Use appropriate test levels:** Higher DCGM diagnostic levels take longer but are more thorough.

<Warning>
  **Workload impact:** Health checks fully utilize the GPU and can interfere with running workloads. Run tests during maintenance windows or on idle nodes.
</Warning>

## Passive health checks

Passive health checks run continuously on every node in your cluster, observing real workloads, system logs, and GPU metrics in the background. Unlike active checks, passive checks require no manual action and have zero impact on running jobs.

### How passive checks work

Passive checks monitor node telemetry and system logs in real time. There is no synthetic load: the system observes your production traffic and flags degradation as it happens. When an issue is detected, the system creates an internal alert with supporting evidence. If the alert meets repair criteria, a [repair recommendation](/docs/node-repair#auto-node-repair) is generated automatically.

### Detected failure modes

Passive checks continuously watch node telemetry, GPU metrics, and system logs, and raise a signal when a node shows one of the conditions below. Each signal that meets repair criteria generates a [repair recommendation](/docs/node-repair#auto-node-repair) with a suggested action.

#### GPU and accelerator

| Condition               | Signal                      | Typical cause                                                                                     |
| ----------------------- | --------------------------- | ------------------------------------------------------------------------------------------------- |
| GPU fell off the bus    | `DmesgGpuFallenOffBus`      | GPU becomes unreachable on the PCIe bus, typically a hardware or connector failure.               |
| GPU thermal throttling  | `GpuSmClockThermalThrottle` | Sustained high temperature forces SM clocks down, degrading throughput with no application error. |
| GPU Xid error           | `DmesgXidError`             | NVIDIA driver or hardware Xid fault reported in the kernel log.                                   |
| Uncorrectable ECC error | `GpuEccDoubleBitError`      | Double-bit GPU memory error.                                                                      |
| GPU row-remap failure   | `GpuRowRemapFailure`        | GPU memory row-remap failed; the GPU likely needs replacement (RMA).                              |
| High PCIe replay rate   | `GpuPcieReplayRateHigh`     | Degraded PCIe link integrity.                                                                     |

#### InfiniBand networking

| Condition              | Signal                  | Typical cause                                                                        |
| ---------------------- | ----------------------- | ------------------------------------------------------------------------------------ |
| Rails down or degraded | `IBRailsDownOrDegraded` | Fewer than 8 active 400G InfiniBand rails (a rail is down or negotiated below 400G). |
| Link flapping          | `IBLinkFlapping`        | Repeated InfiniBand link-down events on a compute rail.                              |

#### Node OS, kernel, and resources

| Condition                     | Signal                           | Typical cause                                |
| ----------------------------- | -------------------------------- | -------------------------------------------- |
| Fatal platform hardware error | `NpdCperHardwareErrorFatal`      | Fatal CPER-reported platform hardware error. |
| Read-only filesystem          | `NpdReadonlyFilesystem`          | Root or data filesystem remounted read-only. |
| XFS shutdown                  | `NpdXfsShutdown`                 | XFS filesystem shut down after an error.     |
| Kernel deadlock               | `NpdKernelDeadlock`              | A kernel task is stuck.                      |
| Frequent kubelet restarts     | `NpdFrequentKubeletRestart`      | kubelet is restarting repeatedly.            |
| Frequent containerd restarts  | `NpdFrequentContainerdRestart`   | containerd is restarting repeatedly.         |
| Frequent netdev unregister    | `NpdFrequentUnregisterNetDevice` | Repeated network device unregister events.   |
| Memory pressure               | `KubeNodeMemoryPressure`         | Node under sustained memory pressure.        |
| Disk pressure                 | `KubeNodeDiskPressure`           | Node under sustained disk pressure.          |
| PID pressure                  | `KubeNodePIDPressure`            | Node is exhausting available process IDs.    |

#### Scheduler

| Condition              | Signal                 | Typical cause                       |
| ---------------------- | ---------------------- | ----------------------------------- |
| Slurm node unavailable | `SlurmNodeUnavailable` | Slurm marks the node DOWN or DRAIN. |

See [Recommended repair actions](/docs/node-repair#recommended-repair-actions) for the repair action each signal maps to.

<Note>
  Detection coverage is expanding, and automated recommendations are enabled per cluster. Not every signal triggers an automated recommendation today; some raise an internal alert that Together's team reviews. Future releases will cover additional hardware and software signals.
</Note>

### Active vs. passive checks

* **Active checks:** Run on demand, use synthetic workloads that require full GPU utilization, and validate specific hardware capabilities. Best for targeted diagnostics and pre-deployment validation.
* **Passive checks:** Run continuously with zero overhead, observe real workloads and system logs, and catch degradation as it happens. Best for ongoing monitoring and early detection.

Both types feed into [node repair](/docs/node-repair) recommendations.

## Next steps

<CardGroup cols={2}>
  <Card title="Node repair" icon="wrench" href="/docs/node-repair">
    Restore unhealthy nodes through automated recommendations or manual repair actions.
  </Card>

  <Card title="Cluster management" icon="server" href="/docs/gpu-clusters-management">
    Manage, monitor, and scale your GPU clusters.
  </Card>
</CardGroup>