> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Node repair

> Restore unhealthy GPU nodes through automated recommendations or manual repair actions.

Node repair restores GPU nodes that [health checks](/docs/health-checks) have flagged as unhealthy. You can repair nodes through two paths: [auto repair](#auto-node-repair), where the system generates a recommendation based on detected issues, and [manual repair](#manual-node-repair), where you trigger a repair action directly from the UI.

## Auto node repair

When [passive](/docs/health-checks#passive-health-checks) or [active](/docs/health-checks#active-health-checks) health checks detect a node-level issue, the system generates a repair recommendation and surfaces it for your review. This is a human-in-the-loop process: Together handles detection and recommends a remediation, but you decide when to proceed.

### How auto repair works

1. Health checks detect an issue on a node and create an alert with supporting evidence.
2. The system evaluates the alert and generates a repair recommendation with a suggested mode (for example, migrate to new host).
3. The recommendation appears in the **Repairs** tab of your cluster.
4. You review the recommendation and accept or dismiss it.
5. Once accepted, Together executes the repair: cordon, graceful drain, remediation action, and node rejoin.

<Note>
  Auto repair accounts for in-flight work. Training jobs need to checkpoint before a node drains, and inference workloads need their replicas rebalanced. Review recommendations before accepting to confirm your workloads are ready for the disruption.
</Note>

### Recommended repair actions

When the system generates a repair recommendation, it selects the appropriate action based on the detected issue:

| **Detected issue**                                  | **Recommended action** |
| --------------------------------------------------- | ---------------------- |
| **GPU fell off the bus**                            | Migrate to new host    |
| **GPU thermal throttling** (SmClockThermalThrottle) | Migrate to new host    |
| **Xid error**                                       | Reboot                 |
| **Drained Slurm node**                              | Migrate to new host    |

### Behavioral details

* **Auto-resolution mid-approval:** Recommendations can disappear if the underlying alert clears before you accept (5-minute default CompactTTL).
* **Cooldown window:** After a repair completes (succeeded, failed, or cancelled), no new recommendation is generated for \~30 minutes on the same node.
* **Mode escalation:** A pending recommendation can change its suggested mode in-place if a higher-severity failure is detected while it's waiting in the queue.

### The Repairs tab

To view repair recommendations and history:

1. Navigate to your cluster in the Together Cloud UI.
2. Select the **Repairs** tab.

The Repairs table shows all repair events with the following columns:

* **Node:** The affected node name.
* **State:** The current status of the repair. Values include Auto Resolved (issue resolved before action was taken), Succeeded (repair completed), and in-progress states.
* **Mode:** The remediation action (for example, Migrate to new host).
* **Trigger:** How the repair was initiated. Automated (generated by health checks) or Manual (triggered by a user).
* **Created:** When the repair recommendation was generated.

### Repair details

Select any row in the Repairs table to view the full repair details:

* **Node:** The affected node name.
* **State:** The current repair state (for example, Succeeded).
* **Mode:** The remediation action taken.
* **Created / Started:** When the recommendation was generated and when the repair execution began.
* **Requested by:** The source that initiated the repair. For auto repairs, this shows Together Health Checker.
* **Reviewed by:** Who approved the repair (your user name or Auto-Approved for auto-approved repairs).
* **Review time:** When the repair was approved.
* **Review comment:** Any notes from the approval (for example, "auto-approved: approved").
* **Repair ID:** Unique identifier for tracking and support requests.
* **Alert evidence:** Expandable section showing the underlying alerts that triggered the recommendation, including failure type and affected hardware.

## Manual node repair

When you encounter node problems or want to trigger a repair without waiting for an automated recommendation, you can start a repair directly from the Worker Nodes UI.

### How to trigger manual repair

1. Navigate to your cluster in the Together Cloud UI.
2. Go to the **Worker Nodes** section.
3. Find the problematic node.
4. Select the **⋮** (three dots) menu in the **State** column.
5. Select **Repair** from the dropdown.
6. A repair dialog appears showing:
   * Node details (name, GPU configuration).
   * Issue detected (if applicable).
   * Impact warning.
7. Choose one of the repair actions:
   * **Reboot:** For transient software issues (preserves local data).
   * **Quick reprovision:** For persistent software issues.
   * **Migrate to new host:** For hardware issues.
   * **Remove:** Permanently removes the node for RMA (return merchandise authorization).
   * **Report an issue** (optional): To notify support.

The repair process begins immediately and the node rejoins your cluster once complete.

### Available repair actions

**Reboot**

Reboots the VM in place on the same physical host.

* **When to use:** Transient software issues (GPU driver hangs, stuck processes, kernel-level errors) where a restart is likely to clear the problem.
* **What happens:** The node follows the Cordon → Drain → Reboot → Rejoin lifecycle. The VM restarts on the same physical hardware without reimaging. Local scratch and temporary data on `/scratch` and `/tmp` is preserved.

<Note>
  Reboot is the lightest repair action. Because the VM is not reimaged, it is faster than a reprovision and preserves local data. Try a reboot first for transient issues before escalating to a reprovision.
</Note>

**Quick reprovision**

Reprovisions the GPU node VM on a random underlying physical host.

* **When to use:** Persistent software-level issues (driver crashes, library corruption), VM configuration problems, or application-level issues that a reboot did not resolve.
* **What happens:** The node follows the Cordon → Drain → Reprovision lifecycle. The VM is recreated with a fresh software stack and rejoins the cluster automatically.

<Warning>
  You lose all local VM data during reprovision. Store data on PersistentVolumes or back it up before proceeding. No new jobs are scheduled on this node until remediation completes.
</Warning>

**Migrate to new host**

Provisions a new VM on a different underlying physical host.

* **When to use:** Hardware-level issues (GPU failures, PCIe problems), issues that persist after a quick reprovision, or physical component failures.
* **What happens:** The node follows the Cordon → Drain → Migrate lifecycle. A new VM is created on different physical hardware with different GPUs assigned, and rejoins the cluster automatically.

<Warning>
  You lose all local VM data during migration. Store data on PersistentVolumes or back it up before proceeding. No new jobs are scheduled on this node until remediation completes.
</Warning>

**Remove**

Permanently removes the node from the cluster. The cluster node count drops below the desired count.

* **When to use:** Faulty GPU hardware that needs to be returned to the provider for RMA. Use this when the node has a confirmed hardware defect that cannot be resolved by migration.
* **What happens:** The node follows the Cordon → Drain lifecycle, then is permanently removed from the cluster. The node is not replaced automatically.

<Warning>
  Removing a node is irreversible from the cluster's perspective. The node is taken out of service entirely and your cluster runs with fewer nodes until a replacement is provisioned. Only use this for confirmed hardware failures that require physical RMA.
</Warning>

**Report an issue**

Use this option if:

* You are unsure which repair action to use.
* You want Together support to investigate before taking action.
* The issue requires additional context or diagnosis.

## Repair lifecycle

Both auto and manual repairs follow the same lifecycle:

```text theme={null}
Cordon → Drain → Reboot/Reprovision/Migrate/Remove → Rejoin (or permanent removal)
```

**Cordon:** The node is marked as unschedulable. No new workloads are placed on the node, but existing workloads continue running.

**Drain:** Running workloads are gracefully terminated and pods are evicted from the node.

**Reboot/Reprovision/Migrate:**

* **Reboot:** The VM restarts in place on the same hardware. Local `/scratch` and `/tmp` data is preserved.
* **Quick reprovision:** The VM is recreated on a random physical node (could be the same as the original host). Local data is lost.
* **Migrate to new host:** A new VM is created on different physical hardware. Local data is lost.
* **Remove:** The node is permanently removed from the cluster for RMA. No rejoin occurs.

**Rejoin:** The node automatically rejoins the cluster, becomes schedulable, and is ready to accept new workloads.

You can monitor repair progress in the **Repairs** tab (for auto repairs) or the **Worker Nodes** section (for manual repairs). The node progresses through these states: Cordoning → Draining → Repairing/Migrating → Joining → Running.

## Choosing a repair action

Use this table to determine which repair action fits your issue. Start with the lightest action (reboot) and escalate if the issue persists.

| **Issue type**                         | **Reboot**  | **Reprovision**   | **Migrate to new host** |
| -------------------------------------- | ----------- | ----------------- | ----------------------- |
| **GPU driver hang**                    | ✓ Try first | ✓ If reboot fails |                         |
| **Stuck GPU processes**                | ✓ Try first | ✓ If reboot fails |                         |
| **GPU watchdog timeouts**              | ✓ Try first | ✓ If reboot fails |                         |
| **Stuck GPU contexts**                 | ✓ Try first | ✓ If reboot fails |                         |
| **Recoverable Xid errors**             | ✓ Try first | ✓ If reboot fails |                         |
| **Application memory leaks**           | ✓ Try first | ✓ If reboot fails |                         |
| **Software-based throttling**          | ✓ Try first | ✓ If reboot fails |                         |
| **Driver crashes/corruption**          |             | ✓ Yes             |                         |
| **CUDA/ROCm library issues**           |             | ✓ Yes             |                         |
| **Incorrect GPU mode settings**        |             | ✓ Yes             |                         |
| **GPU not attached to VM**             |             | ✓ Yes             |                         |
| **Device permissions/cgroup issues**   |             | ✓ Yes             |                         |
| **NUMA affinity problems**             |             | ✓ Yes             |                         |
| **Single-bit ECC errors (occasional)** |             | ✓ Yes             |                         |
| **Complete GPU card failure**          |             |                   | ✓ Yes                   |
| **Persistent multi-bit ECC errors**    |             |                   | ✓ Yes                   |
| **GPU falling off PCIe bus**           |             |                   | ✓ Yes                   |
| **Fan failures**                       |             |                   | ✓ Yes                   |
| **PCIe lane degradation**              |             |                   | ✓ Yes                   |
| **Power delivery (VRM) issues**        |             |                   | ✓ Yes                   |
| **Thermal/cooling problems**           |             |                   | ✓ Yes                   |
| **Persistent Xid errors**              |             |                   | ✓ Yes                   |
| **Physical connector damage**          |             |                   | ✓ Yes                   |
| **Backplane/riser issues**             |             |                   | ✓ Yes                   |

<Note>
  Escalation path: reboot → reprovision → migrate to new host. If the issue persists after reprovisioning the VM to a fresh instance on the same physical GPU, it is a hardware problem requiring migration to a new host.
</Note>

## Best practices

**Before triggering a repair:**

* Store important data on PersistentVolumes, not local storage.
* Optionally drain workloads manually for more control over migration.
* Document symptoms for troubleshooting if the repair does not resolve the problem.
* Check running jobs so you know what will be interrupted.

**Choosing the right action:**

* **Start with reboot:** It is the fastest option, preserves local data, and resolves most transient software issues.
* **Escalate to quick reprovision:** When a reboot did not fix the issue, or the problem is a corrupted driver, library, or VM configuration that requires a fresh software stack.
* **Use migrate to new host:** When reprovision did not fix the issue, you see hardware error indicators (ECC errors, Xid errors, thermal warnings), or GPU diagnostics show hardware problems.

**After a repair:**

* Verify the node shows as Running in the cluster.
* Run a GPU workload to confirm operation.
* Monitor for recurrence of the same issue.
* Check GPU metrics to confirm normal operation.

## Common diagnostic commands

Before triggering a repair, you can SSH into the node to diagnose issues:

```bash theme={null}
# Check GPU status
nvidia-smi

# Check for Xid errors in system logs
sudo dmesg | grep -i xid

# Check GPU memory errors
nvidia-smi -q | grep -i ecc

# Check GPU temperature and throttling
nvidia-smi -q | grep -E 'Temperature|Throttle'

# Check PCIe link status
nvidia-smi -q | grep -E 'Link Width|Link Speed'

# Check running processes on GPU
nvidia-smi pmon

# Detailed GPU query
nvidia-smi -q
```

[Learn how to SSH into nodes →](/docs/gpu-clusters-management#direct-ssh-access)

## When to contact support

Contact [support@together.ai](mailto:support@together.ai) if:

* Issues persist after all repair actions.
* You see repeated failures on multiple nodes.
* You need help diagnosing whether an issue is software or hardware.
* Repair actions fail to complete.
* You are unsure which repair action to use.
* The node does not rejoin after repair completes.

Alternatively, use the **Report an issue** button in the repair dialog to notify support directly.

## Next steps

<CardGroup cols={2}>
  <Card title="Health checks" icon="activity-heartbeat" href="/docs/health-checks">
    Monitor node health with active diagnostic tests and continuous passive monitoring.
  </Card>

  <Card title="Cluster management" icon="server" href="/docs/gpu-clusters-management">
    Manage, monitor, and scale your GPU clusters.
  </Card>
</CardGroup>