Node repair

• Auto repair: Health checks detect a fault and the system remediates it, either after your approval or on its own, depending on the cluster’s confirmation policy.
• Manual repair: You start a repair action directly from the Together AI console.

Auto node repair

How auto repair works

1. Health checks detect an issue on a node and create an alert with supporting evidence.
2. The system evaluates the alert and generates a repair recommendation with a suggested mode (for example, migrate to new host).
3. The recommendation appears in the Repairs tab of your cluster.
4. Under Approve before repair, you review the recommendation and approve a repair action. The system marks its suggested action as recommended, but you can override it and approve a different action instead. Under Fully automatic, the system auto-approves an in-scope recommendation with no review and runs the recommended action.
5. The system cordons the node so no new work lands on it, then applies the cluster’s wait policy.
6. Together drains the node, executes the remediation action, and rejoins the node to the cluster.

Recommended repair actions

Confirmation policy

Fully automatic

Scope which faults repair automatically

Turn off auto repair for individual nodes

Control job interruption

• Wait for idle off: The system cordons the node immediately, waits out the Grace period, then drains the node and interrupts any work still running.
• Wait for idle on: The repair holds until the node becomes idle or Maximum wait expires, whichever comes first. To remove the upper bound and wait indefinitely for the node to go idle, check Do not interrupt running jobs.

Choosing a policy

• Workloads checkpoint frequently, or are replicated inference deployments that tolerate losing a replica.
• The cluster is large enough that manual review is the bottleneck in mean time to recovery.
• The cluster has spare capacity, so a node leaving the pool doesn’t block scheduling.

• Long-running training jobs checkpoint infrequently.
• The cluster runs at full capacity, so losing a node stalls a job.
• You’re debugging a recurring fault and want the node preserved for inspection.

Override the recommended action

Behavioral details

• Auto-resolution before approval: If a recommendation’s underlying alert clears before you approve it, the recommendation moves to Auto resolved, usually within a minute of the alert clearing.
• Cooldown window: After a repair finishes (succeeded, failed, or cancelled), the system generates no new recommendation for the same node for about 30 minutes.
• Mode change while waiting: If a higher-severity failure is detected while a recommendation waits for your approval, its suggested mode can change in place.
• Repair concurrency: A cluster repairs a limited number of nodes at once: one node on clusters with 1 to 4 nodes, two on clusters with 5 to 10 nodes, and 15% of nodes (rounded up) on larger clusters. Replacement nodes that are still provisioning count toward the limit. Approved repairs beyond the limit stay Pending until a slot frees up.

Escalation for recurring faults

• Reboot to quick reprovision: After two reboots for the same fault on the same host, the next repair runs as a quick reprovision without review. This applies only to reboot faults that repair automatically, which requires a Fully automatic cluster, the fault checked under Repair actions, and Auto Repair enabled on the node. Reboot recommendations that wait for your approval don’t escalate.
• Quick reprovision to migrate: After two quick reprovisions for the same fault on the same host, the next one runs as a migration to a new host, as long as at least two spare hosts of the same type are available in the same datacenter. This applies on any cluster. It also covers quick reprovisions you approve or request manually while the fault is still firing. An escalated repair that you approved or requested still shows quick reprovision as its Mode, even though it runs as a migration.

The Repairs tab

1. Navigate to your cluster on the GPU clusters page.
2. Select the Repairs tab.

• Node: The affected node name.
• State: The repair’s current state.
• Mode: The repair action, for example Migrate to new host.
• Trigger: How the repair started. Automated means health checks generated it, and Manual means a user triggered it.
• Created: When the repair recommendation was generated.

Repair states

Repair details

• Node: The affected node name.
• State: The current repair state, for example Succeeded.
• Mode: The repair action.
• Created: When the recommendation was generated.
• Dispatched: When Together sent the approved repair to your cluster. Any wait for quarantine or idle time falls between this and Execution started.
• Execution started: When the repair interrupted the node and began running.
• Requested by: The source that initiated the repair. For auto repairs, this shows Together Health Checker.
• Reviewed by: The user name of whoever approved the repair, or Together Health Checker for repairs that ran automatically.
• Review time: When the repair was approved.
• Review comment: Any notes from the approval. Automatically approved repairs show auto-advanced: tenant confirmationPolicy auto_advance, followed by an escalation summary when the escalation ladder changed the action.
• Repair ID: The repair’s unique identifier, for tracking and support requests.
• Health Check failures: An expandable section that shows the alerts that triggered the recommendation, including failure type and affected hardware.

Linked alerts in API responses

• passive_health_check_alert_id: Alert UUID.
• alert_name: Alertmanager alert name.
• severity: PHC_SEVERITY_INFO, PHC_SEVERITY_WARNING, or PHC_SEVERITY_CRITICAL.
• started_at and resolved_at: When the alert fired and cleared (resolved_at is empty while the alert is still firing).
• target_vm: VM name from the alert labels.
• annotation: Structured alert details, containing title, summary_line, and description, plus xid (Xid events with code, mnemonic, and count) or slurm_node_unavailable (the Slurm reason) when they apply.
• annotations: Raw Alertmanager annotation key-value pairs.
• cluster_id: Cluster UUID the alert was raised against.
• instance_id: Resolved instance UUID (empty until the alert is joined to an instance).
• node_remediation_intent_id: Remediation intent UUID attached to the alert, if any.

from together import Together

client = Together()

remediation = client.clusters.remediations.retrieve(
    "<REMEDIATION_ID>",
    cluster_id="<CLUSTER_ID>",
    instance_id="<INSTANCE_ID>",
)
for alert in remediation.linked_alerts or []:
    print(alert.alert_name, alert.severity, alert.started_at)

import Together from "together-ai";

const client = new Together();

const remediation = await client.clusters.remediations.retrieve(
  "<REMEDIATION_ID>",
  { instance_id: "<INSTANCE_ID>", cluster_id: "<CLUSTER_ID>" },
);
for (const alert of remediation.linked_alerts ?? []) {
  console.log(alert.alert_name, alert.severity, alert.started_at);
}

Manual node repair

Start a manual repair

1. Navigate to your cluster on the GPU clusters page.
2. Select the Nodes tab.
3. Select the ⋮ (three dots) menu at the end of the node’s row.
4. Select Repair from the dropdown.
5. In the repair dialog, review the node details (name and GPU configuration), the detected issue if there is one, and the impact warning.
6. Select a repair action:
   • Reboot: For transient software issues (preserves local data).
   • Cycle: For issues that persist after a reboot but don’t need a fresh VM (power-cycles the physical host). Available only on supported hardware.
   • Reprovision: For persistent software issues (a quick reprovision).
   • Migrate: For hardware issues (a migration to a new host).
   • Remove: For hardware that needs RMA (permanently removes the node).

Available repair actions

Reboot

• When to use: Transient software issues (GPU driver hangs, stuck processes, kernel-level errors) where a restart is likely to clear the problem.
• What happens: The node follows the Cordon → Drain → Reboot → Rejoin lifecycle. Local scratch and temporary data on /scratch and /tmp is preserved.

Cycle

• When to use: Issues that survive a VM reboot but don’t require reprovisioning the VM or migrating to different hardware.
• What happens: The node follows the Cordon → Drain → Power cycle → Rejoin lifecycle. Together power-cycles the physical host through its baseboard management controller (BMC). Local scratch and temporary data on /scratch and /tmp is preserved, and the node keeps its host and node identity.

Quick reprovision

• When to use: Persistent software-level issues (driver crashes, library corruption), VM configuration problems, application-level issues that a reboot did not resolve, or nodes stuck in Pending or Scheduling.
• What happens: The node follows the Cordon → Drain → Reprovision lifecycle. The VM is recreated with a fresh software stack and rejoins the cluster automatically.

Migrate to new host

• When to use: Hardware-level issues (GPU failures, PCIe problems), issues that persist after a quick reprovision, or physical component failures.
• What happens: The node follows the Cordon → Drain → Migrate lifecycle. A new VM is created on different physical hardware with different GPUs assigned, and rejoins the cluster automatically.

Remove

• When to use: A confirmed hardware defect that migration can’t resolve, where the faulty GPU hardware needs to go back to the provider for RMA.
• What happens: The node follows the Cordon → Drain lifecycle, then leaves the cluster permanently.

Repair lifecycle

Cordon → Drain → Reboot/Cycle/Reprovision/Migrate/Remove → Rejoin (or permanent removal)

1. Cordon: The node is marked as unschedulable. No new workloads are placed on the node, but existing workloads continue running.
2. Drain: Running workloads are gracefully terminated and pods are evicted from the node.
3. Repair: The repair action runs. What it does to the node and its local data depends on the action:
   • Reboot: The VM restarts in place on the same hardware. Local /scratch and /tmp data is preserved.
   • Cycle: The bare-metal host is power-cycled after cordoning. Local /scratch and /tmp data is preserved, and the node keeps its identity.
   • Quick reprovision: The VM is recreated on any available host. Local data is lost.
   • Migrate to new host: A new VM is created on different physical hardware. Local data is lost.
   • Remove: The node is permanently removed from the cluster for RMA. No rejoin occurs.
4. Rejoin: The node automatically rejoins the cluster, becomes schedulable, and is ready to accept new workloads.

Choosing a repair action

Best practices

• Store important data on PersistentVolumes, not local storage.
• Check running jobs so you know what will be interrupted.
• Optionally drain workloads manually for more control over migration.
• Record the symptoms, so you have them for troubleshooting if the repair doesn’t resolve the problem.

• Start with reboot: It’s the fastest option, preserves local data, and resolves most transient software issues.
• Escalate to cycle: Try a cycle when a reboot didn’t fix the issue, the VM software stack doesn’t need to be reimaged, and cycle is available for the node. A host power cycle preserves local data and node identity.
• Escalate to quick reprovision: Reprovision when a reboot or cycle didn’t fix the issue, or when the problem is a corrupted driver, library, or VM configuration that requires a fresh software stack.
• Migrate to new host for hardware problems: Migrate when a reprovision didn’t fix the issue, you see hardware error indicators (ECC errors, Xid errors, thermal warnings), or GPU diagnostics show hardware problems.

• Verify that the node shows as Running in the cluster.
• Run a GPU workload to confirm operation.
• Monitor for recurrence of the same issue.
• Check GPU metrics to confirm normal operation.

Common diagnostic commands

# Check GPU status
nvidia-smi

# Check for Xid errors in system logs
sudo dmesg | grep -i xid

# Check GPU memory errors
nvidia-smi -q | grep -i ecc

# Check GPU temperature and throttling
nvidia-smi -q | grep -E 'Temperature|Throttle'

# Check PCIe link status
nvidia-smi -q | grep -E 'Link Width|Link Speed'

# Check running processes on GPU
nvidia-smi pmon

# Detailed GPU query
nvidia-smi -q

When to contact support

• An issue persists after you’ve tried every repair action.
• You see repeated failures on multiple nodes.
• You need help diagnosing whether an issue is software or hardware.
• Repair actions fail to complete.
• You’re unsure which repair action to use.
• A node doesn’t rejoin after its repair completes.

Next steps