Health checks

Health checks detect hardware and software issues on your GPU nodes before they impact running workloads. Together supports two types:

• Active checks run diagnostic tests on demand.
• Passive checks monitor nodes continuously in the background.

Passive checks can trigger node repair recommendations automatically. Use active checks to confirm a fault before a manual repair and to validate the node afterward.

​

Active health checks

Active health checks let you run targeted diagnostic tests on your GPU nodes to validate GPUs, InfiniBand networking, storage, and other components. These tests require full GPU utilization and run on demand.

​

Run a health check

1. Navigate to your cluster on the GPU clusters page.
2. Select the Health checks tab.
3. Select Run Health Check.
4. In the Run Health Check form, select a test:
   • DCGM Diag: NVIDIA GPU diagnostics.
   • GPU Burn: GPU stress test.
   • TorchTitan Training: Training throughput benchmark on B200 nodes.
   • Single-Node NCCL: Single-node GPU communication test.
   • Single-Node NCCL InfiniBand Fallback: A variant of the Single-Node NCCL test.
   • Multi-Node NCCL: GPU communication test across two or more nodes.
   • NVBandwidth: CPU to GPU Bandwidth: PCIe bandwidth test.
   • NVBandwidth: GPU to CPU Bandwidth: PCIe bandwidth test.
   • NVBandwidth: GPU-CPU Latency: PCIe latency test.
   • InfiniBand Write Bandwidth: InfiniBand network performance test.
   • Pairwise InfiniBand Write Bandwidth: Two-node InfiniBand network performance test.
   • Storage Perf: Storage throughput and data-integrity test.
5. Optionally, set the parameters that appear under the selected test, such as diagnostic level or duration.
6. Select Next: Select Nodes.
7. Select the nodes to test.
8. Select Run Health Check to start the test.

​

Available tests

Each health check validates different aspects of your GPU infrastructure:

​

GPU diagnostics

DCGM Diag

• Runs NVIDIA Data Center GPU Manager (DCGM) diagnostics.
• Validates GPU compute capability, memory integrity, and thermal performance.
• Configurable: Diagnostic level, from 1 to 4 (4 is the most comprehensive).
• Use for: Comprehensive GPU health validation.
• Learn more: NVIDIA DCGM documentation.

GPU Burn

• Stress-tests GPUs with intensive compute workloads.
• Validates stability under sustained high utilization.
• Configurable: Test duration.
• Use for: Identifying thermal issues, power problems, or instability.
• Learn more: GPU Burn on GitHub.

​

Network performance

Single-Node NCCL

• Tests the NVIDIA Collective Communications Library (NCCL) on a single node.
• Validates GPU-to-GPU communication within the node.
• Use for: Multi-GPU training readiness.
• Learn more: NVIDIA NCCL documentation.

Multi-Node NCCL

• Tests NCCL across all GPUs on the selected nodes.
• Validates GPU-to-GPU communication between nodes.
• Requires: A cluster with at least two GPU nodes. Select two or more nodes.
• Configurable: NCCL algorithm (optional).
• Use for: Distributed training readiness.

InfiniBand Write Bandwidth

• Measures InfiniBand network write throughput.
• Validates high-speed interconnect performance.
• Use for: Network readiness for distributed training and multi-node workloads.

Pairwise InfiniBand Write Bandwidth

• Measures InfiniBand write throughput between two GPU nodes, across their InfiniBand rails.
• Validates node-to-node interconnect performance and rail-level connectivity.
• Requires: A cluster with at least two GPU nodes. Select exactly two nodes. The first node you select is the server, and the second is the client.
• Configurable: RDMA memory (cpu or gpu, default cpu) and direction (both, client-to-server, or server-to-client, default both).
• Use for: Investigating slow or degraded links between specific node pairs before distributed training.

​

PCIe performance

NVBandwidth Tests

• CPU to GPU Bandwidth: Host-to-device transfer rates.
• GPU to CPU Bandwidth: Device-to-host transfer rates.
• GPU-CPU Latency: Data transfer latency.
• Use for: Identifying PCIe bottlenecks or degraded lanes.
• Learn more: NVIDIA nvbandwidth documentation.

​

Storage

Storage Perf

• Runs storage benchmarks against the shared and local storage tiers available to the cluster.
• Measures sequential read and write throughput.
• Validates data integrity with a checksummed write and read-back.
• Use for: Detecting slow or degraded storage before it bottlenecks data loading or checkpointing.

​

Training performance

TorchTitan Training

• Runs a short TorchTitan training benchmark on a single node or across multiple nodes.
• Measures the steady-state median model FLOPs utilization (MFU), skipping the warmup step.
• Validates that a node sustains the expected training throughput for its GPU type. It catches badly degraded nodes that still pass point-in-time diagnostics.
• Requires: NVIDIA B200 (Blackwell) GPU nodes with NVIDIA driver 580 or later. The benchmark’s training stack (CUDA and PyTorch with FlashAttention-4) is built for the B200’s compute capability, sm_100, so it doesn’t run on other GPU types.
• Runtime: About 15 minutes in total, roughly 9 minutes to initialize and 6 minutes to run.
• Use for: End-to-end training-readiness validation.

​

View test results

The Health checks tab lists each run in a table with Check, Status, Last run, and Nodes tested columns. After a run finishes, select View details from its ⋮ menu to see the run’s status and parameters, plus each node’s status and output summary. From a node’s ⋮ menu in the details, select Download output to save its output, or select Repair to repair the node.

​

Pass/fail thresholds

Performance-based health checks compare the measured result against a threshold. A bandwidth test passes when the measured value is greater than or equal to the threshold. Lower latency is better, so a latency test passes when the measured value is less than or equal to the threshold. The following thresholds are the defaults for on-demand health checks and acceptance tests. They’re tuned for current GPU node types and may be adjusted over time as hardware and reference baselines change. The TorchTitan pass threshold and reference MFU depend on the model and node topology. The 30% default is a loose floor for Llama 3 8B on a single node, where the reference MFU is about 36.5%. Larger models and multi-node runs use different reference values, which are set per run. For example, Llama 3 70B reaches about 38% MFU on a four-node cluster.

​

Acceptance testing

Acceptance tests validate each node before it joins your cluster. They add their run time to provisioning, so they’re off by default. Turn them on for multi-GPU training or long-running clusters, where catching a bad node before the job starts saves more time than the tests cost. To turn them on when you create a cluster, open Advanced configuration in the create form and select Run all, or turn on individual tests:

• DCGM Diag: Runs GPU diagnostics at the Diagnostic Level you choose (Level 2 by default).
• GPU Burn: Stress-tests the GPUs for the Test duration you choose (5 minutes by default).
• Single-Node NCCL: Validates GPU-to-GPU communication within each node.
• Multi-Node NCCL: Validates GPU-to-GPU communication across nodes.

The form shows the estimated total test time. To set the same options through the API, pass acceptance_tests_params to cluster create.

​

Cluster status during testing

When acceptance tests are on, the nodes run them before the cluster becomes available. The cluster status shows their progress:

• Running Tests: Acceptance tests are in progress.
• Tests Failed: One or more acceptance tests didn’t pass.
• Ready: Tests passed, and the cluster is ready for workloads.

​

Investigate failed acceptance tests

When the cluster status shows Tests Failed, find out which test failed and why:

1. Navigate to your cluster on the GPU clusters page.
2. Select the Health checks tab.
3. Find the acceptance test runs for the affected nodes.
4. Select View details to see:
   • Which specific test failed (DCGM Diag, GPU Burn, or NCCL).
   • Detailed error messages and logs.
   • Performance metrics from the tests.

​

When to run active health checks

Run active checks proactively, before a problem shows up:

• Before you deploy critical workloads.
• After you scale the cluster.
• After a maintenance window.
• On a regular schedule, such as weekly or monthly.

Run them reactively, in response to a problem:

• When jobs fail for no clear reason.
• When performance degrades.
• Before you repair a node, to confirm the fault.
• After a repair, to validate the fix.

To investigate a specific symptom, run the tests that target it:

• Training instability: Run GPU Burn and DCGM Diag.
• Slow data loading: Run Storage Perf and the NVBandwidth tests.
• Multi-GPU failures: Run Single-Node NCCL.
• Distributed training issues: Run the InfiniBand tests. To isolate a slow link between two specific nodes, run Pairwise InfiniBand Write Bandwidth.
• Low training throughput: Run TorchTitan Training.

​

Best practices

1. Schedule workload-free windows: Health checks require full GPU utilization.
2. Start with DCGM Diag: It provides a comprehensive overview of GPU health.
3. Run baseline tests: Test new nodes immediately to establish a performance baseline.
4. Document results: Keep records of passed tests for comparison.
5. Test after repair: Always validate node health after repair actions.
6. Use appropriate test levels: Higher DCGM diagnostic levels take longer but are more thorough.

​

Passive health checks

Passive health checks run continuously in the background on every GPU node in your cluster, observing real workloads, system logs, and GPU metrics. Unlike active checks, they need no action from you and don’t affect running jobs.

​

How passive checks work

Passive checks monitor node telemetry and system logs in real time. There is no synthetic load: the system observes your production traffic and flags degradation as it happens. When an issue is detected, the system creates an internal alert with supporting evidence. If the alert meets repair criteria, a repair recommendation is generated automatically.

​

Detected failure modes

Passive checks continuously watch node telemetry, GPU metrics, and system logs, and raise a signal when a node shows one of the conditions below. Each signal that meets repair criteria generates a repair recommendation with a suggested action.

​

GPU and accelerator

​

InfiniBand networking

​

Node OS, kernel, and resources

​

Host

​

Scheduler

Recommended repair actions lists the repair action each signal maps to.

​

Active vs. passive checks

• Active checks: Run on demand, using synthetic workloads that require full GPU utilization to validate specific hardware capabilities. They’re best for targeted diagnostics and pre-deployment validation, and they help you confirm a fault and validate a node after repair.
• Passive checks: Run continuously with zero overhead, observing real workloads and system logs to catch degradation as it happens. They’re best for ongoing monitoring and early detection, and they feed node repair recommendations.

​

Next steps