Skip to main content
Together runs Slurm on top of Kubernetes through Slinky, an integration layer between the two. You work with standard Slurm commands such as sbatch and srun, and the Kubernetes platform underneath handles reliability and scaling. Each Slurm cluster has these components:
  • Login node: The SSH-accessible entry point where you write jobs, submit them, and retrieve results.
  • Slurm controller: Manages job queues and scheduling. It runs as Kubernetes pods.
  • Compute nodes: The GPU workers that run your jobs, registered with both Kubernetes and Slurm.
For how Slurm’s vocabulary maps onto the cluster’s layers, see Concepts.

Submit and manage jobs

Run these commands from the login node to submit, monitor, and cancel jobs:
Set memory limits explicitly in your sbatch scripts.Use a specific --mem value, such as --mem=500G, rather than --mem=0. --mem=0 tells Slurm to use all memory on the node, which can crash the node under load. If a job exceeds an explicit allocation, Slurm fails it with an OUT_OF_MEMORY error instead of crashing the node.Keep --mem at or below 90% of the node’s memory, which leaves headroom for system processes. If your job needs less, set it lower.

Distribute input files with job arrays

Use a Slurm job array to partition input files into chunks and distribute the chunks across nodes. For an example to adapt to your own processing, see arxiv-clean-slurm.sbatch, which processes RedPajama v1 data.

Run a Jupyter notebook on a GPU node

Reserve a node through Slurm, start Jupyter there, then tunnel to it from your local machine:
  1. From the login node, start an interactive session on the target node:
  2. Launch Jupyter from that session:
  3. From your local machine, open an SSH tunnel to the node through the cluster’s jump host:
    To get the exact hosts, including <LOCATION>, copy the node’s SSH command from the cluster details page. On Slinky v1.0 clusters, the node’s host is <WORKER_HOSTNAME>.slurm-workers-slurm.slurm instead of <NODE_NAME>.slurm-compute.slurm.
  4. Open http://localhost:8888 in your browser and sign in with the token Jupyter printed in step 2.

Install Python packages that persist

Login node environments can be reset when the login pod restarts. Create a virtual environment on the shared filesystem instead of installing packages system-wide:

Troubleshooting

When a job fails, check Slurm’s error output first. Inspect the job with scontrol show job <JOB_ID>, and view the scheduler logs with kubectl logs slurm-controller-0 -n slurm.

Jobs stuck in a pending state

The most common causes:
  • Insufficient resources: All GPUs are currently allocated. Check the queue and node availability with squeue and sinfo.
  • Request exceeds limits: The job requests more resources than any node provides.
  • Partition limits: The job targets a partition with limited capacity. You can adjust partitions in the Slurm configuration.
Inspect a specific job with:

“Unable to contact slurm controller (connect failure)”

The Slurm controller is unreachable, usually because it’s restarting or there’s a network issue between nodes. Wait a few minutes and retry. Check the controller pod’s status with kubectl get pods -n slurm, and see the restart and log commands in the Slurm configuration guide. If the issue persists for more than 30 minutes, contact support with the cluster name, the exact error, and when it started.

”couldn’t chdir to home directory” errors

The job’s working directory isn’t accessible on the compute node. On Slurm clusters, /home is a directory on the shared volume, which is mounted on every node, so this error usually means the shared volume mount is missing or unhealthy on that node. Set the working directory explicitly in your job script with #SBATCH --chdir=<path>, and confirm the path exists on the node. If /home isn’t mounted where it should be, contact support. For how /home behaves on each cluster type, see Cluster storage.

Node drained with reason “Kill task failed”

sinfo -R shows a node drained with Kill task failed when a job step’s processes don’t exit within UnkillableStepTimeout after Slurm sends them SIGKILL. Slinky v1.0 clusters set that timeout to 600 seconds. Slurm drains the node so no new work lands there. Common causes:
  • Unkillable processes, such as a process stuck in kernel I/O or in a hung GPU driver call.
  • Container teardown failures.
Other problems drain nodes with different reasons. A failed startup script can drain a node. On Slinky v1.0 clusters, a health check also drains a node whose GPU or DCGM check fails, with the reason GPUHealthcheckError or DCGMHealthcheckError, and resumes it once the check passes. On Slinky v1.0 clusters, the same health check resumes a node drained with Kill task failed once the node is idle and its GPUs are healthy. To recover the node manually:
  1. Confirm the reason and the affected jobs:
  2. As a project admin, resume the node from the login node:
If the node won’t resume, contact support with the cluster name, node name, drain reason, and the job IDs that were running. If your job caused the drain, make it easier to kill next time:
  • Handle SIGTERM promptly in your training script.
  • Checkpoint periodically so exits are fast.
  • Make sure all ranks of a distributed job exit when one rank is terminated.

Array jobs fail with “Invalid job array specification”

The array exceeds the cluster’s MaxArraySize limit. By default, the highest array index allowed is 1,000. To allow higher indexes, raise MaxArraySize in slurm.conf.

Jobs are killed before checkpointing finishes

Check the KillWait setting, which defaults to 30 seconds:
If your checkpoint takes longer than KillWait, raise KillWait in slurm.conf.

Next steps

Slurm configuration

Customize partitions, GPU resources, and scheduler settings.

Slurm startup scripts

Run lifecycle hook scripts at node startup, job start, and job completion.

Connect over SSH

Reach the login and compute nodes with OIDC or key-based SSH.

Quickstart

Create a Slurm cluster and run your first job.