Slurm startup scripts

​

Script types at a glance

• Worker init script: Runs on each Slurm worker at boot. Install system packages, configure drivers, or pull container images.
• Login init script: Runs on the login node at startup. Install CLI tools and packages for interactive SSH sessions.

• Worker prolog: Runs on each worker node before a job starts. Set up directories, load datasets into local scratch, or export environment variables.
• Worker epilog: Runs on each worker node after a job ends. Clean up scratch files, flush logs, or reset node state.
• Controller prolog: Runs on the Slurm controller (slurmctld) at job allocation. Validate job parameters, send notifications, or log job metadata centrally.
• Controller epilog: Runs on the Slurm controller (slurmctld) at job completion. Log results, trigger downstream pipelines, or send completion notifications.

• Extra slurm.conf: Custom lines appended to slurm.conf on all nodes. Use this for scheduler tuning, prolog flags, or partition overrides.

​

Node init scripts

​

Worker init script

• Install system packages (apt-get install -y sox ffmpeg).
• Install the AWS CLI or other data-transfer tools.
• Pull container images or download shared assets.
• Configure environment variables that apply to every job.

#!/bin/bash
set -e

# Install audio and video processing tools
apt-get update && apt-get install -y sox ffmpeg

# Install AWS CLI for S3 data transfers
pip install awscli

echo "Worker init complete"

​

Login init script

• Install CLI utilities for data exploration.
• Set up shared Python environments.
• Configure shell defaults for all users.

#!/bin/bash
set -e

# Install interactive tools
apt-get update && apt-get install -y htop tmux tree

echo "Login init complete"

​

Job lifecycle scripts

​

Worker prolog

• Create per-job scratch directories on local NVMe.
• Stage input data from shared storage to local disk.
• Export environment variables for the job.
• Verify GPU health before the job runs.

#!/bin/bash
set -e

# Create a per-job scratch directory on local NVMe
JOB_SCRATCH="/scratch/job_${SLURM_JOB_ID}"
mkdir -p "$JOB_SCRATCH"
chown "$SLURM_JOB_USER" "$JOB_SCRATCH"

echo "Worker prolog complete for job ${SLURM_JOB_ID}"

​

Worker epilog

• Remove per-job scratch directories.
• Flush job logs to shared storage.
• Reset GPU state or clear shared memory.
• Kill orphaned processes.

#!/bin/bash

# Clean up per-job scratch directory
JOB_SCRATCH="/scratch/job_${SLURM_JOB_ID}"
rm -rf "$JOB_SCRATCH"

# Kill any orphaned user processes
pkill -u "$SLURM_JOB_USER" || true

echo "Worker epilog complete for job ${SLURM_JOB_ID}"

​

Controller prolog

• Log job metadata to a central database.
• Send a Slack or webhook notification when a job starts.
• Validate job parameters before workers are assigned.

#!/bin/bash

# Log job start to a central file
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) JOB_START job_id=${SLURM_JOB_ID} user=${SLURM_JOB_USER}" >> /var/log/slurm/job_events.log

​

Controller epilog

• Log job completion and exit status.
• Trigger a downstream pipeline (fine-tuning, evaluation, deployment).
• Send a completion notification.

#!/bin/bash

# Log job completion
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) JOB_END job_id=${SLURM_JOB_ID} user=${SLURM_JOB_USER}" >> /var/log/slurm/job_events.log

​

Extra slurm.conf

• Add partitions or raise limits such as MaxArraySize.
• Tune scheduler parameters.
• Configure accounting or job completion plugins.

MaxArraySize=10001
SchedulerParameters=batch_sched_delay=10,bf_interval=180

​

View configured startup scripts

​

Add startup scripts to a new cluster

1. On the GPU clusters page, select Create cluster and set Orchestration to Slurm.
2. Under Cluster add-ons, set Startup Scripts to Yes. The text boxes below the toggle become editable.
3. Enter each script you need in its text box (Worker Prolog, Worker Epilog, Login Init Script, Worker Init Script, Extra slurm.conf, Controller Prolog, or Controller Epilog). Each script can be up to 64 KB. Every script except Extra slurm.conf must start with #!/bin/bash, #!/usr/bin/env bash, #!/bin/sh, or #!/usr/bin/env sh.
4. Complete the rest of the form, select Create Cluster, and then select Confirm & Create.

​

Update startup scripts on an existing cluster

1. On the cluster details page, select Edit Configuration.
2. Change the scripts, and then select Save.

• Extra slurm.conf: Takes effect on the running cluster through a live scontrol reconfigure. Workers briefly re-register with the controller, but running jobs aren’t interrupted.
• Prolog and epilog scripts: Take effect on a worker only when its pod restarts, because each worker fetches its scripts through Slurm’s configless mechanism at pod startup. Until then, new jobs on that worker keep using the old scripts, even though the underlying ConfigMaps update immediately. To apply the update without ending running jobs, drain the workers, wait for their jobs to finish, and then restart their pods with kubectl delete pods -n slurm -l app.kubernetes.io/name=slurmd.
• Init scripts: Take effect the next time each worker or login pod starts or restarts, because init scripts run at pod startup.

​

Failure handling

• The node is set to DRAIN state.
• The job is requeued (batch jobs only). Interactive jobs (salloc, srun) are cancelled.

• The node is set to DRAIN state.
• A drained node does not accept new jobs until an admin resumes it.

• The job is requeued (batch jobs) or cancelled (interactive jobs).
• The node is not affected.

• The failure is logged but has no other effect on the job or node.

​

Best practices

• Keep scripts short and fast, because long-running scripts delay job scheduling.
• Use set -e in init scripts so the script stops at the first failed command instead of continuing silently.
• Don’t call Slurm commands such as squeue, scontrol, or sacctmgr from prolog or epilog scripts. They can cause deadlocks and degrade scheduler performance.
• Use the SLURM_JOB_ID and SLURM_JOB_USER environment variables to scope cleanup and logging to the correct job.

​

Troubleshooting

​

Node stuck in DRAIN state after script failure

1. Find the drained node with sinfo -R.
2. Check that worker’s slurmd log (/var/log/slurm/slurmd.log) for the script’s error.
3. Fix the script.
4. As a project admin, resume the node from the login node:
   sudo scontrol update NodeName=<NODE_NAME> State=resume Reason="script fixed"
   

​

Init script packages not available in jobs

• SSH into a worker node and verify the package is installed.
• Add set -e to your init script to catch failures.

​

Additional resources

• Slurm prolog and epilog guide.
• Slurm environment variables.
• Slurm configuration.