Slurm configuration

Requirements

• The kubectl CLI installed and configured.
• Your cluster’s kubeconfig, downloaded with tg clusters get-credentials. Only project admins can download it.
• Access to your cluster’s slurm namespace.

Slinky v1.0 clusters

1. On the cluster details page in the Together AI console, select Edit Configuration.
2. Add your lines to Extra slurm.conf, and then select Save.

Slinky v0.1 clusters

View the configuration

# List the Slurm ConfigMaps
kubectl get configmaps -n slurm | grep slurm

# View slurm.conf content
kubectl get configmap slurm -n slurm -o jsonpath='{.data.slurm\.conf}'

# View gres.conf content
kubectl get configmap slurm -n slurm -o jsonpath='{.data.gres\.conf}'

Edit the configuration

kubectl edit configmap slurm -n slurm

# Export to local file
kubectl get configmap slurm -n slurm -o yaml > slurm-config.yaml

# Edit locally
# ... make your changes ...

# Apply changes
kubectl apply -f slurm-config.yaml

Restart components

kubectl rollout restart statefulset slurm-controller -n slurm

# Drain every compute node so no new jobs start on it
kubectl exec slurm-controller-0 -n slurm -- scontrol update nodename=<NODE_NAMES> state=drain reason="config change"

# After the running jobs finish, restart the compute node pods
kubectl delete pods -n slurm -l app.kubernetes.io/name=slurmd

kubectl rollout restart statefulset slurm-accounting -n slurm

Verify changes

# Check rollout status
kubectl rollout status statefulset slurm-controller -n slurm

# Watch pod status until pods are back up
kubectl get pods -n slurm -w

# Verify configuration in pod
kubectl exec -it slurm-controller-0 -n slurm -- cat /etc/slurm/slurm.conf

# Test Slurm functionality
kubectl exec -it slurm-controller-0 -n slurm -- scontrol show config
kubectl exec -it slurm-controller-0 -n slurm -- sinfo

Configuration examples

Configure GPU resources

Name=gpu Type=default File=/dev/nvidia[0-7]

Modify partitions

PartitionName=long Nodes=ALL MaxTime=7-00:00:00 State=UP

Tune the scheduler

SchedulerParameters=batch_sched_delay=10,bf_interval=180,sched_max_job_start=500

Update resource allocation

DefMemPerNode=512000  # 500 GB per node unless the job sets --mem

Enable cgroup limits

CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainRAMSpace=yes

ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity

Troubleshooting

• Configuration not applied: On Slinky v1.0 clusters, check that scontrol show config lists your setting. On Slinky v0.1 clusters, confirm the ConfigMap holds your edit with kubectl get configmap slurm -n slurm -o yaml, and check that the pod ages reset after the restart with kubectl get pods -n slurm. Then view the controller logs with kubectl logs slurm-controller-0 -n slurm.
• Syntax errors: Search the controller logs for errors with kubectl logs slurm-controller-0 -n slurm | grep -i error, and view recent events with kubectl get events -n slurm --sort-by='.lastTimestamp'.
• Pods not restarting: Check the controller’s rollout status with kubectl rollout status statefulset slurm-controller -n slurm, then delete the controller pod with kubectl delete pod slurm-controller-0 -n slurm to force Kubernetes to recreate it.
• Jobs failing after changes: Check node status with kubectl exec -it slurm-controller-0 -n slurm -- sinfo. Then, with the same kubectl exec prefix, inspect a node with scontrol show node <NODE_NAME> and view job errors with scontrol show job <JOB_ID>.

Additional resources

• Slurm configuration tool: Interactive configuration generator.
• Slurm configuration reference: Complete parameter documentation.
• GRES configuration: GPU and resource configuration guide.
• Scheduling configuration: Scheduler tuning guide.