Automate cluster operations

Cluster lifecycle operations (create, scale, download credentials, delete) run headlessly, so a CI pipeline, script, or cron job can create a cluster, run work on it, and delete it without human interaction. The examples use the Together CLI, but the SDKs and REST API support the same workflows. Scripts and CI jobs can’t log in with a browser, so authenticate with the TOGETHER_API_KEY environment variable instead of running tg login. Pass --non-interactive or --json on CLI commands so they never wait for a prompt. For CLI setup in a pipeline, see Use the CLI in a CI/CD environment.

Create and delete clusters from CI

Create an on-demand cluster for a pipeline run and delete it when the job finishes, so you pay only for the run itself. To set this up in GitHub Actions, store an API key as a secret named TOGETHER_API_KEY, and add your Kubernetes job manifest to the repository as training-job.yaml. The key must belong to a project admin. With a key from any other role, the API omits the kubeconfig, so get-credentials writes nothing and the kubectl command in the training step fails. Then add this workflow, which runs the training job on a new cluster for every push:

name: Train Model

on: push

jobs:
  train:
    runs-on: ubuntu-latest
    timeout-minutes: 240
    env:
      TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
    steps:
      - uses: actions/checkout@v5

      - uses: astral-sh/setup-uv@v7

      - name: Install the Together CLI
        run: uv tool install "together[cli]"

      - name: Create GPU Cluster
        run: |
          CLUSTER_ID=$(tg clusters create \
            --name training-${{ github.sha }} \
            --num-gpus 8 \
            --billing-type ON_DEMAND \
            --gpu-type H100_SXM \
            --region us-central-8 \
            --nvidia-driver-version 570 \
            --cuda-version 12.8 \
            --cluster-type KUBERNETES \
            --json | jq -r '.cluster_id')
          echo "CLUSTER_ID=$CLUSTER_ID" >> "$GITHUB_ENV"

      - name: Wait for the cluster and fetch credentials
        run: |
          until tg clusters retrieve "$CLUSTER_ID" --json | jq -e '.status == "Ready"' > /dev/null; do
            sleep 60
          done
          tg clusters get-credentials "$CLUSTER_ID" --set-default-context

      - name: Run Training
        run: |
          # Submit training job to cluster
          kubectl apply -f training-job.yaml

      - name: Cleanup
        if: always() && env.CLUSTER_ID != ''
        run: |
          tg clusters delete "$CLUSTER_ID" --non-interactive

The create step writes the new cluster’s ID to $GITHUB_ENV as CLUSTER_ID, so later steps can find the cluster. If the cluster never reaches Ready, the job’s timeout-minutes limit ends the wait loop. The Cleanup step runs whenever the create step returned an ID, so the workflow deletes the cluster even when a later step fails.

Create clusters on a schedule

To create a cluster on a schedule, such as for daily batch work, save the clusters create command in a script:

create-daily-batch.sh

#!/usr/bin/env bash
set -euo pipefail
export PATH="$HOME/.local/bin:$PATH"  # cron doesn't load your shell profile

tg clusters create \
  --name daily-batch \
  --num-gpus 16 \
  --billing-type ON_DEMAND \
  --gpu-type H100_SXM \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --cluster-type KUBERNETES \
  --non-interactive

Because cron doesn’t load your shell profile, set TOGETHER_API_KEY in the crontab or the script. Then add the script to your crontab:

# Create a cluster daily at 06:00 (server time) for batch processing
0 6 * * * /path/to/create-daily-batch.sh

The script only creates the cluster. Delete the cluster when the batch finishes so it doesn’t keep billing.

Scale on your own signals

To scale a cluster up or down on your own signals, such as job queue length, run tg clusters update with a new GPU count:

# Scale based on job queue length
if [ "$JOB_QUEUE_LENGTH" -gt 100 ]; then
  tg clusters update <CLUSTER_ID> --num-gpus 16
else
  tg clusters update <CLUSTER_ID> --num-gpus 8
fi

To scale automatically when pods are pending, use cluster autoscaling instead.

Best practices

Use environment variables for API keys: Store the key as a CI secret and expose it as TOGETHER_API_KEY. Never hardcode it in scripts or pipeline definitions.
Reference clusters by ID: IDs are generally more reliable than names in scripts. Capture the ID from the --json output of tg clusters create, as the CI workflow does.
Check cluster status before submitting jobs: After you create a cluster, wait until its status is Ready.
Implement retry logic: Handle transient failures with exponential backoff.
Clean up resources: Delete on-demand clusters when the job completes, including when it fails.

Troubleshooting

Commands fail with an authentication error: The API key is missing or invalid. Check that it’s set without printing it, for example with test -n "$TOGETHER_API_KEY" && echo set, and confirm that the key is valid in your account settings. To see which project and organization the CLI is using, run tg whoami.
Commands use the wrong project or credentials on a machine where you ran tg login: While you’re logged in, the CLI uses your login instead of TOGETHER_API_KEY. To use the key, run tg logout or pass --api-key.
Commands fail with rate-limit errors: The script is calling the API too often. Retry with exponential backoff, and batch operations where possible. For higher limits, contact support.

Next steps

Manage GPU clusters

Run every cluster operation from the console, CLI, or SDKs.

Clusters CLI reference

Look up every tg clusters command and flag.

Clusters API reference

Check request and response shapes for the clusters REST API.

SkyPilot

Launch AI workloads on your cluster through SkyPilot task files.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Create and delete clusters from CI

Create clusters on a schedule

Scale on your own signals

Best practices

Troubleshooting

Next steps

Manage GPU clusters

Clusters CLI reference

Clusters API reference

SkyPilot

​Create and delete clusters from CI

​Create clusters on a schedule

​Scale on your own signals

​Best practices

​Troubleshooting

​Next steps

Manage GPU clusters

Clusters CLI reference

Clusters API reference

SkyPilot

Create and delete clusters from CI

Create clusters on a schedule

Scale on your own signals

Best practices

Troubleshooting

Next steps