> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Automate cluster operations

> Create, scale, and delete GPU clusters from CI/CD pipelines, scripts, and cron jobs.

Cluster lifecycle operations (create, scale, download credentials, delete) run headlessly, so a CI pipeline, script, or cron job can create a cluster, run work on it, and delete it without human interaction. The examples use the Together CLI, but the SDKs and [REST API](https://docs.together.ai/reference/clusters-create.md) support the same workflows.

Scripts and CI jobs can't log in with a browser, so authenticate with the `TOGETHER_API_KEY` environment variable instead of running `tg login`. Pass `--non-interactive` or `--json` on CLI commands so they never wait for a prompt. For CLI setup in a pipeline, see [Use the CLI in a CI/CD environment](https://docs.together.ai/reference/cli/getting-started.md#use-the-cli-in-a-ci%2Fcd-environment).

## Create and delete clusters from CI

Create an on-demand cluster for a pipeline run and delete it when the job finishes, so you pay only for the run itself.

To set this up in GitHub Actions, store an API key as a secret named `TOGETHER_API_KEY`, and add your Kubernetes job manifest to the repository as `training-job.yaml`. The key must belong to a project admin. With a key from any other role, the API omits the kubeconfig, so `get-credentials` writes nothing and the `kubectl` command in the training step fails.

Then add this workflow, which runs the training job on a new cluster for every push:

```yaml theme={null}
name: Train Model

on: push

jobs:
  train:
    runs-on: ubuntu-latest
    timeout-minutes: 240
    env:
      TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
    steps:
      - uses: actions/checkout@v5

      - uses: astral-sh/setup-uv@v7

      - name: Install the Together CLI
        run: uv tool install "together[cli]"

      - name: Create GPU Cluster
        run: |
          CLUSTER_ID=$(tg clusters create \
            --name training-${{ github.sha }} \
            --num-gpus 8 \
            --billing-type ON_DEMAND \
            --gpu-type H100_SXM \
            --region us-central-8 \
            --nvidia-driver-version 570 \
            --cuda-version 12.8 \
            --cluster-type KUBERNETES \
            --json | jq -r '.cluster_id')
          echo "CLUSTER_ID=$CLUSTER_ID" >> "$GITHUB_ENV"

      - name: Wait for the cluster and fetch credentials
        run: |
          until tg clusters retrieve "$CLUSTER_ID" --json | jq -e '.status == "Ready"' > /dev/null; do
            sleep 60
          done
          tg clusters get-credentials "$CLUSTER_ID" --set-default-context

      - name: Run Training
        run: |
          # Submit training job to cluster
          kubectl apply -f training-job.yaml

      - name: Cleanup
        if: always() && env.CLUSTER_ID != ''
        run: |
          tg clusters delete "$CLUSTER_ID" --non-interactive
```

The create step writes the new cluster's ID to `$GITHUB_ENV` as `CLUSTER_ID`, so later steps can find the cluster. If the cluster never reaches `Ready`, the job's `timeout-minutes` limit ends the wait loop. The `Cleanup` step runs whenever the create step returned an ID, so the workflow deletes the cluster even when a later step fails.

## Create clusters on a schedule

To create a cluster on a schedule, such as for daily batch work, save the `clusters create` command in a script:

```bash create-daily-batch.sh theme={null}
#!/usr/bin/env bash
set -euo pipefail
export PATH="$HOME/.local/bin:$PATH"  # cron doesn't load your shell profile

tg clusters create \
  --name daily-batch \
  --num-gpus 16 \
  --billing-type ON_DEMAND \
  --gpu-type H100_SXM \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --cluster-type KUBERNETES \
  --non-interactive
```

Because cron doesn't load your shell profile, set `TOGETHER_API_KEY` in the crontab or the script. Then add the script to your crontab:

```bash theme={null}
# Create a cluster daily at 06:00 (server time) for batch processing
0 6 * * * /path/to/create-daily-batch.sh
```

The script only creates the cluster. [Delete the cluster](https://docs.together.ai/docs/gpu-clusters-management.md#delete-a-cluster) when the batch finishes so it doesn't keep billing.

## Scale on your own signals

To scale a cluster up or down on your own signals, such as job queue length, run `tg clusters update` with a new GPU count:

```bash theme={null}
# Scale based on job queue length
if [ "$JOB_QUEUE_LENGTH" -gt 100 ]; then
  tg clusters update <CLUSTER_ID> --num-gpus 16
else
  tg clusters update <CLUSTER_ID> --num-gpus 8
fi
```

To scale automatically when pods are pending, use [cluster autoscaling](https://docs.together.ai/docs/gpu-clusters-management.md#autoscaling) instead.

## Best practices

* **Use environment variables for API keys:** Store the key as a CI secret and expose it as `TOGETHER_API_KEY`. Never hardcode it in scripts or pipeline definitions.
* **Reference clusters by ID:** IDs are generally more reliable than names in scripts. Capture the ID from the `--json` output of `tg clusters create`, as the CI workflow does.
* **Check cluster status before submitting jobs:** After you create a cluster, wait until its status is `Ready`.
* **Implement retry logic:** Handle transient failures with exponential backoff.
* **Clean up resources:** Delete on-demand clusters when the job completes, including when it fails.

## Troubleshooting

* **Commands fail with an authentication error:** The API key is missing or invalid. Check that it's set without printing it, for example with `test -n "$TOGETHER_API_KEY" && echo set`, and confirm that the key is valid in your [account settings](https://api.together.ai/settings/projects/~first/api-keys). To see which project and organization the CLI is using, run `tg whoami`.
* **Commands use the wrong project or credentials on a machine where you ran `tg login`:** While you're logged in, the CLI uses your login instead of `TOGETHER_API_KEY`. To use the key, run `tg logout` or pass `--api-key`.
* **Commands fail with rate-limit errors:** The script is calling the API too often. Retry with exponential backoff, and batch operations where possible. For higher limits, [contact support](https://portal.usepylon.com/together-ai/forms/support-request).

## Next steps

> **[Manage GPU clusters](https://docs.together.ai/docs/gpu-clusters-management.md)**
>
> Run every cluster operation from the console, CLI, or SDKs.

> **[Clusters CLI reference](https://docs.together.ai/reference/cli/clusters.md)**
>
> Look up every `tg clusters` command and flag.

> **[Clusters API reference](https://docs.together.ai/reference/clusters-create.md)**
>
> Check request and response shapes for the clusters REST API.

> **[SkyPilot](https://docs.together.ai/docs/skypilot-on-gpu-clusters.md)**
>
> Launch AI workloads on your cluster through SkyPilot task files.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.