> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Together Storage Accelerator

> Cache reads from object storage and Hugging Face Hub inside your GPU cluster.

Together Storage Accelerator (TSA) is a free add-on for GPU clusters that caches reads from S3-compatible object storage and Hugging Face Hub on your cluster's GPU nodes. The first read of an object fetches it from the origin. Every later read, from any node in the cluster, is served from local NVMe instead. Your workloads keep their existing S3 and Hugging Face clients and point them at an in-cluster endpoint.

In preview testing on an H100 node, warm reads ran at up to 3.4 GB/s per node, about 20x the throughput of reading the same object directly from a cross-cloud origin. Cold reads are bound by the origin's throughput. See [Performance](#performance) for the measurements. TSA is built on [ASTRA](https://docs.together.ai/docs/astra.md), Together's storage acceleration technology.

## When to use TSA

TSA helps most when many nodes read the same objects, or when the same objects are read repeatedly across jobs:

* **Model weights at startup:** A checkpoint that every replica loads is fetched from the origin once per cluster, not once per node.
* **Training data shared across nodes:** Dataset shards that every worker reads are served from the cache after the first epoch.
* **Hugging Face Hub downloads:** Model and dataset repositories are mirrored inside the cluster, with your token applied for gated repositories and rate limits.
* **Cross-region or cross-cloud buckets:** Objects in a distant region or another cloud are pulled over the WAN once, which cuts egress and removes the WAN from every later read.
* **Many pods starting at once:** Concurrent requests for the same uncached object collapse into a single origin fetch, so a fleet-wide cold start doesn't become an origin fetch storm.

TSA accelerates reads. Write to your origin directly: uploads and listings aren't cached, multipart uploads aren't supported through the TSA endpoint, and a plain `PUT` or `DELETE` sent to the endpoint passes through to the origin with the backend's credentials.

## How it works

TSA runs one cache instance on every GPU node in your cluster, in the `tsa` namespace. A single in-cluster service, `tsa.tsa.svc.cluster.local`, routes each pod to the instance on its own node, so the client-side hop never leaves the host. Instances share a cluster-wide cache: an object is split into fixed-size blocks, each block is cached on one node, and other nodes fetch it from that peer rather than from the origin. Nodes added to the cluster join the cache automatically.

Each instance uses a fixed portion of its node's local NVMe for cached blocks. When the cache fills, the least valuable blocks are evicted and fetched again from the origin on the next read.

## Enable the add-on

Before you start, you need:

* TSA enabled for your organization. If the TSA section doesn't appear in your cluster's create form or details page, contact your account team.
* Credentials for at least one S3-compatible backend, a Hugging Face token, or both.

> [tab] Console
>
> **1. Open the TSA section**
>
> In the [console](https://api.together.ai), turn on the **TSA** toggle in the cluster create form, or open an existing cluster's details page and turn on the **TSA** toggle there.
>
> **2. Add backends**
>
> Select **Add backend** and fill in the fields described in [Configure backends](#configure-backends). Add a **Hugging Face token** to enable the Hugging Face Hub mirror.
>
> **3. Save**
>
> Create the cluster, or select **Update** on the details page. Together installs TSA on the cluster's GPU nodes within a few minutes.

> [tab] API
>
> Create the add-on on an existing cluster. Backend credentials are sent once and stored encrypted; responses return them masked.
>
> ```bash cURL theme={null}
> curl -X POST "https://api.together.ai/v1/compute/clusters/<CLUSTER_ID>/addons" \
>   -H "Authorization: Bearer $TOGETHER_API_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "tsa",
>     "addOnType": "cs3",
>     "config": {
>       "tsa": {
>         "enabled": true,
>         "backends": [
>           {
>             "name": "training-data",
>             "url": "https://s3.us-west-2.amazonaws.com",
>             "region": "us-west-2",
>             "accessKeyId": "<ACCESS_KEY_ID>",
>             "secretAccessKey": "<SECRET_ACCESS_KEY>"
>           }
>         ],
>         "hfToken": "<HF_TOKEN>"
>       }
>     }
>   }'
> ```
>
> To change backends later, send a `PUT` to `/v1/compute/clusters/<CLUSTER_ID>/addons/<ADDON_ID>` with the same `config` shape. Omit credential fields to keep the stored values.

Confirm the install from inside the cluster. One pod runs per GPU node:

```bash theme={null}
kubectl -n tsa get daemonset,pods
```

If the cluster has no GPU nodes yet, the pods appear when the first GPU node joins. The install can report as failed in the meantime and retries on its own.

TSA installs and reports healthy even with no backends configured, but every read fails until you add one. Configure at least one backend or a Hugging Face token before pointing workloads at it.

## Configure backends

A backend is one S3-compatible origin that TSA caches. You can configure up to three backends per cluster, plus one Hugging Face token.

| Field | Description |
| - | - |
| Backend name | How your workloads address this backend in request paths. Use lowercase letters, digits, and hyphens. Defaults to `primary` when left blank. |
| Backend URL | The origin's S3 endpoint, for example `https://s3.us-west-2.amazonaws.com` or your Cloudflare R2 account endpoint. Don't include a bucket name or path. |
| Backend region | The signing region for the origin, for example `us-west-2`. For R2, use `auto`. |
| Access key ID and secret access key | Credentials TSA uses to sign requests to the origin. Required for every backend. Use read-only credentials: any pod in the cluster can send requests through TSA, including writes and deletes if the credentials allow them. |

The Hugging Face token is optional. Add one to enable the Hugging Face Hub mirror, to download gated repositories, and to avoid the rate limits Hugging Face applies to anonymous downloads. In preview testing, anonymous downloads started returning 429 responses after roughly 49 GB.

Together stores backend credentials and the Hugging Face token encrypted in a secrets vault and delivers them to your cluster as Kubernetes secrets in the `tsa` namespace. The console shows them masked after you save. To rotate credentials, enter the new values and select **Update**. Leave a credential field blank to keep its stored value.

## Use TSA in workloads

The TSA endpoint is `http://tsa.tsa.svc.cluster.local:18080`. It isn't exposed outside the cluster. It's reachable from pods and from anything else on the cluster's node network, because the TSA pods run on the host network.

### Read from an S3-compatible backend

Point your S3 client at the TSA endpoint and prefix every object path with the backend name: `<BACKEND_NAME>/<BUCKET>/<KEY>`. Your S3 client treats the backend name as the bucket and the rest of the path as the key. Client credentials aren't used, so any value works. TSA signs the origin request with the backend's credentials.

Set the endpoint for every S3 client in a pod with `AWS_ENDPOINT_URL`:

```yaml theme={null}
env:
  - name: AWS_ENDPOINT_URL
    value: "http://tsa.tsa.svc.cluster.local:18080"
  - name: AWS_ACCESS_KEY_ID
    value: "unused"
  - name: AWS_SECRET_ACCESS_KEY
    value: "unused"
```

With a backend named `training-data`, read `datasets/tokens.bin` from the bucket `my-bucket`:

```python Python theme={null}
import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="http://tsa.tsa.svc.cluster.local:18080",
    aws_access_key_id="unused",
    aws_secret_access_key="unused",
    config=Config(s3={"addressing_style": "path"}),
)
s3.download_file(
    "training-data",
    "my-bucket/datasets/tokens.bin",
    "/data/tokens.bin",
)
```

```bash AWS CLI theme={null}
aws s3 cp s3://training-data/my-bucket/datasets/tokens.bin /data/tokens.bin \
  --endpoint-url http://tsa.tsa.svc.cluster.local:18080
```

```bash cURL theme={null}
curl -o /data/tokens.bin \
  http://tsa.tsa.svc.cluster.local:18080/training-data/my-bucket/datasets/tokens.bin
```

TSA caches `GET` and `HEAD` requests, including range requests, so parallel downloaders that fetch byte ranges work as expected. Listings pass through to the origin and are always current. Addressing is path-style only. Virtual-hosted-style requests, where the bucket is part of the hostname, aren't supported.

### Download from Hugging Face Hub

Set `HF_ENDPOINT` in the pod. `huggingface_hub`, and every library built on it such as `transformers`, then downloads through the mirror:

```yaml theme={null}
env:
  - name: HF_ENDPOINT
    value: "http://tsa.tsa.svc.cluster.local:18080"
```

Existing code works unmodified:

```python huggingface_hub theme={null}
from huggingface_hub import snapshot_download

snapshot_download("Qwen/Qwen3-8B")
```

```python transformers theme={null}
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
```

The mirror serves the Hub's `/api/*` and `*/resolve/*` paths. TSA resolves a branch or tag such as `main` to its commit hash before caching, so a repository whose `main` has moved never serves stale files. The mirror is active only when a Hugging Face token is configured for the add-on.

### Warm the cache before a job

Prefetch an object so a job's first read is already warm. Prefetching populates the cache across the cluster, with each node fetching the blocks it owns, without writing files anywhere:

```bash theme={null}
curl -X POST \
  "http://tsa.tsa.svc.cluster.local:18080/prefetch/training-data/my-bucket/checkpoints/model.safetensors"
```

The path is `/prefetch/<BACKEND_NAME>/<BUCKET>/<KEY>`. Run it from any pod in the cluster before the job starts, for example in an init container or a pre-job step. The response reports per-node results.

## Monitor TSA

Check the add-on from inside the cluster:

```bash theme={null}
# One pod per GPU node, all Ready
kubectl -n tsa get daemonset,pods

# Health of the instance that serves this pod
curl http://tsa.tsa.svc.cluster.local:18080/health
```

`/health` returns `OK` when the local instance is running. It doesn't check the cache disk or the origins.

TSA exposes Prometheus metrics on port `9091` of every GPU node, not on the `tsa` service. The pods run on the node network, so scrape `http://<NODE_IP>:9091/metrics` for each GPU node, or discover the pods in the `tsa` namespace and scrape their `metrics` port. The metrics cover cache hit rate by tier, bytes served from cache and from the origin, origin request latency and errors by backend type, evictions, disk health, and peer request latency.

## Performance

These numbers were measured on 2026-10-28 from a pod on a single H100 GPU node in a Together GPU Cluster in us-central, with Cloudflare R2 as the S3-compatible backend and Hugging Face Hub as the origin for the model download. Each read was a single stream unless noted. Clients wrote to `/dev/null` or the node's local NVMe, so the numbers reflect the cache, not the client's disk.

![Bar chart of single-stream read throughput on one H100 node: a warm TSA read at 3.2 GB/s, the Hugging Face origin at 0.97 GB/s, the R2 origin at 0.16 GB/s, and a cold TSA read at 0.14 GB/s, with the node-local cache ceiling at 3.4 GB/s](https://mintcdn.com/togetherai-52386018/XzYKxioNK_GZTd6u/images/tsa-benchmark-throughput.png?fit=max\&auto=format\&n=XzYKxioNK_GZTd6u\&q=85\&s=2cfeeb770e4921940eb59788ee0aeb15)

| Read | Object | Wall time | Throughput |
| - | - | - | - |
| R2 origin, direct with the AWS CLI | 2 GiB | 12.7 to 15.0 s | 143 to 169 MB/s |
| R2 through TSA, cold (first read) | 2 GiB | 16.0 s | 135 MB/s |
| R2 through TSA, warm | 2 GiB | 0.64 to 0.72 s | 3.0 to 3.4 GB/s |
| R2 through TSA, warm, four parallel range requests | 2 GiB | 0.67 s | 3.2 GB/s aggregate |
| Small objects through TSA, cold then warm | 5 KB and 2.4 MB | 320 ms then 5 ms, 680 ms then 8 ms | |
| Hugging Face Hub origin, `Qwen/Qwen3-8B` with `hf_transfer` | 16.4 GB | 16.9 s | 0.97 GB/s |
| Node-local cache ceiling, object served with no origin | 4 GiB | 1.23 to 1.28 s | 3.4 to 3.5 GB/s |

What to expect from these numbers:

* **A cold read costs about the same as reading the origin directly.** The proxy hop added about 5 percent to the wall time of a 2 GiB download.
* **Warm reads run within 10 percent of the node's local cache ceiling.** The proxy isn't the bottleneck; a single-stream client is. Four parallel readers on one node share the same ceiling.
* **Origin throughput varies by region.** The same Hugging Face download ran at 0.20 GB/s from a cluster in ca-west and 0.97 GB/s from us-central, so the warm-read speedup is largest where the origin is slowest.
* **Reads served from a peer node weren't measured.** Every number above is from the node that holds the cached blocks. A read whose blocks live on another GPU node also crosses the cluster network.

## Caching behavior and consistency

TSA is built for immutable content: model weights, datasets, and checkpoints written once under a new key. Keep these behaviors in mind:

* **Overwritten objects:** An object overwritten at the same key in the origin can keep serving the previously cached bytes until they're evicted, and a read that spans the change can return a mix of old and new data. Don't overwrite objects in place. Write changed data under a new key or version suffix, or read it with `Cache-Control: no-store` to bypass the cache.
* **Listings:** `List` requests always go to the origin, so listings reflect the origin's current state.
* **Hugging Face revisions:** Cache keys are pinned to the resolved commit, so moved branches and tags never serve stale files.
* **Cache capacity:** Each node contributes a fixed share of its local NVMe. When a node leaves the cluster, the blocks it held are fetched again from the origin on the next read.
* **Origin outages:** Cached blocks can keep serving while the origin is unavailable, as long as the object's cached metadata hasn't expired. Uncached objects fail with the origin's error.

## Security

* **Network access:** The TSA endpoint isn't exposed outside the cluster. Because the TSA pods run on the host network, the proxy listens on each GPU node's IP, so it's reachable from pods and from anything else on the cluster's node network. Any pod in the cluster can send requests through any configured backend, so treat the cluster's workloads as trusted, give backends read-only credentials, and add a Kubernetes NetworkPolicy to restrict which namespaces can reach the `tsa` namespace when more than one team shares the cluster.
* **Client credentials:** TSA doesn't authenticate S3 clients. It signs origin requests with the backend credentials you configured, for writes and deletes as well as reads. Give each backend read-only credentials unless workloads need to write through it.
* **Credential storage:** Together stores backend credentials and the Hugging Face token encrypted in a secrets vault, delivers them to your cluster as Kubernetes secrets in the `tsa` namespace, and masks them in the console. Anyone with cluster-admin access to your cluster can read those secrets.

## Troubleshooting

* **The TSA section doesn't appear in the console:** TSA isn't enabled for your organization yet. Contact your account team.
* **The console won't save the add-on:** Enabling TSA requires at least one backend with a name, access key ID, and secret access key, or a Hugging Face token. A backend with a name but no credentials, or credentials but no name, fails validation.
* **Pods in the `tsa` namespace are in `CrashLoopBackOff`:** Run `kubectl -n tsa logs daemonset/tsa-s3-cache-proxy-chart --tail=5`. A `Failed to create S3 source client` error that ends in `credentials files ... are empty` means the backend's credentials didn't reach the cluster. Re-enter the access key ID and secret access key in the console and select **Update**. If the pods keep crashing, contact support.
* **No pods in the `tsa` namespace:** TSA runs only on GPU nodes. If the cluster is still provisioning, the pods appear when the first GPU node joins. If the cluster has GPU nodes and the add-on is enabled, contact support.
* **Every read returns `500` with `InternalError` and a message about `EC2 IMDS` or `get credentials`:** The request reached TSA's default origin instead of one of your backends. That happens when the add-on has no backends configured, or when the request path doesn't start with a backend name. Add a backend, then prefix the path with its name.
* **`403` or `NoSuchBucket` on a read:** Check that the request path starts with the backend name. A path that starts with a bucket name is sent to the default origin, not to your backend. Then check the backend's region and credentials against the origin.
* **`404` on a read that works against the origin directly:** Confirm the backend URL is the origin's endpoint only, with no bucket name or path appended.
* **`502` on a read:** TSA couldn't reach the origin. Check the backend URL and the origin's availability from inside the cluster.
* **Hugging Face downloads bypass the cache:** The mirror is active only when a Hugging Face token is configured. Add a token to the add-on, then confirm `HF_ENDPOINT` is set in the pod.
* **Hugging Face returns `401`, `403`, or `429`:** The repository is gated or the download hit Hugging Face's anonymous rate limit. Add a token with access to the repository.
* **Rotated credentials don't take effect:** TSA picks up updated credentials a few minutes after you save them, without a restart. If reads still fail after that, check the new credentials against the origin directly.

## Limits

* Up to three S3-compatible backends and one Hugging Face token per cluster.
* Path-style addressing only.
* Reads only. Uploads, multipart uploads, and listings go to the origin.
* Container image caching and bulk data ingest into cluster storage aren't part of the add-on.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.