Skip to main content
ASTRA (Accelerated Storage Transport and Retrieval Architecture) is Together’s distributed caching layer for object storage. It runs on every GPU node in a cluster, presents protocol-compatible endpoints for S3-compatible stores and Hugging Face Hub, and caches the blocks it fetches on node-local NVMe. Nodes share their caches over a peer mesh, so an object is fetched from the origin once per cluster and served from inside the cluster after that. ASTRA is the technology behind Together Storage Accelerator (TSA), the free add-on for GPU clusters. This page explains how it works and why it’s designed the way it is. For enabling and using the add-on, see the TSA page.

The problem

Object storage is durable and cheap, and it’s the right place to keep model weights, checkpoints, and datasets. It’s also far from the GPUs. A single request costs tens to hundreds of milliseconds. Reads across regions or clouds are metered, and public origins rate-limit heavy pulls. A training job that starts by pulling a 300 GB checkpoint onto 64 nodes pays that cost 64 times, and every replica waits on the slowest link. ASTRA moves the cost to the first read. After that, reads come from NVMe inside the cluster, at local speeds, with no egress.

How a read is served

A client pod connects to the ASTRA instance on its own node. That instance looks for each block of the object in order:
After warm-up, a cluster serving a stable working set runs at a high cache hit rate with close to zero origin traffic. In preview deployments on GPU clusters, warm reads measured up to 12x faster than the origin, at up to 3.4 GB/s per node. Cold reads are bound by the origin: 300 to 380 MB/s from Hugging Face Hub in the same tests.

Design choices

Blocks, not objects

ASTRA splits every object into fixed-size blocks and caches each block independently. A request for a byte range fetches only the blocks that cover it, so HTTP range requests are first-class: a parallel downloader that reads an object in slices gets exactly the slices it asked for, from whichever tier holds them. Memory use doesn’t depend on object size, because an instance never holds a whole object in RAM.

One owner per block

Every block has exactly one owning node, chosen by rendezvous hashing over the cluster’s current membership. Every instance computes the same owner independently, so there’s no master or metadata service to run. An instance that receives a request for a block it doesn’t own fetches it from the owner over a lightweight binary peer protocol, then serves it to the client. This means:
  • Each block is normally cached once across the cluster, so cache capacity isn’t spent on duplicates.
  • A cache hit is at most one network hop away.
  • Cluster cache capacity grows linearly with node count.
Nodes that join or leave the cluster are picked up automatically, and ownership rebalances. Blocks whose owner changes are refilled from the origin on their next read, so scaling a cluster up or down causes a brief, proportional increase in origin traffic.

One origin fetch per cluster

Concurrent requests for the same uncached block collapse into a single origin fetch, and every waiting request is served from that one result. Because requests from every node are routed to the block’s owner, the origin normally sees one download per block for the whole cluster. When hundreds of pods cold-start on the same checkpoint, the origin sees one download, not hundreds, and no pod waits on a request that another pod already made.

Streaming with readahead

For whole-object downloads, an instance fetches several blocks ahead of the position the client has consumed, so a reader streaming a multi-gigabyte file never stalls waiting for the next block. Object metadata (size, ETag, last-modified) is cached separately for a few minutes, so repeated HEAD requests don’t round-trip to the origin.

Node-local by construction

The in-cluster service routes each pod to the ASTRA instance on its own node, so the client-side hop never leaves the host. Peer traffic between instances stays on the cluster network. The origin is involved only on a cluster-wide miss.

Integrity checks on disk

Each block on disk carries a checksum. The checksum is verified every time the block is read from disk, and a block that fails verification is dropped and refetched rather than served. At startup the instance rebuilds its index from disk, so blocks cached before a restart are hits immediately, and a background pass verifies every one of them.

Warming the cache ahead of time

You can ask ASTRA to fetch an object before any workload reads it. The request fans out across the cluster, and each node fetches only the blocks it owns, so a large checkpoint is warmed in parallel by every node and stored once. When the job starts, its first read is already a cache hit.

Multiple origins, one cache

Every origin flows through the same block, peer, and NVMe path:
  • S3-compatible stores: AWS S3, Cloudflare R2, MinIO, and any other S3-compatible endpoint. Each configured backend has its own endpoint, region, and credentials, and its blocks are namespaced so two backends never collide.
  • Hugging Face Hub: A transparent mirror of the Hub’s download and API URL layout. A moving reference such as main is resolved to its commit hash before any bytes are cached, and blocks are keyed by that commit, so a repository whose main has moved never serves stale files. The mirror is active when a Hugging Face token is configured for the cluster. ASTRA uses it for gated repositories and to avoid anonymous rate limits, and picks up a moved reference within about a minute.
Clients don’t need to know which tier served them. An S3 client pointed at the ASTRA endpoint, or a Hugging Face client with HF_ENDPOINT set, works unmodified.

Self-protection

ASTRA is built to stay out of the way when something degrades:
  • Origin protection: An adaptive limiter bounds concurrent origin fetches per node and backs off when the origin slows or throttles, so a cluster-wide cold start can’t turn into an origin fetch storm.
  • Disk health: Each instance periodically checks that its cache filesystem is writable and still mounted. When a check fails, the instance stops writing new blocks and keeps serving what it has, instead of filling a sick disk or evicting its neighbors.
  • Bounded memory: The number of in-flight streams per instance is capped, and each stream holds a fixed number of blocks in flight, so memory use has a deterministic ceiling at any client concurrency. Requests over the cap queue and are admitted as streams complete.
  • Eviction: When the cache fills, blocks are evicted by a frequency-aware policy weighted by block size, so a single large scan doesn’t flush frequently read data. Evicted blocks are refetched from the origin on the next read.

Consistency model

Because every byte ASTRA caches also lives in the origin, any block can be evicted or lost without losing data. Correctness never depends on the cache. ASTRA assumes content is immutable: model weights, datasets, and checkpoints written once under a new key. If an object is overwritten at the same key in the origin, ASTRA can keep serving the previously cached bytes until they’re evicted. A read that spans the change can also return a mix of old and new data. Don’t overwrite objects in place. For mutable data, write under a new key or version suffix, or send Cache-Control: no-store to read that request’s data directly from the origin. Listings always pass through to the origin. ETag-based conditional requests (If-Match, If-None-Match, and If-Range with an ETag) are honored on cached reads. Date-based conditions aren’t evaluated. A range that starts past the end of the object returns 416.

Workload use cases

Training

A training job’s data dependencies are shared by every node and read many times: the base checkpoint or the last checkpoint to resume from, the dataset shards, and the tokenizer and config files. Without a cache, every node pulls its own copy from the origin at job start, which puts the whole cluster behind the slowest WAN link, multiplies egress by the node count, and turns a restart after a node failure into another full pull. Each epoch that re-reads the dataset from the origin pays the same cost again. With ASTRA in front of the origin:
  • Job start: The checkpoint and dataset are fetched from the origin once per cluster. Every other node reads them from a peer at local speeds. Warm the cache before the job is scheduled so even the first node starts from a hit.
  • Epochs: Data loaders that re-read shards each epoch, or read them in slices with range requests, are served from NVMe inside the cluster. The origin is never touched again for data that’s already cached.
  • Restarts and resumes: When a job restarts, the data it needs is still in the cache, so recovery time is bound by the engine, not the download.
  • Cross-region data: A dataset that lives in another region or cloud is pulled over the WAN once, not once per node per run.
Write checkpoints to your origin directly. ASTRA accelerates reads.

Inference

A serving replica can’t take traffic until its weights are loaded, and model weights are tens to hundreds of gigabytes. Without a cache, every new replica pulls the full model from the origin, so scaling out is as slow as the origin and the network path, and public origins such as Hugging Face Hub throttle heavy pulls. With ASTRA in front of the origin:
  • Cold start: The first replica pays the origin cost once. Prefetch the model ahead of a rollout so even the first replica starts warm.
  • Scale-out: Every additional replica, on any node, reads the weights from inside the cluster, so time to ready is bound by how fast the engine loads from local NVMe rather than by the origin.
  • Hugging Face models: With a token configured, the Hub mirror serves gated repositories, and branch and tag references are pinned to a commit so a rollout never mixes versions.

ASTRA in Together’s inference clusters

ASTRA was built for Together’s own inference fleet before it shipped to GPU clusters as Together Storage Accelerator. On those clusters it’s the data path for bringing a model online: model weights are cached in-cluster so the first replica of a model pays the origin cost once and every replica after it starts from a warm, node-local read. That takes the weight download off the cold-start critical path and is the same block, peer, and NVMe path described above.

Where ASTRA runs

  • Together Storage Accelerator: The free add-on for GPU clusters. See Together Storage Accelerator to enable it and point your workloads at it.
  • Together’s inference clusters: Model weight caching for cold starts, across Together’s serving fleet.
  • Hardware: x86-64 and Arm (NVIDIA Grace) GPU nodes.