> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference metrics: TTFT & TPS

> Two numbers that describe how fast an LLM feels.

export const TimingDiagram = () => {
  const CSS = `
  .learn-diagram .timing { display: grid; gap: var(--space-sm); }
  .learn-diagram .timing__lab {
    font-family: var(--font-mono);
    font-size: 11px;
    letter-spacing: 0.1em;
    text-transform: uppercase;
    color: var(--color-ink-mute);
    margin-bottom: 6px;
    display: flex;
    justify-content: space-between;
    gap: var(--space-2xs);
  }
  .learn-diagram .timing__track {
    height: 64px;
    background: var(--color-paper-3);
    border-radius: var(--radius-md);
    position: relative;
    overflow: hidden;
  }
  .learn-diagram .timing__prefill {
    position: absolute;
    top: 0;
    bottom: 0;
    left: 0;
    background: var(--tg-purple-mid);
    display: flex;
    align-items: center;
    justify-content: center;
    font-family: var(--font-mono);
    font-size: 11px;
    letter-spacing: 0.06em;
    color: var(--tg-navy);
    transition: width var(--dur-slow) var(--ease-out);
    overflow: hidden;
    white-space: nowrap;
  }
  .learn-diagram .timing__tick {
    position: absolute;
    top: 14px;
    width: 3px;
    height: 36px;
    border-radius: 1px;
    background: var(--tg-orange);
    transition: opacity var(--dur-fast) linear;
  }
  .learn-diagram .timing__playhead {
    position: absolute;
    top: 0;
    bottom: 0;
    width: 2px;
    background: var(--tg-navy);
  }
  .learn-diagram .timing__scale {
    display: flex;
    justify-content: space-between;
    font-family: var(--font-mono);
    font-size: 11px;
    color: var(--color-ink-faint);
    margin-top: 6px;
    font-variant-numeric: tabular-nums;
  }
  .learn-diagram .timing__metrics {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));
    gap: var(--space-sm);
  }
  .learn-diagram .timing__metric { display: grid; gap: 2px; }
  .learn-diagram .timing__metric b {
    font-family: var(--font-display);
    font-size: 30px;
    font-weight: 500;
    line-height: 1.05;
    letter-spacing: -0.02em;
    color: var(--tg-orange);
    font-variant-numeric: tabular-nums;
  }
  .learn-diagram .timing__metric span {
    font-family: var(--font-mono);
    font-size: 11px;
    letter-spacing: 0.08em;
    text-transform: uppercase;
    color: var(--color-ink-mute);
  }
  `;
  const SPEEDUP = 4;
  const MAX_TICKS = 110;
  const [promptK, setPromptK] = useState(2);
  const [outTokens, setOutTokens] = useState(200);
  const [tps, setTps] = useState(100);
  const [play, setPlay] = useState({
    active: false,
    elapsed: 0
  });
  const raf = useRef(null);
  const ttft = 80 + promptK * 50;
  const totalMs = ttft + outTokens / tps * 1000;
  const stop = () => {
    if (raf.current) cancelAnimationFrame(raf.current);
    raf.current = null;
  };
  useEffect(() => stop, []);
  const run = () => {
    stop();
    const start = performance.now();
    const total = totalMs;
    const tick = now => {
      const elapsed = (now - start) * SPEEDUP;
      if (elapsed < total) {
        setPlay({
          active: true,
          elapsed
        });
        raf.current = requestAnimationFrame(tick);
      } else {
        setPlay({
          active: false,
          elapsed: total
        });
        raf.current = null;
      }
    };
    setPlay({
      active: true,
      elapsed: 0
    });
    raf.current = requestAnimationFrame(tick);
  };
  const reset = () => {
    stop();
    setPlay({
      active: false,
      elapsed: 0
    });
  };
  const change = setter => v => {
    stop();
    setter(v);
    setPlay({
      active: false,
      elapsed: 0
    });
  };
  const finished = !play.active && play.elapsed > 0;
  const tokensDone = finished ? outTokens : Math.max(0, Math.min(outTokens, Math.floor((play.elapsed - ttft) / (1000 / tps))));
  const perTick = Math.max(1, Math.ceil(outTokens / MAX_TICKS));
  const nTicks = Math.ceil(outTokens / perTick);
  const ticks = [];
  for (let i = 0; i < nTicks; i++) {
    const tokenIndex = Math.min((i + 1) * perTick, outTokens);
    const t = ttft + tokenIndex * (1000 / tps);
    ticks.push(<div key={i} className="timing__tick" style={{
      left: "calc(" + t / totalMs * 100 + "% - 1.5px)",
      opacity: tokenIndex <= tokensDone ? 0.9 : 0.16
    }} />);
  }
  return <DiagramFrame eyebrow="Latency" title="A request is one slow gulp, then a steady drip" caption="Prefill reads the whole prompt in parallel; it sets time-to-first-token and grows with prompt length. Decode then emits one token at a time at a roughly fixed rate, so total latency is dominated by how much you ask the model to write." css={CSS} controls={<>
          <div className="btn-row">
            <button className="btn btn--accent" onClick={run}>
              {play.active ? "Restart" : "Run request"}
            </button>
            <button className="btn" onClick={reset} disabled={play.elapsed === 0}>
              Reset
            </button>
          </div>
          <Slider label="prompt" value={promptK} display={promptK + "K tok"} min={1} max={64} onChange={change(setPromptK)} />
          <Slider label="output" value={outTokens} display={outTokens + " tok"} min={20} max={800} step={20} onChange={change(setOutTokens)} />
          <Slider label="decode speed" value={tps} display={tps + " tok/s"} min={20} max={300} step={10} onChange={change(setTps)} />
        </>} readout={<span>
          <strong>{tokensDone}</strong> / {outTokens} tokens ·{" "}
          <strong>{(Math.min(play.elapsed, totalMs) / 1000).toFixed(2)}s</strong>
        </span>}>
      <div className="timing">
        <div>
          <div className="timing__lab">
            <span>one request, playback at {SPEEDUP}× speed</span>
            <span>{outTokens} output tokens</span>
          </div>
          <div className="timing__track">
            <div className="timing__prefill" style={{
    width: ttft / totalMs * 100 + "%"
  }}>
              prefill
            </div>
            {ticks}
            {play.active && <div className="timing__playhead" style={{
    left: Math.min(play.elapsed / totalMs * 100, 100) + "%"
  }} />}
          </div>
          <div className="timing__scale">
            <span>0 s</span>
            <span>{(totalMs / 2000).toFixed(1)} s</span>
            <span>{(totalMs / 1000).toFixed(1)} s</span>
          </div>
        </div>

        <div className="timing__metrics">
          <div className="timing__metric">
            <b>{Math.round(ttft)}</b>
            <span>ms to first token</span>
          </div>
          <div className="timing__metric">
            <b>{tps}</b>
            <span>tokens / second</span>
          </div>
          <div className="timing__metric">
            <b>{(totalMs / 1000).toFixed(1)}</b>
            <span>seconds total</span>
          </div>
          <div className="timing__metric">
            <b>{Math.round(ttft / totalMs * 100)}%</b>
            <span>of the wait is prefill</span>
          </div>
        </div>

        <Legend items={[{
    color: "#CAAEF5",
    label: "prefill: read the prompt (time to first token)"
  }, {
    color: "#FC4C02",
    label: "decode: one mark per " + (perTick > 1 ? perTick + " tokens" : "token")
  }]} />
      </div>
    </DiagramFrame>;
};

export const Legend = ({items}) => <ul className="legend">
    {items.map((it, i) => <li className="legend__item" key={i}>
        <span className="legend__sw" style={it.style || ({
  background: it.color
})} />
        {it.label}
      </li>)}
  </ul>;

export const Slider = ({label, value, display, min, max, step = 1, onChange}) => <label className="slider-group">
    <span className="slider-group__label">
      {label}
      <span className="slider-group__value">{display === undefined ? value : display}</span>
    </span>
    <input type="range" min={min} max={max} step={step} value={value} onChange={e => onChange(parseFloat(e.target.value))} />
  </label>;

export const DiagramFrame = ({eyebrow, title, caption, css, controls, readout, children}) => {
  const BASE_CSS = `
.learn-diagram {
  /* ── Together AI brand palette ── */
  --tg-orange: #FC4C02;
  --tg-orange-dark: #972E02;
  --tg-orange-soft: #FEE8DF;
  --tg-orange-tint: #FFDCCD;
  --tg-pink: #EF2CC1;
  --tg-pink-soft: #FDE3F6;
  --tg-purple: #A373ED;
  --tg-purple-mid: #CAAEF5;
  --tg-purple-soft: #EDE4FC;
  --tg-blue: #3D99F5;
  --tg-blue-mid: #A3D1FF;
  --tg-blue-soft: #E5F3FF;
  --tg-slate: #5E86AE;
  --tg-slate-soft: #EEF3F6;
  --tg-navy: #010120;

  /* ── Surfaces ── */
  --color-paper:   #FFFFFF;
  --color-paper-2: #F7F8FA;
  --color-paper-3: #F0F1F4;
  --color-paper-4: #E2E4E9;
  /* ── Ink ── */
  --color-ink:       #090909;
  --color-ink-soft:  #414B58;
  --color-ink-mute:  #626B84;
  --color-ink-faint: #98A0B3;
  /* ── Lines ── */
  --color-rule:        #C4C9D4;
  --color-rule-soft:   #E2E4E9;
  --color-rule-strong: #98A0B3;
  /* ── Accent ── */
  --color-accent:      var(--tg-orange);
  --color-accent-soft: var(--tg-orange-soft);
  --color-empty:       #F0F1F4;

  /* ── Type ── */
  --font-display: 'Jost', 'Helvetica Neue', Arial, sans-serif;
  --font-body:    'The Future', 'Jost', 'Helvetica Neue', Arial, sans-serif;
  --font-mono:    'The Future Mono', ui-monospace, 'SFMono-Regular', 'Courier New', monospace;

  /* ── Spacing (4px base) ── */
  --space-3xs: 4px;
  --space-2xs: 8px;
  --space-xs:  12px;
  --space-sm:  16px;
  --space-md:  24px;
  --space-lg:  32px;

  /* ── Radius ── */
  --radius-sm: 4px;
  --radius-md: 8px;
  --radius-lg: 12px;

  /* ── Motion ── */
  --ease-out: cubic-bezier(0.22, 1, 0.36, 1);
  --dur-fast: 120ms;
  --dur-base: 200ms;
  --dur-slow: 360ms;
  --focus-ring: 0 0 0 3px rgba(252, 76, 2, 0.35);

  font-family: var(--font-body);
  font-size: 15px;
  line-height: 1.65;
  font-weight: 400;
  color: var(--color-ink);
  -webkit-font-smoothing: antialiased;
  text-wrap: pretty;
  margin: 24px 0;
}
.learn-diagram, .learn-diagram * { box-sizing: border-box; }

/* ============ SHELL ============ */
.learn-diagram .diagram {
  margin: 0;
  background: var(--color-paper);
  border: 1px solid var(--color-rule);
  border-radius: var(--radius-lg);
  padding: var(--space-md);
  box-shadow: 0 1px 2px rgba(9, 9, 9, 0.04), 0 2px 10px rgba(9, 9, 9, 0.03);
}
.learn-diagram .diagram__head { margin-bottom: var(--space-md); }
.learn-diagram .diagram__eyebrow {
  font-family: var(--font-mono);
  font-size: 11px;
  font-weight: 500;
  letter-spacing: 0.1em;
  text-transform: uppercase;
  color: var(--tg-orange);
  margin-bottom: 6px;
}
.learn-diagram .diagram__title {
  font-family: var(--font-display);
  font-size: 20px;
  font-weight: 600;
  line-height: 1.3;
  letter-spacing: -0.02em;
  color: var(--color-ink);
  margin: 0;
}
.learn-diagram .diagram__caption {
  font-size: 13px;
  line-height: 1.55;
  color: var(--color-ink-mute);
  margin: 6px 0 0;
  max-width: 68ch;
}
.learn-diagram .diagram__controls {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-2xs) var(--space-md);
  align-items: center;
  margin-top: var(--space-md);
  padding-top: var(--space-sm);
  border-top: 1px solid var(--color-rule-soft);
}
.learn-diagram .diagram__readout {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.08em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  margin-left: auto;
}
.learn-diagram .diagram__readout strong {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
}

/* ============ BUTTONS ============ */
.learn-diagram .btn {
  font-family: var(--font-body);
  font-size: 13px;
  font-weight: 500;
  padding: 9px 14px;
  border: 1px solid var(--color-rule);
  border-radius: var(--radius-md);
  background: var(--color-paper);
  color: var(--color-ink);
  cursor: pointer;
  white-space: nowrap;
  transition: background var(--dur-fast) ease, border-color var(--dur-fast) ease,
              color var(--dur-fast) ease;
}
.learn-diagram .btn:hover:not([disabled]) { background: var(--color-paper-3); }
.learn-diagram .btn:focus-visible { outline: none; box-shadow: var(--focus-ring); }
.learn-diagram .btn[disabled] { opacity: 0.4; cursor: not-allowed; }
.learn-diagram .btn--accent {
  background: var(--tg-orange);
  border-color: var(--tg-orange);
  color: #fff;
}
.learn-diagram .btn--accent:hover:not([disabled]) {
  background: var(--tg-orange-dark);
  border-color: var(--tg-orange-dark);
}
.learn-diagram .btn--on {
  background: var(--tg-orange-soft);
  border-color: var(--tg-orange);
  color: var(--tg-orange-dark);
}
.learn-diagram .btn-row { display: flex; flex-wrap: wrap; gap: var(--space-2xs); flex: 0 0 auto; }

/* ============ SLIDERS ============ */
.learn-diagram .slider-group {
  display: flex;
  flex-direction: column;
  gap: 2px;
  min-width: 190px;
  flex: 1 1 190px;
}
.learn-diagram .slider-group__label {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.08em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  display: flex;
  justify-content: space-between;
  gap: var(--space-2xs);
}
.learn-diagram .slider-group__value {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
  text-transform: none;
}
.learn-diagram input[type="range"] {
  -webkit-appearance: none;
  appearance: none;
  width: 100%;
  height: 2px;
  background: var(--color-rule);
  border-radius: 2px;
  outline: none;
  cursor: pointer;
  margin: 8px 0 2px;
}
.learn-diagram input[type="range"]::-webkit-slider-thumb {
  -webkit-appearance: none;
  appearance: none;
  width: 14px;
  height: 14px;
  background: var(--tg-orange);
  border-radius: 50%;
  border: 2px solid var(--color-paper);
  box-shadow: 0 1px 3px rgba(9, 9, 9, 0.18);
}
.learn-diagram input[type="range"]::-moz-range-thumb {
  width: 14px;
  height: 14px;
  background: var(--tg-orange);
  border-radius: 50%;
  border: 2px solid var(--color-paper);
}
.learn-diagram input[type="range"]:focus-visible { box-shadow: var(--focus-ring); }

/* ============ PANELS · STATS · LEGEND ============ */
.learn-diagram .panel {
  background: var(--color-paper-2);
  border: 1px solid var(--color-rule-soft);
  border-radius: var(--radius-md);
  padding: var(--space-sm);
}
.learn-diagram .panel__head {
  font-family: var(--font-mono);
  font-size: 11px;
  font-weight: 500;
  letter-spacing: 0.1em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  margin-bottom: var(--space-2xs);
}
.learn-diagram .stats {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-3xs) var(--space-md);
  font-family: var(--font-mono);
  font-size: 12px;
  color: var(--color-ink-mute);
}
.learn-diagram .stats strong {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
}
.learn-diagram .legend {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-3xs) var(--space-md);
  list-style: none;
  margin: var(--space-sm) 0 0;
  padding: 0;
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.04em;
  color: var(--color-ink-soft);
}
.learn-diagram .legend__item { display: flex; align-items: center; gap: 6px; }
.learn-diagram .legend__sw {
  width: 12px;
  height: 12px;
  border-radius: 2px;
  flex: none;
  display: inline-block;
}
.learn-diagram .tag {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.06em;
  text-transform: uppercase;
  padding: 3px 8px;
  border-radius: var(--radius-sm);
  background: var(--color-paper-3);
  color: var(--color-ink-soft);
  white-space: nowrap;
}
.learn-diagram .tag--accent { background: var(--tg-orange-soft); color: var(--tg-orange-dark); }
.learn-diagram .tag--muted { background: transparent; border: 1px dashed var(--color-rule); color: var(--color-ink-faint); }
.learn-diagram .note {
  font-size: 13px;
  line-height: 1.55;
  color: var(--color-ink-mute);
}
.learn-diagram code, .learn-diagram .mono { font-family: var(--font-mono); font-size: 13px; }

@media (prefers-reduced-motion: reduce) {
  .learn-diagram *, .learn-diagram *::before, .learn-diagram *::after {
    transition-duration: 1ms !important;
    animation-duration: 1ms !important;
    animation-iteration-count: 1 !important;
  }
}
`;
  return <div className="learn-diagram">
    <style>{BASE_CSS}</style>
    {css ? <style>{css}</style> : null}
    <figure className="diagram">
      {(eyebrow || title || caption) && <div className="diagram__head">
          {eyebrow && <div className="diagram__eyebrow">{eyebrow}</div>}
          {title && <h3 className="diagram__title">{title}</h3>}
          {caption && <p className="diagram__caption">{caption}</p>}
        </div>}
      <div className="diagram__inner">{children}</div>
      {(controls || readout) && <div className="diagram__controls">
          {controls}
          {readout && <span className="diagram__readout">{readout}</span>}
        </div>}
    </figure>
  </div>;
};

**TL;DR:** There are two numbers that describe how fast an LLM feels in practice. The first is **TTFT** (time to first token), which is how long you wait between sending your request and seeing the first word of the response appear. The second is **TPS** (tokens per second), which is how fast each word appears after that. These two numbers are affected by different things, and you usually want to optimize them differently depending on the application.

## Why TTFT and TPS matter

The two most important metrics when it comes to LLM inference are time to first token (TTFT) and tokens per second (TPS). Other metrics such as tokens per minute (TPM) per GPU, time between tokens (TBT), and more can be understood as functions of these.

When you call an LLM, there is a pause before tokens stream back to you. This initial wait time is the TTFT: the pause between sending your request and the model showing you the first word of its reply.

Once the tokens start streaming, how fast they're generated is measured using TPS. This is the rate at which tokens stream in once the model has actually started generating.

You can have the same total time across two different combinations of TTFT and TPS and the experience will feel very different. A 5-second TTFT followed by 100 TPS feels sluggish to start and then snappy once it gets going. A 200ms TTFT followed by 20 TPS feels responsive at first and then laggy. Users perceive these two flavors of "slow" differently, which is why the system has to be tuned with both numbers in mind. Voice applications require a very low TTFT, while coding agent applications can have a slightly more relaxed TTFT but benefit from higher TPS.

<TimingDiagram />

## Two phases of inference: prefill and decode

Inference happens in two phases, and TTFT and TPS each correspond to one of them:

* **Prefill:** The model processes your entire prompt in one pass. The compute in this phase is parallel across positions, which means the prefill is essentially a single big matrix multiplication that the GPU is well-suited to handle. The cost of prefill scales with prompt length.
* **Decode:** The model generates output tokens one at a time, with each token depending on the previous one. The cost of decode scales with output length.

TTFT is mostly prefill latency plus a little overhead from the network and platform. TPS is mostly decode speed. The two phases share a model and a GPU, but the bottleneck for each is different:

* Prefill is **compute-bound**, meaning the GPU is multiplying matrices flat-out, and the limiting factor is how fast it can do the math.
* Decode is **memory-bound**, meaning the GPU spends most of its time reading model weights and the cached state from memory rather than actually multiplying.

For more on the forward-pass details that produce these two phases, see [How LLMs work](/learn/how-llms-work).

## What affects TTFT

A handful of things affect how long you wait for the first token:

* **Prompt length:** This is the single biggest factor. A 16K-token prompt takes meaningfully longer to prefill than a 1K-token prompt.
* **Model size:** A 70B-parameter model has more matmuls to do per token than an 8B-parameter model does. Larger models have higher TTFT on the same prompt, all other things being equal.
* **Prompt caching:** If you reuse the same system prompt or the same document across many calls, the platform can cache the prefill state and skip that work on a cache hit. When this happens, TTFT and compute cost drop to near-zero for the cached portion. On Together, prompt caching is enabled by default on [dedicated endpoints](/docs/dedicated-endpoints/settings).
* **Server load and queueing:** On shared serverless infrastructure, if the GPU is busy when your request arrives, you wait in the queue. Quiet times have lower TTFT than peak times for this reason.
* **Cold starts:** If a model has to be loaded fresh into GPU memory (rare on serverless, more common on dedicated endpoints that scale to zero), TTFT can be several seconds because the model has to be moved from disk into VRAM before any inference can happen. Consecutive calls thereafter should be much faster.

## What affects TPS

A different set of things affects how fast each token streams in:

* **Model size:** Bigger models have more weights to read per token. A 405B model has lower TPS than an 8B model, all other things being equal.
* **Quantization:** If you store the weights using fewer bits (fp8 instead of fp16, int4 instead of int8), the memory bandwidth needed to read them goes down. Lower memory bandwidth means higher TPS, sometimes by a meaningful amount. See [Quantization](/learn/quantization) for more on this.
* **Batching:** Servers run many requests at once to amortize the cost of reading model weights from memory. More requests in a batch means more total throughput across the GPU, but the per-request TPS can dip slightly because the GPU is doing more work per cycle.
* **Speculative decoding:** A small "draft" model guesses ahead and the big model verifies. When the guesses are right, you get 2 to 3 times the TPS for the same model. (This is mostly a server-side concern.)
* **Context length:** As the response grows, decoding gets a little bit slower per token, because the KV cache the model has to read from grows as well. The effect is usually small unless the output is very long.
* **Mixture-of-Experts (MoE):** MoE models split the feed-forward layers into many small "experts" and route each token through only a handful of them. That means the *active* parameters per token are a small fraction of the total. A 400B MoE with \~17B active params decodes closer to the speed of a 17B dense model than a 400B dense one. The total VRAM footprint still matches the full model (all experts have to be resident in memory in case they get routed to), but per-token TPS scales with active params rather than total params. Most modern frontier open models (DeepSeek-V3, Llama 4, Qwen3-Coder, Kimi K2) are MoE for exactly this reason.

## Next steps

<CardGroup cols={3}>
  <Card title="Context windows" icon="layout-board" href="/learn/context-windows">
    Why long inputs get slow before they get expensive.
  </Card>

  <Card title="Quantization" icon="zoom-in" href="/learn/quantization">
    The most-bang-for-your-buck TPS lever.
  </Card>

  <Card title="Choosing a deployment option" icon="server" href="/learn/choosing-a-deployment-option">
    When serverless variance is hurting your latency budget.
  </Card>
</CardGroup>