> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# How LLMs work

> How a large language model produces text from a prompt, one token at a time.

export const AttentionMatrixDiagram = () => {
  const CSS = `
  .learn-diagram .attn {
    display: grid;
    grid-template-columns: minmax(0, 1fr) 240px;
    gap: var(--space-md);
    align-items: start;
  }
  @media (max-width: 760px) { .learn-diagram .attn { grid-template-columns: minmax(0, 1fr); } }
  .learn-diagram .attn__grid { min-width: 0; }
  .learn-diagram .attn__hdr, .learn-diagram .attn__row {
    display: grid;
    grid-template-columns: 86px repeat(8, minmax(0, 1fr));
    gap: 3px;
  }
  .learn-diagram .attn__hdr { margin-bottom: 6px; }
  .learn-diagram .attn__hdr-cell {
    font-family: var(--font-mono);
    font-size: 11px;
    color: var(--color-ink-mute);
    text-align: center;
    letter-spacing: -0.02em;
    overflow: hidden;
    text-overflow: ellipsis;
  }
  .learn-diagram .attn__hdr-cell.is-key { color: var(--tg-orange); }
  .learn-diagram .attn__axis {
    font-family: var(--font-mono);
    font-size: 10px;
    letter-spacing: 0.1em;
    text-transform: uppercase;
    color: var(--color-ink-faint);
    text-align: right;
    padding-right: 8px;
  }
  .learn-diagram .attn__row {
    margin-bottom: 3px;
    cursor: pointer;
    border-radius: var(--radius-sm);
  }
  .learn-diagram .attn__row:focus-visible { outline: none; box-shadow: var(--focus-ring); }
  .learn-diagram .attn__rowlab {
    font-family: var(--font-mono);
    font-size: 12px;
    color: var(--color-ink-soft);
    text-align: right;
    padding-right: 8px;
    align-self: center;
    overflow: hidden;
    text-overflow: ellipsis;
    white-space: nowrap;
  }
  .learn-diagram .attn__row.is-active .attn__rowlab { color: var(--tg-orange); font-weight: 500; }
  .learn-diagram .attn__cell {
    aspect-ratio: 1;
    min-height: 24px;
    background: var(--color-paper-3);
    border: 1px solid var(--color-rule-soft);
    border-radius: var(--radius-sm);
    font-family: var(--font-mono);
    font-size: 10px;
    font-variant-numeric: tabular-nums;
    display: flex;
    align-items: center;
    justify-content: center;
    color: transparent;
    transition: background var(--dur-base) var(--ease-out), color var(--dur-base) var(--ease-out);
  }
  .learn-diagram .attn__cell.is-masked {
    background: repeating-linear-gradient(45deg, #F7F8FA, #F7F8FA 3px, #EDEEF2 3px, #EDEEF2 6px);
    border-color: var(--color-rule-soft);
  }
  .learn-diagram .attn__row.is-active .attn__cell.is-attended { box-shadow: inset 0 0 0 1px var(--tg-orange-dark); }
  .learn-diagram .attn__panel-q {
    font-family: var(--font-mono);
    font-size: 15px;
    color: var(--color-ink);
    background: var(--color-paper);
    border: 1px solid var(--color-rule);
    border-radius: var(--radius-sm);
    padding: 4px 8px;
    display: inline-block;
    margin-bottom: var(--space-sm);
  }
  .learn-diagram .attn__wrow { margin-bottom: var(--space-2xs); }
  .learn-diagram .attn__wtop {
    display: flex;
    justify-content: space-between;
    font-family: var(--font-mono);
    font-size: 12px;
    color: var(--color-ink-soft);
    margin-bottom: 3px;
  }
  .learn-diagram .attn__wtop b { color: var(--tg-orange); font-weight: 500; font-variant-numeric: tabular-nums; }
  .learn-diagram .attn__wtrack { height: 4px; background: var(--color-paper-4); border-radius: 2px; overflow: hidden; }
  .learn-diagram .attn__wfill { height: 100%; background: var(--tg-orange); border-radius: 2px; transition: width var(--dur-base) var(--ease-out); }
  .learn-diagram .attn__foot {
    margin-top: var(--space-xs);
    padding-top: var(--space-2xs);
    border-top: 1px solid var(--color-rule-soft);
    font-family: var(--font-mono);
    font-size: 11px;
    color: var(--color-ink-faint);
  }
  `;
  const TOKENS = ["The", " cat", " sat", " on", " the", " mat", ".", "<eos>"];
  const WEIGHTS = [[1.0, 0, 0, 0, 0, 0, 0, 0], [0.2, 0.8, 0, 0, 0, 0, 0, 0], [0.1, 0.55, 0.35, 0, 0, 0, 0, 0], [0.05, 0.3, 0.45, 0.2, 0, 0, 0, 0], [0.04, 0.12, 0.16, 0.18, 0.5, 0, 0, 0], [0.03, 0.42, 0.18, 0.1, 0.22, 0.05, 0, 0], [0.02, 0.1, 0.06, 0.12, 0.08, 0.55, 0.07, 0], [0.01, 0.18, 0.08, 0.1, 0.05, 0.4, 0.1, 0.08]];
  const label = t => t.trim() === "" ? "␣" : t.trim();
  const [active, setActive] = useState(5);
  const prev = () => setActive(a => (a - 1 + TOKENS.length) % TOKENS.length);
  const next = () => setActive(a => (a + 1) % TOKENS.length);
  const ranked = WEIGHTS[active].map((v, i) => ({
    v,
    i
  })).filter(x => x.v > 0).sort((a, b) => b.v - a.v).slice(0, 3);
  const onKey = (e, q) => {
    if (e.key === "Enter" || e.key === " ") {
      e.preventDefault();
      setActive(q);
    }
  };
  return <DiagramFrame eyebrow="Attention" title="Every token only looks backwards" caption="Rows are queries, columns are keys. A cell shows how much of the row token's attention goes to the column token. The shaded upper triangle is masked out, so nothing can attend to a token that has not been generated yet." css={CSS} controls={<div className="btn-row">
          <button className="btn" onClick={prev}>
            Previous token
          </button>
          <button className="btn btn--accent" onClick={next}>
            Next token
          </button>
        </div>} readout={<span>
          query <strong>{active + 1}</strong> / {TOKENS.length} · click any row
        </span>}>
      <div className="attn">
        <div className="attn__grid">
          <div className="attn__hdr">
            <div className="attn__axis">keys →</div>
            {TOKENS.map((t, i) => <div className={"attn__hdr-cell" + (i <= active ? " is-key" : "")} key={i}>
                {label(t)}
              </div>)}
          </div>
          {TOKENS.map((t, q) => <div key={q} className={"attn__row" + (q === active ? " is-active" : "")} onClick={() => setActive(q)} onKeyDown={e => onKey(e, q)} role="button" tabIndex={0} aria-pressed={q === active} aria-label={"query token " + label(t)}>
              <div className="attn__rowlab">{label(t)}</div>
              {TOKENS.map((_, k) => {
    if (k > q) return <div className="attn__cell is-masked" key={k} />;
    const w = WEIGHTS[q][k];
    if (w <= 0.005) return <div className="attn__cell" key={k} />;
    return <div className="attn__cell is-attended" key={k} style={{
      background: "rgba(252, 76, 2, " + (0.12 + w * 0.88).toFixed(2) + ")",
      borderColor: "rgba(252, 76, 2, " + (0.3 + w * 0.7).toFixed(2) + ")",
      color: w > 0.4 ? "#fff" : "var(--color-ink-soft)"
    }}>
                    {Math.round(w * 100)}
                  </div>;
  })}
            </div>)}
          <Legend items={[{
    color: "rgba(252,76,2,0.95)",
    label: "high weight"
  }, {
    color: "rgba(252,76,2,0.25)",
    label: "low weight"
  }, {
    style: {
      background: "repeating-linear-gradient(45deg,#F7F8FA,#F7F8FA 3px,#EDEEF2 3px,#EDEEF2 6px)",
      border: "1px solid #E2E4E9"
    },
    label: "masked (future token)"
  }]} />
        </div>

        <div className="panel">
          <div className="panel__head">Query token</div>
          <div className="attn__panel-q">{label(TOKENS[active])}</div>
          {ranked.map(r => <div className="attn__wrow" key={r.i}>
              <div className="attn__wtop">
                <span>{label(TOKENS[r.i])}</span>
                <b>{Math.round(r.v * 100)}%</b>
              </div>
              <div className="attn__wtrack">
                <div className="attn__wfill" style={{
    width: r.v * 100 + "%"
  }} />
              </div>
            </div>)}
          <div className="attn__foot">
            top {ranked.length} of {active + 1} visible keys · row sums to 100%
          </div>
        </div>
      </div>
    </DiagramFrame>;
};

export const Legend = ({items}) => <ul className="legend">
    {items.map((it, i) => <li className="legend__item" key={i}>
        <span className="legend__sw" style={it.style || ({
  background: it.color
})} />
        {it.label}
      </li>)}
  </ul>;

export const Slider = ({label, value, display, min, max, step = 1, onChange}) => <label className="slider-group">
    <span className="slider-group__label">
      {label}
      <span className="slider-group__value">{display === undefined ? value : display}</span>
    </span>
    <input type="range" min={min} max={max} step={step} value={value} onChange={e => onChange(parseFloat(e.target.value))} />
  </label>;

export const DiagramFrame = ({eyebrow, title, caption, css, controls, readout, children}) => {
  const BASE_CSS = `
.learn-diagram {
  /* ── Together AI brand palette ── */
  --tg-orange: #FC4C02;
  --tg-orange-dark: #972E02;
  --tg-orange-soft: #FEE8DF;
  --tg-orange-tint: #FFDCCD;
  --tg-pink: #EF2CC1;
  --tg-pink-soft: #FDE3F6;
  --tg-purple: #A373ED;
  --tg-purple-mid: #CAAEF5;
  --tg-purple-soft: #EDE4FC;
  --tg-blue: #3D99F5;
  --tg-blue-mid: #A3D1FF;
  --tg-blue-soft: #E5F3FF;
  --tg-slate: #5E86AE;
  --tg-slate-soft: #EEF3F6;
  --tg-navy: #010120;

  /* ── Surfaces ── */
  --color-paper:   #FFFFFF;
  --color-paper-2: #F7F8FA;
  --color-paper-3: #F0F1F4;
  --color-paper-4: #E2E4E9;
  /* ── Ink ── */
  --color-ink:       #090909;
  --color-ink-soft:  #414B58;
  --color-ink-mute:  #626B84;
  --color-ink-faint: #98A0B3;
  /* ── Lines ── */
  --color-rule:        #C4C9D4;
  --color-rule-soft:   #E2E4E9;
  --color-rule-strong: #98A0B3;
  /* ── Accent ── */
  --color-accent:      var(--tg-orange);
  --color-accent-soft: var(--tg-orange-soft);
  --color-empty:       #F0F1F4;

  /* ── Type ── */
  --font-display: 'Jost', 'Helvetica Neue', Arial, sans-serif;
  --font-body:    'The Future', 'Jost', 'Helvetica Neue', Arial, sans-serif;
  --font-mono:    'The Future Mono', ui-monospace, 'SFMono-Regular', 'Courier New', monospace;

  /* ── Spacing (4px base) ── */
  --space-3xs: 4px;
  --space-2xs: 8px;
  --space-xs:  12px;
  --space-sm:  16px;
  --space-md:  24px;
  --space-lg:  32px;

  /* ── Radius ── */
  --radius-sm: 4px;
  --radius-md: 8px;
  --radius-lg: 12px;

  /* ── Motion ── */
  --ease-out: cubic-bezier(0.22, 1, 0.36, 1);
  --dur-fast: 120ms;
  --dur-base: 200ms;
  --dur-slow: 360ms;
  --focus-ring: 0 0 0 3px rgba(252, 76, 2, 0.35);

  font-family: var(--font-body);
  font-size: 15px;
  line-height: 1.65;
  font-weight: 400;
  color: var(--color-ink);
  -webkit-font-smoothing: antialiased;
  text-wrap: pretty;
  margin: 24px 0;
}
.learn-diagram, .learn-diagram * { box-sizing: border-box; }

/* ============ SHELL ============ */
.learn-diagram .diagram {
  margin: 0;
  background: var(--color-paper);
  border: 1px solid var(--color-rule);
  border-radius: var(--radius-lg);
  padding: var(--space-md);
  box-shadow: 0 1px 2px rgba(9, 9, 9, 0.04), 0 2px 10px rgba(9, 9, 9, 0.03);
}
.learn-diagram .diagram__head { margin-bottom: var(--space-md); }
.learn-diagram .diagram__eyebrow {
  font-family: var(--font-mono);
  font-size: 11px;
  font-weight: 500;
  letter-spacing: 0.1em;
  text-transform: uppercase;
  color: var(--tg-orange);
  margin-bottom: 6px;
}
.learn-diagram .diagram__title {
  font-family: var(--font-display);
  font-size: 20px;
  font-weight: 600;
  line-height: 1.3;
  letter-spacing: -0.02em;
  color: var(--color-ink);
  margin: 0;
}
.learn-diagram .diagram__caption {
  font-size: 13px;
  line-height: 1.55;
  color: var(--color-ink-mute);
  margin: 6px 0 0;
  max-width: 68ch;
}
.learn-diagram .diagram__controls {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-2xs) var(--space-md);
  align-items: center;
  margin-top: var(--space-md);
  padding-top: var(--space-sm);
  border-top: 1px solid var(--color-rule-soft);
}
.learn-diagram .diagram__readout {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.08em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  margin-left: auto;
}
.learn-diagram .diagram__readout strong {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
}

/* ============ BUTTONS ============ */
.learn-diagram .btn {
  font-family: var(--font-body);
  font-size: 13px;
  font-weight: 500;
  padding: 9px 14px;
  border: 1px solid var(--color-rule);
  border-radius: var(--radius-md);
  background: var(--color-paper);
  color: var(--color-ink);
  cursor: pointer;
  white-space: nowrap;
  transition: background var(--dur-fast) ease, border-color var(--dur-fast) ease,
              color var(--dur-fast) ease;
}
.learn-diagram .btn:hover:not([disabled]) { background: var(--color-paper-3); }
.learn-diagram .btn:focus-visible { outline: none; box-shadow: var(--focus-ring); }
.learn-diagram .btn[disabled] { opacity: 0.4; cursor: not-allowed; }
.learn-diagram .btn--accent {
  background: var(--tg-orange);
  border-color: var(--tg-orange);
  color: #fff;
}
.learn-diagram .btn--accent:hover:not([disabled]) {
  background: var(--tg-orange-dark);
  border-color: var(--tg-orange-dark);
}
.learn-diagram .btn--on {
  background: var(--tg-orange-soft);
  border-color: var(--tg-orange);
  color: var(--tg-orange-dark);
}
.learn-diagram .btn-row { display: flex; flex-wrap: wrap; gap: var(--space-2xs); flex: 0 0 auto; }

/* ============ SLIDERS ============ */
.learn-diagram .slider-group {
  display: flex;
  flex-direction: column;
  gap: 2px;
  min-width: 190px;
  flex: 1 1 190px;
}
.learn-diagram .slider-group__label {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.08em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  display: flex;
  justify-content: space-between;
  gap: var(--space-2xs);
}
.learn-diagram .slider-group__value {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
  text-transform: none;
}
.learn-diagram input[type="range"] {
  -webkit-appearance: none;
  appearance: none;
  width: 100%;
  height: 2px;
  background: var(--color-rule);
  border-radius: 2px;
  outline: none;
  cursor: pointer;
  margin: 8px 0 2px;
}
.learn-diagram input[type="range"]::-webkit-slider-thumb {
  -webkit-appearance: none;
  appearance: none;
  width: 14px;
  height: 14px;
  background: var(--tg-orange);
  border-radius: 50%;
  border: 2px solid var(--color-paper);
  box-shadow: 0 1px 3px rgba(9, 9, 9, 0.18);
}
.learn-diagram input[type="range"]::-moz-range-thumb {
  width: 14px;
  height: 14px;
  background: var(--tg-orange);
  border-radius: 50%;
  border: 2px solid var(--color-paper);
}
.learn-diagram input[type="range"]:focus-visible { box-shadow: var(--focus-ring); }

/* ============ PANELS · STATS · LEGEND ============ */
.learn-diagram .panel {
  background: var(--color-paper-2);
  border: 1px solid var(--color-rule-soft);
  border-radius: var(--radius-md);
  padding: var(--space-sm);
}
.learn-diagram .panel__head {
  font-family: var(--font-mono);
  font-size: 11px;
  font-weight: 500;
  letter-spacing: 0.1em;
  text-transform: uppercase;
  color: var(--color-ink-mute);
  margin-bottom: var(--space-2xs);
}
.learn-diagram .stats {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-3xs) var(--space-md);
  font-family: var(--font-mono);
  font-size: 12px;
  color: var(--color-ink-mute);
}
.learn-diagram .stats strong {
  color: var(--tg-orange);
  font-weight: 500;
  font-variant-numeric: tabular-nums;
}
.learn-diagram .legend {
  display: flex;
  flex-wrap: wrap;
  gap: var(--space-3xs) var(--space-md);
  list-style: none;
  margin: var(--space-sm) 0 0;
  padding: 0;
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.04em;
  color: var(--color-ink-soft);
}
.learn-diagram .legend__item { display: flex; align-items: center; gap: 6px; }
.learn-diagram .legend__sw {
  width: 12px;
  height: 12px;
  border-radius: 2px;
  flex: none;
  display: inline-block;
}
.learn-diagram .tag {
  font-family: var(--font-mono);
  font-size: 11px;
  letter-spacing: 0.06em;
  text-transform: uppercase;
  padding: 3px 8px;
  border-radius: var(--radius-sm);
  background: var(--color-paper-3);
  color: var(--color-ink-soft);
  white-space: nowrap;
}
.learn-diagram .tag--accent { background: var(--tg-orange-soft); color: var(--tg-orange-dark); }
.learn-diagram .tag--muted { background: transparent; border: 1px dashed var(--color-rule); color: var(--color-ink-faint); }
.learn-diagram .note {
  font-size: 13px;
  line-height: 1.55;
  color: var(--color-ink-mute);
}
.learn-diagram code, .learn-diagram .mono { font-family: var(--font-mono); font-size: 13px; }

@media (prefers-reduced-motion: reduce) {
  .learn-diagram *, .learn-diagram *::before, .learn-diagram *::after {
    transition-duration: 1ms !important;
    animation-duration: 1ms !important;
    animation-iteration-count: 1 !important;
  }
}
`;
  return <div className="learn-diagram">
    <style>{BASE_CSS}</style>
    {css ? <style>{css}</style> : null}
    <figure className="diagram">
      {(eyebrow || title || caption) && <div className="diagram__head">
          {eyebrow && <div className="diagram__eyebrow">{eyebrow}</div>}
          {title && <h3 className="diagram__title">{title}</h3>}
          {caption && <p className="diagram__caption">{caption}</p>}
        </div>}
      <div className="diagram__inner">{children}</div>
      {(controls || readout) && <div className="diagram__controls">
          {controls}
          {readout && <span className="diagram__readout">{readout}</span>}
        </div>}
    </figure>
  </div>;
};

**TL;DR:** An LLM is a series of calculations that, when applied to input text, produce output text. You give it some text, it turns the text into numbers, runs the numbers through a big stack of matrix multiplications, and gives you back a probability for every possible next word. Then it picks one and adds it to the end of the output. To produce a longer reply, it repeats this in a loop.

That's the whole top-level picture. Everything else on this page is detail on what makes this interesting: how text becomes numbers, what the matrix multiplications actually are, how to think about what a transformer does to input text, and what "training" a model actually means.

## What an LLM is trying to do

An LLM is, at its core, trying to build a statistical model of the language data it was trained on. The cleanest way to imagine what this means is to picture yourself in a familiar place:

<img src="https://mintcdn.com/togetherai-52386018/Yt4DEYDLSbELFeod/images/wheel-of-fortune.png?fit=max&auto=format&n=Yt4DEYDLSbELFeod&q=85&s=85f087ead82f4d5df8319ec108f86546" alt="A Wheel of Fortune puzzle board showing 'ANOTHER FEATHER _N YO_R _A_' under the category PHRASE." style={{ maxWidth: "500px", borderRadius: "10px", display: "block", margin: "16px auto" }} width="1024" height="576" data-path="images/wheel-of-fortune.png" />

Almost everyone fills in the missing letters instantly, spelling out: "another feather **in your cap**". But let's play devil's advocate for a second—why not "another feather **on your cat**"? It's a perfectly grammatical English sentence. Why does that completion not even occur to us?

Because you've heard the first phrase countless times, and never the second; because you know feathers go in caps and not on cats; because one has a known meaning and the other is nonsense. All of these factors contribute to the former being much more probable than the latter.

An LLM has direct access only to the first of these factors: raw frequency. The LLM has seen "feather in your cap" repeated in its training data orders of magnitude more often than "feather on your cat", and the probabilities it assigns to the next token reflect that. It does not *know* that feathers don't go on cats, but it was trained on enormous quantities of text written by humans who did, and the statistics it absorbed into its weights inherit that knowledge by proxy. The model approximates understanding by reproducing the statistical patterns of writers who actually had it.

## Calling an LLM API

Here is about the simplest call you can make on Together AI. We hand a text model the start of a sentence and ask it to continue, capped at a single token of output:

```python theme={null}
from together import Together

client = Together()

# Call the model
response = client.completions.create(
    model="Qwen/Qwen3.5-9B",  # Model ID
    prompt="The largest city in France is",  # Input text
    max_tokens=1,  # Maximum number of tokens to generate
)

print(response.choices[0].text)  # Print the output text
```

On the way in, your input text becomes a list of integers. A module called the **tokenizer** chops the prompt into *tokens*, chunks of text about four characters long, and looks up a fixed integer ID for each one. So `"The largest city in France is"` turns into something like:

```text theme={null}
[ 791, 7928, 3363, 304, 9822, 374 ]
```

This is a deterministic table lookup, not something the model learns at inference time. The model itself only ever maps integers to probabilities. It takes that list of indices and returns a probability for every possible *next* index:

```text theme={null}
input:  a list of integer indices  [ 791, 7928, 3363, 304, 9822, 374 ]
output: a probability for every possible next index
```

The output is a vector with one slot per token in the model's vocabulary (typically between 50,000 and 200,000 tokens). Each number in that vector represents the model's guess for how likely that token is to come next.

Because we asked for `max_tokens=1`, the inference engine takes the single most likely index, decodes it back to text, and hands you `" Paris"`. (In practice, you might need more than one token to reliably finish the sentence.)

To produce a longer reply, the inference engine repeats these steps in a loop:

1. Convert the current input into a list of integer indices (token IDs).
2. Run the model on the current input.
3. Look at the probabilities for the next token.
4. Pick the next token.
5. Add it to the end of the input.
6. Repeat until the model produces a stop signal or you hit a length limit.

The model itself is stateless—it doesn't remember anything between two separate questions unless you repeat yourself. Whatever it appears to "remember" is only because the inference loop keeps feeding the previous outputs back into the next call. The "memory" lives in the repeated turns of back-and-forth conversation, not in the model itself.

Let's break down the steps in more detail:

## Text becomes tokens, tokens become vectors

When the model receives your input text, it first converts it into a list of token IDs (integer indices). After that, it turns each ID into a **vector** or **embedding**: a list of floats, typically 1,024 to 16,384 long. The model has a learned lookup table that maps each token ID to a vector. That vector is the model's working representation of the token, or its "embedding".

Why is each token represented as a vector? Two reasons:

1. First, you can't do useful math on raw token IDs, ID 5279 and ID 5280 are arbitrary. With vectors, the model can express that "Paris" and "Lyon" are similar in some directions and different in others, literally by placing their vectors close together along the "city" axis and far apart along the "size" axis.
2. Second, the main operation throughout the model is a **matrix multiplication** (matmul): you multiply a vector by a **matrix** (a grid of numbers) to get a new, transformed vector, and that only works on vectors, not raw integer IDs. The numbers filling those matrices are the model's weights, so a "learned matrix" (a term that comes up later) just means one of these grids, with values found by training rather than set by hand.

## The transformer block

After the embedding step, every token is a vector, but that vector still only reflects the token in isolation. The embedding for "bank" is identical whether the sentence is "river bank" or "savings bank." The job of the transformer is to refine each vector until it captures what the token means *in this particular context*.

It does that by passing the vectors through *N* identical layers (blocks), where *N* is usually between 32 (small models) and 120+ (frontier models). Every block runs the same two steps:

1. **Attention:** Each token gathers information from the earlier tokens in the sequence and folds the relevant parts into its own vector. This is the only step where tokens exchange information.
2. **Feed-forward network (FFN/MLP):** Each token's vector is then processed on its own, with no reference to the others. This is where the model applies the knowledge stored in its weights to the now-contextualized vector.

A useful shorthand is "communicate, then compute": attention moves information *between* tokens, the feed-forward network does the heavy processing *within* each token. The next block repeats both steps on the result, and the next, and so on through all *N* layers.

The vector that flows down this stack is often called the **residual stream**, because each block doesn't overwrite it but *adds* a correction to it (a **residual connection**). So the representation of "bank" isn't rewritten from scratch at every layer; it accumulates. Early blocks tend to resolve local, surface-level structure (which word attaches to which, basic grammar), and later blocks build up the more abstract, meaning-level features the final prediction depends on. By the last block, the vector at each position is a context-aware summary of everything the model needs in order to predict what comes next.

## Attention

For each token, the attention step computes three things from the token's current vector by multiplying it with three different learned matrices:

* A **query (Q)**, "what am I looking for?"
* A **key (K)**, "what do I look like to others?"
* A **value (V)**, "what do I have to share?"

Then, for each token's query, the model measures how closely it matches every earlier token's key using a dot product. Those match scores are normalized into weights that sum to 1, so the closest matches get the most weight. The token then pulls in a weighted average of those tokens' values, and that blend is the update attention adds to its vector.

Three things to know about attention:

1. **Causal mask:** A token at position 5 can only look at positions 0-4. Never the future. That's what makes the model autoregressive: At training time, it can't peek at the next word it's supposed to predict, and at inference time, it can't look at tokens that haven't been generated yet.
2. **Multi-head:** This whole process runs in parallel many times (typically 32-96 "heads"), with different learned matrices each. Different heads end up specializing in different patterns—one might track the most recent noun, another might find matching brackets in code, another the subject of the current clause.
3. **Q, K, and V are learned matrices:** The interpretation of Q, K, and V above is a useful analogy for explaining what attention does, but in practice it's all matrix multiplication. The model figures out what to put in those matrices during training, purely from the process of trying to predict the next word.

Below is an example of what this might look like for the sentence "The cat sat on the mat." Each row represents one query position. Colored cells are the earlier tokens it pays the most attention to.

Click on any row to see the attention weights for that token:

<AttentionMatrixDiagram />

The pattern you see demonstrates what one head might learn during the attention step. A real model has many of these running in parallel per layer, each picking up something different.

## Feed forward network (FFN)

After attention has mixed information across tokens, the feed forward network (FFN) (AKA multilayer perceptron, or MLP) processes each token's vector on its own. It widens the vector (typically to 4× its size) by multiplying it with one matrix, applies a nonlinear function, then narrows it back down with another matrix.

This is where most of the parameters in the model actually live. By raw count, the feed forward network layers dwarf the attention layers. A useful way to think about it is that this is where the model stores all of its knowledge about the world. The widening step is like asking many questions about the token's current state in parallel, while the narrowing step writes back the answers.

Most of the model's "world knowledge"—what cities are capitals, which functions Python has, that one programming language uses curly braces and another uses indentation—is stored in the feed forward network weights. Nobody has a clean understanding of which weight stores what exactly, because the patterns are distributed across millions of neurons in ways no one fully understands. The entire field of **interpretability** is trying to answer exactly this question.

## Picking a token

After the last block, each position holds a vector that summarizes everything the model thinks up to that point. To turn this into a prediction for the next token, one final linear layer projects the vector back to the vocabulary, producing one number for each possible next token. These numbers are called **logits**.

Logits are raw scores, not probabilities, and they can be any real number, including negative ones. To turn them into probabilities, you apply a **softmax** operation: exponentiate each logit and normalize. This ensures that the probabilities are all positive and sum to 1 (which is a requirement for a valid probability distribution).

Then you pick / sample a token. The most basic choice is to pick the highest-probability token (greedy decoding), but several controls—temperature, top-k, and top-p—shape the distribution before sampling. See [inference parameters & sampling](/learn/inference-parameters-and-sampling) for more details.

## What training actually does

Everything covered above—the matmuls, the attention, the FFN / MLPs, the softmax—is fixed and constant. The model's behavior comes from the numbers inside those matrices. Those numbers are called **weights**. They start off random, and during the training process, the model searches for useful values for each weight. This process of taking the weights from random initializations to useful values is called **pretraining**, and it costs millions of dollars in compute, can take months to complete.

Pretraining works like this:

1. Take a document from the training data (anything from Wikipedia to GitHub to chat logs).
2. For every position in that document, ask the model what comes next.
3. Compare its prediction to the actual next token. The mismatch between the prediction and the actual next token is called the **loss**.
4. Compute how much each weight contributed to the loss.
5. Nudge each weight a tiny bit in the direction that would have reduced the loss. This is called **backpropagation**.

Repeat this process over trillions of token-positions (DeepSeek V4, for example, is trained on 32 trillion tokens) across most of the public internet. The model isn't memorizing the training set, it's adjusting its weights so the statistical patterns that show up across all those documents get reproduced when it generates.

Pretraining is usually followed by three more stages:

* **Instruction tuning:** Training the pretrained model on examples of "good behavior" (helpful answers, polite refusals, structured output) so it stops continuing text and starts responding to instructions.
* **Preference tuning:** Approaches like reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) run a feedback loop where pairs of "good" and "bad" outputs nudge the model toward outputs people actually want.
* **Reinforcement learning with verifiable rewards (RLVR):** Training on tasks whose answers can be checked automatically (math problems, code that passes tests, puzzles with a known solution), so the model gets rewarded for actually being right rather than for sounding right. This is the trick behind modern "reasoning" models that think out loud before answering.

By the end, what you have is a few billion to a few trillion weights. The architecture is primarily the same for all models (with small modifications here or there) and the intelligence is in those numbers. This is what you can download from Hugging Face for open-weight models.

This is also what people mean when they say a model is "just" matrix multiplications. The compute all happens through matmuls, and the behavior is defined by the values inside those matrices. The architecture is compact enough to fit in a few hundred lines of Python. The behavior is rich because every weight has been tuned by hundreds of thousands of GPU-hours of next-token prediction.

## Next steps

<CardGroup cols={3}>
  <Card title="Tokens & tokenization" icon="scissors" href="/learn/tokens-and-tokenization">
    What happens to the input text before the model sees it.
  </Card>

  <Card title="Context windows" icon="layout-board" href="/learn/context-windows">
    How much input the model can take in a single request (and why there's a hard limit).
  </Card>

  <Card title="Inference parameters & sampling" icon="adjustments" href="/learn/inference-parameters-and-sampling">
    Available controls for shaping the output.
  </Card>
</CardGroup>