Llama 3.1 70B

The canonical dense decoder: GQA, SwiGLU, RMSNorm, RoPE — the baseline every other diagram diverges from.

params=70Blayers=80ctx=128Kvocab=128,256

source: config.json ↗ · tech report ↗ snapshot 2026-08-24

attentionFFN / MoEnorm / residualembedding / headauxiliarydata flowresidualauxiliary

Reading this diagram

Data flows top to bottom. Solid edges carry activations (tensor shapes on the labels), dashed gray edges are residual paths, and dotted edges attach auxiliary modules. The layer stack draws one representative layer — the bracketed badge carries the ×80 multiplicity, and labeled variants show which layers differ.

Hover any box for a summary; click it for the details and a PyTorch reference implementation. Selections are shareable — the URL keeps them.

highlights: dense — every parameter active every token · GQA: 64 query heads share 8 KV heads · llama3 RoPE scaling to 128K context

embedding / head

Token embedding

Maps each of the 128,256 vocabulary ids to a learned 8192-dim vector — the model's only lookup table.

The embedding is a learned matrix W_E ∈ ℝ^{128256 × 8192}. A token id indexes a row, so the "computation" is a gather, not a matmul. The output is the initial residual stream every layer will read from and write to.

Subtleties worth knowing:

No positional information is added here in modern decoders — position enters inside attention via RoPE, not as an additive embedding (the original transformer added sinusoids at this point).
Scale: at d=8192 and V=128,256, this table alone is 1.05B parameters.
Weight tying is OFF in this model: the LM head has its own independent matrix, which tends to help at large scale at the cost of a second V×d table.

reference implementation

import torch
import torch.nn as nn

class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size=128256, d_model=8192):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)

    def forward(self, input_ids):          # [B, T] int64
        return self.embed(input_ids)       # [B, T, 8192]

structure

Transformer layer

One pre-norm residual block: normalize → attention → add, then normalize → FFN → add.

The repeating unit of the whole model. Each layer computes two deltas and adds them to the d=8192 residual stream:

h = x + Attention(Norm(x)) — tokens exchange information (the only place sequence positions interact);
y = h + FFN(Norm(h)) — each token is transformed independently.

Subtleties:

Pre-norm (norm on the branch, not the trunk) is what makes 80-layer stacks trainable without fragile warmup schedules — the identity path is never scaled or normalized.
Attention mixes across time, the FFN mixes across features; a transformer is just these two mixers alternated 80 times.
The diagram draws one representative layer — the ×80 badge carries the multiplicity. Where variants exist, the layer keeps this exact skeleton and only swaps the slot's contents:
- attention: all layers
- FFN: all layers

reference implementation

class DecoderLayer(nn.Module):
    def __init__(self, d_model=8192):
        super().__init__()
        self.norm1, self.norm2 = RMSNorm(d_model), RMSNorm(d_model)
        self.attn = Attention(...)
        self.ffn  = FFN(...)          # dense SwiGLU or MoE, per layer index

    def forward(self, x):
        x = x + self.attn(self.norm1(x))
        x = x + self.ffn(self.norm2(x))
        return x

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 8192 features (eps=0.00001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.00001) · g with a learned gain g ∈ ℝ^{8192}.

Compared to LayerNorm, RMSNorm drops mean subtraction and the bias term — it only controls the vector's scale, not its center. Empirically this loses nothing in quality and is faster (one reduction instead of two), which is why every model in this tool uses it.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.00001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=8192, eps=0.00001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 8192]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

attention

Grouped-Query Attention

64 query heads share 8 KV heads (d_h=128) — full attention quality with a 8× smaller KV cache.

Standard causal self-attention with grouped queries: the 64 query heads are split into 8 groups, and each group shares one K/V head. Per token the KV cache stores 8 × 128 keys and values instead of 64 × 128 — at inference, KV-cache size (not FLOPs) is what limits batch size and context, so this is nearly free quality-wise and hugely valuable.

The spectrum: MHA (kvHeads = heads, original), GQA (1 < kvHeads < heads), MQA (kvHeads = 1, maximum sharing, some quality cost).

In this model:

64 query heads, 8 KV heads, d_h=128
RoPE base θ=500000 with llama3 ×8 context extension
no Q/K normalization
no projection biases
no attention sinks

reference implementation

class GQA(nn.Module):
    def __init__(self, d_model=8192, n_heads=64,
                 n_kv_heads=8, head_dim=128):
        super().__init__()
        self.q = nn.Linear(d_model, n_heads * head_dim, bias=False)
        self.k = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.v = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.o = nn.Linear(n_heads * head_dim, d_model, bias=False)
        self.groups = n_heads // n_kv_heads

    def forward(self, x, pos):
        B, T, _ = x.shape
        q = split_heads(self.q(x), 64)        # [B, 64, T, 128]
        k = split_heads(self.k(x), 8)      # [B, 8, T, 128]
        v = split_heads(self.v(x), 8)
        q, k = apply_rope(q, pos), apply_rope(k, pos)
        k, v = repeat_kv(k, self.groups), repeat_kv(v, self.groups)
        out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        return self.o(merge_heads(out))               # [B, T, 8192]

norm / residual

Residual add

Adds the attention sublayer's output back onto the untouched input stream: y = x + f(norm(x)).

The residual connection is the transformer's gradient highway: every layer computes a delta f(norm(x)) and adds it to the stream, so the identity path from embedding to LM head is completely unobstructed.

Subtleties:

In pre-norm blocks (all models here), the add receives the raw stream x — normalization happened on the branch, not the trunk. This is why 60–90-layer models train without warmup tricks that post-norm (original transformer) needed.
The stream's magnitude grows roughly with depth since every layer adds to it; the final RMSNorm absorbs that before the head.
Residual adds are where activation dtype matters: the add is typically done in the trunk dtype (bf16) while branches may compute internally in other precisions.

reference implementation

# inside a decoder layer (pre-norm)
h = x + attention(norm1(x))     # ← this add
y = h + ffn(norm2(h))           # ffn sublayer

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 8192 features (eps=0.00001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.00001) · g with a learned gain g ∈ ℝ^{8192}.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.00001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=8192, eps=0.00001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 8192]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

FFN / MoE

SwiGLU MLP

Gated feed-forward: up-project to 28672, gate with SiLU, multiply, project back to 8192.

The per-token feed-forward network, applied identically at every position:

y = W_down · ( SiLU(W_gate·x) ⊙ (W_up·x) )

Three matrices instead of the original transformer's two: gate and up both map 8192 → 28672, their elementwise product is the "gated" hidden state, and down maps back 28672 → 8192. The gating lets the network modulate information flow multiplicatively, which beats plain ReLU/GELU MLPs at equal parameter count (to compensate for the third matrix, d_ff is usually ~⅔ of the "equivalent" 4·d).

Subtleties:

This is where roughly two thirds of a dense model's parameters live.
No activation clipping is configured — the standard unbounded SwiGLU.
MoE models keep exactly this structure but replicate it into many small experts and route tokens between them.

reference implementation

class SwiGLU(nn.Module):
    def __init__(self, d_model=8192, d_ff=28672):
        super().__init__()
        self.gate = nn.Linear(d_model, d_ff, bias=False)
        self.up   = nn.Linear(d_model, d_ff, bias=False)
        self.down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):                       # [B, T, 8192]
        h = F.silu(self.gate(x)) * self.up(x)   # [B, T, 28672]
        return self.down(h)                     # [B, T, 8192]

norm / residual

Residual add

Adds the ffn sublayer's output back onto the untouched input stream: y = x + f(norm(x)).

Subtleties:

In pre-norm blocks (all models here), the add receives the raw stream x — normalization happened on the branch, not the trunk. This is why 60–90-layer models train without warmup tricks that post-norm (original transformer) needed.
The stream's magnitude grows roughly with depth since every layer adds to it; the final RMSNorm absorbs that before the head.
Residual adds are where activation dtype matters: the add is typically done in the trunk dtype (bf16) while branches may compute internally in other precisions.

reference implementation

# inside a decoder layer (pre-norm)
h = x + attention(norm1(x))     # attention sublayer
y = h + ffn(norm2(h))           # ← this add

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 8192 features (eps=0.00001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.00001) · g with a learned gain g ∈ ℝ^{8192}.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.00001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=8192, eps=0.00001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 8192]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

embedding / head

LM head

Projects the final 8192-dim hidden state to 128,256 logits — one score per vocabulary entry.

A single linear layer ℝ^{8192} → ℝ^{128256} applied at every position. During training all T positions produce logits (next-token loss everywhere); during generation only the last position's logits are needed.

Subtleties:

Untied from the embedding in this model — separate matrices for reading tokens in and scoring tokens out, standard at this scale.
The softmax over 128,256 entries is usually computed in fp32 even when the trunk runs bf16 — logit precision matters for sampling quality.
This matmul is a significant fraction of per-token FLOPs at small batch sizes, which is one motivation for speculative decoding (verify many drafted tokens in one pass).

reference implementation

class LMHead(nn.Module):
    def __init__(self, d_model=8192, vocab_size=128256, tie_to=None):
        super().__init__()
        if tie_to is not None:                     # weight tying
            self.weight = tie_to.embed.weight      # [V, d]
        else:
            self.weight = nn.Parameter(torch.empty(vocab_size, d_model))

    def forward(self, h):                          # [B, T, 8192]
        return h @ self.weight.T                   # [B, T, 128256] logits