Qwen3-235B-A22B

Fine-grained MoE with per-head QK-Norm and no shared expert — a deliberate contrast with DeepSeek's recipe.

params=235Bactive=22Blayers=94ctx=40Kvocab=151,936

source: config.json ↗ · tech report ↗ snapshot 2026-08-24

attentionFFN / MoEnorm / residualembedding / headauxiliarydata flowresidualauxiliary

Reading this diagram

Data flows top to bottom. Solid edges carry activations (tensor shapes on the labels), dashed gray edges are residual paths, and dotted edges attach auxiliary modules. The layer stack draws one representative layer — the bracketed badge carries the ×94 multiplicity, and labeled variants show which layers differ.

Hover any box for a summary; click it for the details and a PyTorch reference implementation. Selections are shareable — the URL keeps them.

highlights: GQA (64/4) with per-head Q/K RMSNorm · 128 experts, top-8, renormalized softmax gates · no shared expert

embedding / head

Token embedding

Maps each of the 151,936 vocabulary ids to a learned 4096-dim vector — the model's only lookup table.

The embedding is a learned matrix W_E ∈ ℝ^{151936 × 4096}. A token id indexes a row, so the "computation" is a gather, not a matmul. The output is the initial residual stream every layer will read from and write to.

Subtleties worth knowing:

No positional information is added here in modern decoders — position enters inside attention via RoPE, not as an additive embedding (the original transformer added sinusoids at this point).
Scale: at d=4096 and V=151,936, this table alone is 0.62B parameters.
Weight tying is OFF in this model: the LM head has its own independent matrix, which tends to help at large scale at the cost of a second V×d table.

reference implementation

import torch
import torch.nn as nn

class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size=151936, d_model=4096):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)

    def forward(self, input_ids):          # [B, T] int64
        return self.embed(input_ids)       # [B, T, 4096]

structure

Transformer layer

One pre-norm residual block: normalize → attention → add, then normalize → FFN → add.

The repeating unit of the whole model. Each layer computes two deltas and adds them to the d=4096 residual stream:

h = x + Attention(Norm(x)) — tokens exchange information (the only place sequence positions interact);
y = h + FFN(Norm(h)) — each token is transformed independently.

Subtleties:

Pre-norm (norm on the branch, not the trunk) is what makes 94-layer stacks trainable without fragile warmup schedules — the identity path is never scaled or normalized.
Attention mixes across time, the FFN mixes across features; a transformer is just these two mixers alternated 94 times.
The diagram draws one representative layer — the ×94 badge carries the multiplicity. Where variants exist, the layer keeps this exact skeleton and only swaps the slot's contents:
- attention: all layers
- FFN: all layers

reference implementation

class DecoderLayer(nn.Module):
    def __init__(self, d_model=4096):
        super().__init__()
        self.norm1, self.norm2 = RMSNorm(d_model), RMSNorm(d_model)
        self.attn = Attention(...)
        self.ffn  = FFN(...)          # dense SwiGLU or MoE, per layer index

    def forward(self, x):
        x = x + self.attn(self.norm1(x))
        x = x + self.ffn(self.norm2(x))
        return x

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 4096 features (eps=0.000001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.000001) · g with a learned gain g ∈ ℝ^{4096}.

Compared to LayerNorm, RMSNorm drops mean subtraction and the bias term — it only controls the vector's scale, not its center. Empirically this loses nothing in quality and is faster (one reduction instead of two), which is why every model in this tool uses it.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.000001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=4096, eps=0.000001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 4096]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

structure

Grouped-Query Attention

64 query heads share 4 KV heads (d_h=128) — full attention quality with a 16× smaller KV cache.

Standard causal self-attention with grouped queries: the 64 query heads are split into 4 groups, and each group shares one K/V head. Per token the KV cache stores 4 × 128 keys and values instead of 64 × 128 — at inference, KV-cache size (not FLOPs) is what limits batch size and context, so this is nearly free quality-wise and hugely valuable.

The spectrum: MHA (kvHeads = heads, original), GQA (1 < kvHeads < heads), MQA (kvHeads = 1, maximum sharing, some quality cost).

In this model:

64 query heads, 4 KV heads, d_h=128
RoPE base θ=1000000
per-head Q/K RMSNorm before RoPE (see the QK-Norm box)
no projection biases
no attention sinks

reference implementation

class GQA(nn.Module):
    def __init__(self, d_model=4096, n_heads=64,
                 n_kv_heads=4, head_dim=128):
        super().__init__()
        self.q = nn.Linear(d_model, n_heads * head_dim, bias=False)
        self.k = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.v = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.o = nn.Linear(n_heads * head_dim, d_model, bias=False)
        self.groups = n_heads // n_kv_heads

    def forward(self, x, pos):
        B, T, _ = x.shape
        q = split_heads(self.q(x), 64)        # [B, 64, T, 128]
        k = split_heads(self.k(x), 4)      # [B, 4, T, 128]
        v = split_heads(self.v(x), 4)
        q, k = self.qk_norm(q, k)                # per-head RMSNorm
        q, k = apply_rope(q, pos), apply_rope(k, pos)
        k, v = repeat_kv(k, self.groups), repeat_kv(v, self.groups)
        out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        return self.o(merge_heads(out))               # [B, T, 4096]

norm / residual

Q/K RMSNorm

Per-head RMSNorm applied to projected queries and keys (d_h=128) before RoPE — keeps attention logits bounded at scale.

After the Q/K projections split into heads, each head vector (d_h=128) is RMS-normalized (eps=0.000001) with its own learned gain, then rotated by RoPE.

Why it exists: attention logits are q·k/√d, and if a few heads' projections grow during training, their logits saturate softmax and destabilize training (the "attention entropy collapse" failure). Normalizing q and k caps the logit scale at roughly the learned gains' product, independent of how the projection weights drift.

Subtleties:

It is per-head (normalize each d_h=128 slice separately), not across the full concatenated width.
Order matters: norm before RoPE, so the rotation acts on unit-scale vectors.
Qwen3 adopted this in place of Qwen2's QKV bias — same goal (logit control), cleaner mechanism.

reference implementation

class QKNorm(nn.Module):
    """Applied inside attention, after head split, before RoPE."""
    def __init__(self, head_dim=128, eps=0.000001):
        super().__init__()
        self.q_norm = RMSNorm(head_dim, eps)
        self.k_norm = RMSNorm(head_dim, eps)

    def forward(self, q, k):    # [B, H, T, 128] each
        return self.q_norm(q), self.k_norm(k)

attention

Grouped-Query Attention

64 query heads share 4 KV heads (d_h=128) — full attention quality with a 16× smaller KV cache.

The spectrum: MHA (kvHeads = heads, original), GQA (1 < kvHeads < heads), MQA (kvHeads = 1, maximum sharing, some quality cost).

In this model:

64 query heads, 4 KV heads, d_h=128
RoPE base θ=1000000
per-head Q/K RMSNorm before RoPE (see the QK-Norm box)
no projection biases
no attention sinks

reference implementation

class GQA(nn.Module):
    def __init__(self, d_model=4096, n_heads=64,
                 n_kv_heads=4, head_dim=128):
        super().__init__()
        self.q = nn.Linear(d_model, n_heads * head_dim, bias=False)
        self.k = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.v = nn.Linear(d_model, n_kv_heads * head_dim, bias=False)
        self.o = nn.Linear(n_heads * head_dim, d_model, bias=False)
        self.groups = n_heads // n_kv_heads

    def forward(self, x, pos):
        B, T, _ = x.shape
        q = split_heads(self.q(x), 64)        # [B, 64, T, 128]
        k = split_heads(self.k(x), 4)      # [B, 4, T, 128]
        v = split_heads(self.v(x), 4)
        q, k = self.qk_norm(q, k)                # per-head RMSNorm
        q, k = apply_rope(q, pos), apply_rope(k, pos)
        k, v = repeat_kv(k, self.groups), repeat_kv(v, self.groups)
        out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        return self.o(merge_heads(out))               # [B, T, 4096]

norm / residual

Residual add

Adds the attention sublayer's output back onto the untouched input stream: y = x + f(norm(x)).

The residual connection is the transformer's gradient highway: every layer computes a delta f(norm(x)) and adds it to the stream, so the identity path from embedding to LM head is completely unobstructed.

Subtleties:

In pre-norm blocks (all models here), the add receives the raw stream x — normalization happened on the branch, not the trunk. This is why 60–90-layer models train without warmup tricks that post-norm (original transformer) needed.
The stream's magnitude grows roughly with depth since every layer adds to it; the final RMSNorm absorbs that before the head.
Residual adds are where activation dtype matters: the add is typically done in the trunk dtype (bf16) while branches may compute internally in other precisions.

reference implementation

# inside a decoder layer (pre-norm)
h = x + attention(norm1(x))     # ← this add
y = h + ffn(norm2(h))           # ffn sublayer

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 4096 features (eps=0.000001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.000001) · g with a learned gain g ∈ ℝ^{4096}.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.000001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=4096, eps=0.000001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 4096]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

structure

Mixture-of-Experts FFN

The FFN split into 128 small experts; each token runs through only 8 — most parameters idle per token.

MoE decouples parameter count from per-token compute: the dense FFN is replaced by 128 narrow SwiGLU experts (d=4096 → 1536), and a router sends each token to its top-8. Total capacity scales with expert count while FLOPs stay near a small dense model's — the mechanism behind "128-expert, 8-active" models.

Design axes visible in this tool's models:

Granularity: many small experts (d_expert=1536) rather than few big ones — finer routing granularity consistently wins at fixed compute.
Shared expert: absent — all capacity is routed (Qwen3/GPT-OSS style).
Load balancing: routers collapse onto favorite experts without countermeasures — see the router box for this model's approach.
Systems reality: experts shard across GPUs (expert parallelism); tokens are all-to-all'd to their experts and back, which is why balanced routing matters for wall-clock, not just quality.

reference implementation

class MoEFFN(nn.Module):
    def __init__(self, d_model=4096, n_experts=128, top_k=8):
        super().__init__()
        self.router = Router(d_model, n_experts, top_k)
        self.experts = nn.ModuleList(
            SwiGLU(d_model, 1536) for _ in range(n_experts))

    def forward(self, x):
        weights, idx = self.router(x)            # [N, 8] each
        out = torch.zeros_like(x)
        for slot in range(8):            # gather/scatter in real kernels
            e = idx[:, slot]
            out += weights[:, slot, None] * dispatch(self.experts, e, x)
        return out

FFN / MoE

MoE router

Scores all 128 experts per token (softmax) and dispatches to the top-8.

A linear layer ℝ^{d} → ℝ^{128} producing per-expert affinities, turned into a top-8 choice and combine weights. Small module, most consequential knobs in the architecture:

Scoring: softmax. Classic — affinities compete (weights sum to 1 across experts).
Top-k selection: standard top-k over scores.
No group restriction — any token may use any expert.
Renormalization ON: the selected top-k weights are rescaled to sum to 1.
The router usually runs in fp32 even in quantized models — tiny cost, and routing decisions are too consequential for low precision.

reference implementation

class Router(nn.Module):
    def __init__(self, d_model, n_experts=128, top_k=8):
        super().__init__()
        self.gate = nn.Linear(d_model, n_experts, bias=False)

    def forward(self, x):                               # [N, d]
        logits = self.gate(x.float())
        scores = logits.softmax(dim=-1)
        w, sel = scores.topk(8, dim=-1)
        w = w / w.sum(dim=-1, keepdim=True)             # renormalize
        return w, sel

FFN / MoE

Routed experts

128 independent SwiGLU MLPs (d=4096 → 1536); each token executes only its 8.

Each expert is a standard SwiGLU MLP, just narrow: d_ff=1536 vs the thousands a dense FFN would use. A token's routed output is the weighted sum of its 8 experts' outputs.

Subtleties:

Specialization is emergent: experts aren't assigned topics; clustering arises from routing pressure (and is famously hard to interpret cleanly).
Batched execution: kernels group all tokens per expert and run one GEMM per expert (grouped GEMM) — the loop in reference code is purely pedagogical.
Expert weights dominate total parameter count — they are the first target for quantization in serving.
Capacity note: with 128 experts × 8 active, the ratio of stored to exercised FFN parameters per token is 16:1.

reference implementation

self.experts = nn.ModuleList(
    SwiGLU(d_model=4096, d_ff=1536)   # narrow expert
    for _ in range(128)
)

# grouped-GEMM style dispatch (conceptual):
for e in selected_experts.unique():
    rows = (selected_experts == e).nonzero()
    out[rows] += gate_w[rows] * self.experts[e](x[rows])

FFN / MoE

Expert combine

Weighted sum of expert outputs.

The reduction that reassembles a token's result: y = Σ wᵢ·eᵢ(x).

Subtleties:

The weights wᵢ come from the router's scores renormalized over the selected top-k (they sum to 1, so the combine is a convex average).
In distributed serving this sum happens after the all-to-all returns expert outputs to their home devices — it's the merge point of the scatter/gather.

reference implementation

# w: [N, k] gate weights, e_out: [N, k, d] expert outputs
w = w / w.sum(-1, keepdim=True)      # convex combination
y = (w.unsqueeze(-1) * e_out).sum(1)  # [N, d]

norm / residual

Residual add

Adds the ffn sublayer's output back onto the untouched input stream: y = x + f(norm(x)).

Subtleties:

In pre-norm blocks (all models here), the add receives the raw stream x — normalization happened on the branch, not the trunk. This is why 60–90-layer models train without warmup tricks that post-norm (original transformer) needed.
The stream's magnitude grows roughly with depth since every layer adds to it; the final RMSNorm absorbs that before the head.
Residual adds are where activation dtype matters: the add is typically done in the trunk dtype (bf16) while branches may compute internally in other precisions.

reference implementation

# inside a decoder layer (pre-norm)
h = x + attention(norm1(x))     # attention sublayer
y = h + ffn(norm2(h))           # ← this add

norm / residual

RMSNorm

Rescales each token vector to unit RMS over its 4096 features (eps=0.000001), then applies a learned per-feature gain.

y = x / sqrt(mean(x²) + 0.000001) · g with a learned gain g ∈ ℝ^{4096}.

Subtleties:

Pre-norm placement: the norm sits before attention/FFN, not after. The residual stream itself is never normalized in place — that's what keeps very deep stacks trainable (the identity path stays untouched).
The reduction is usually done in fp32 even in bf16 models; eps=0.000001 guards the near-zero-norm case.
The final RMSNorm before the LM head is the only place the residual stream's growing magnitude gets tamed before scoring tokens.

reference implementation

class RMSNorm(nn.Module):
    def __init__(self, d_model=4096, eps=0.000001):
        super().__init__()
        self.g = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):                     # [B, T, 4096]
        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.g

embedding / head

LM head

Projects the final 4096-dim hidden state to 151,936 logits — one score per vocabulary entry.

A single linear layer ℝ^{4096} → ℝ^{151936} applied at every position. During training all T positions produce logits (next-token loss everywhere); during generation only the last position's logits are needed.

Subtleties:

Untied from the embedding in this model — separate matrices for reading tokens in and scoring tokens out, standard at this scale.
The softmax over 151,936 entries is usually computed in fp32 even when the trunk runs bf16 — logit precision matters for sampling quality.
This matmul is a significant fraction of per-token FLOPs at small batch sizes, which is one motivation for speculative decoding (verify many drafted tokens in one pass).

reference implementation

class LMHead(nn.Module):
    def __init__(self, d_model=4096, vocab_size=151936, tie_to=None):
        super().__init__()
        if tie_to is not None:                     # weight tying
            self.weight = tie_to.embed.weight      # [V, d]
        else:
            self.weight = nn.Parameter(torch.empty(vocab_size, d_model))

    def forward(self, h):                          # [B, T, 4096]
        return h @ self.weight.T                   # [B, T, 151936] logits