LLM architectures, drawn to spec.

Interactive block diagrams built from each model's published config.json. Hover a module for the short version; click it for the subtleties and a PyTorch reference implementation.

Models

DeepSeek-V4-Flash-DSparkDeepSeek
1M-context MoE with sparse compressed attention, hyper-connections, and the DSpark speculative-decoding module.
params=284Bactive=13Blayers=43ctx=1M
view diagram →
Kimi K3Moonshot AI
2.8T-parameter natively-multimodal MoE: a 3:1 hybrid of KDA linear attention and NoPE gated MLA over a 896-expert LatentMoE.
params=2.8Tactive=104Blayers=93ctx=1M
view diagram →
DeepSeek-V3DeepSeek
The MLA + aux-loss-free MoE reference: 671B parameters, 37B active, with multi-token prediction.
params=671Bactive=37Blayers=61ctx=160K
view diagram →
GPT-OSS-120BOpenAI
OpenAI's open-weight MoE: alternating sliding/full attention with learned sinks, MXFP4 experts.
params=117Bactive=5.1Blayers=36ctx=128K
view diagram →
Qwen3-235B-A22BQwen
Fine-grained MoE with per-head QK-Norm and no shared expert — a deliberate contrast with DeepSeek's recipe.
params=235Bactive=22Blayers=94ctx=40K
view diagram →
Llama 3.1 70BMeta
The canonical dense decoder: GQA, SwiGLU, RMSNorm, RoPE — the baseline every other diagram diverges from.
params=70Blayers=80ctx=128K
view diagram →