Twenty-two thousand, five hundred and eighty. That's how many GPT-2 models fit inside Kimi K3. But the interesting part of the last seven years isn't the multiplier — it's what had to change inside the architecture to make that scale usable.
This page rebuilds the whole lineage as interactive diagrams. Every mechanism is drawn, every playground runs real numbers on small tensors, and every architectural step exists because the previous one hit a wall: attention couples every token to every token; memory must be stored, updated, and evicted; routing spends capacity where it has a job; residuals carry information through depth.
Legend:SOURCEstated in the worklog / papersILLUSTRATIVEexplanatory simplification
The journey at a glance — click any stage to jump to its section:
01 / FOUNDATIONS
The GPT-2 starting point
GPT-2 (2019) is the baseline: a decoder-only Transformer that predicts the next token from everything before it. Every later architecture in this story is a modification of this exact skeleton — so we draw the whole pipeline first.
FIG 01GPT-2 FORWARD PASS · 124M PARAMS
CONFIG [SOURCE] — vocab 50304 (50257 padded to a multiple of 64) · 12 layers · 12 heads · n_embd 768 · MLP expansion 4× (768→3072→768). Click the block labeled ×12 to open one transformer layer.
The inefficiency that starts everythingDecoder-only generation computes representations for every input position, but each decode step consumes only the final position's logits. Without caching, that work is repeated for every new token — this is what the KV cache fixes in §04.
02 / FOUNDATIONS
Attention, visually
Each position asks a question (its query), scores every visible position's key, and mixes their values. The matrix below runs the real computation — hover any token to inspect its query, keys, scores, and the weighted values it receives.
FIG 02CAUSAL ATTENTION MATRIX
Hover a row to inspect a token; hover symbols in the equation to light up the matching part of the computation. Scores are seeded but the mask, scaling and softmax are computed live. Attention weights are shown as blue intensity.
The cost of attention is not one number. Training, prefill, decoding, and KV-cache memory each scale differently — and conflating them is how the O(N²) story gets muddy. Drag the sequence length and watch each cost separately.
FIG 03COST vs SEQUENCE LENGTH N
Left: the attention matrix as N grows (log-compressed display; true size below). Right: the four costs, stated separately. FlashAttention (2020) avoids materializing the N×N matrix and cuts memory IO — the N² score compute itself remains.
—score cells = N²
—matrix pixels @ 18px/cell
—screens wide (1512px)
The four separate costs [SOURCE]Training — full N×N score matrices were materialized before FlashAttention (2020). Prefill — the whole prompt is processed at once: O(N²) score computation per layer. Decode — each step reads the whole KV cache: 2·N·D reads + 2·D writes to HBM — memory-bandwidth bound, not FLOP bound. KV cache — grows O(N) and can become the memory-bandwidth bottleneck at long context.
04 / FOUNDATIONS
The KV cache
After appending a generated token, the model would otherwise recompute projections for all previous tokens. Storing their keys and values avoids that redundant work — that storage is the KV cache. Step through generation and watch it grow; then compare it with the constant-size state that linear attention introduces in §05.
FIG 04AUTOREGRESSIVE DECODE
Prefill processes the whole prompt in one pass; each decode step appends one K column and one V column. Per step, standard attention reads 2·N·D and writes 2·D to HBM, while the cache grows linearly, O(N) [SOURCE]. Continuation tokens are illustrative.
05 / LINEAR MEMORY
Linear attention — re-associating the product
Softmax applies its nonlinearity after the q·k product, coupling every query to every key. Linear attention applies a feature map to q and k separately — which makes the product re-associable: instead of building an N×N matrix, keys and values fold into a fixed d×d state.
FIG 05RE-ASSOCIATION: (QKᵀ)V → Q(KᵀV)
Press re-associate to watch the N×N matrix dissolve into a fixed d×d state. The math is exact for the re-association — the approximation happens earlier, in the feature map φ (see below).
Linear attention with feature map φ(x) = elu(x) + 1 [SOURCE]
Fixed d×d state per head — decode memory is O(1) per step, no growing KV cache. The same three-step attention contract: make scores non-negative, normalize, take a weighted average of values.
VS
What you give up
ELU+1 is a less expressive approximation of the softmax kernel — fidelity can drop depending on architecture and workload. And because the state only accumulates, associations will eventually interfere (§06).
FIG 05bKERNELS: SOFTMAX vs ELU+1
Both maps make scores positive; neither is exact. Softmax normalizes globally, ELU+1 leaves normalization to the denominator φ(q)·z.
06 / LINEAR MEMORY
DeltaNet — writing only the difference
A finite state that only accumulates eventually overflows. When N is much larger than the state dimension, "endlessly adding new associations to a memory of finite size… inevitably will reach a limit" (Schlag, Fast Weight Programmers). DeltaNet's fix: before writing, read what is already stored at that key, then write only the delta.
FIG 06DELTA-RULE STATE UPDATE · LIVE NUMBERS
State S is d×d (d = 4 here, k normalized to unit length, β fixed at 0.85 for clarity — β is learned per token in practice). Values are computed live. The read-back identity: writing kᵀ(βv) and reading with q = k returns β·‖k‖²·v = β·v.
Delta rule, re-parameterized for parallel chunks [SOURCE]
The delta rule can overwrite a memory only when it has a specific replacement; it cannot clear multiple associations during a context switch or decay memory to free capacity. Gated DeltaNet adds what Mamba-2 contributed: a decay gate α applied to the previous state before each delta write.
FIG 07GATED UPDATE: S ← α·S + kᵀu
Step through updates and watch old cells fade by α before the delta lands. At α = 1 the gate is off (pure DeltaNet); at α = 0 memory is cleared. Uniform decay forgets everything equally — which is exactly the limitation KDA's per-channel gates address in §08.
DeltaNet (α = 1)
VS
Gated DeltaNet (α = 0.75)
08 / KIMI'S ATTENTION
Kimi Delta Attention — the centerpiece
KDA is the core of Kimi Linear: a gated delta-rule linear attention with fine-grained, per-channel decay. Instead of one scalar α, it learns Diag(αₜ) — a separate decay value for each channel of the state. Toggle the pathways below to trace how a chunk of tokens flows through it.
FIG 08KDA MODULE · PATHWAYS
drag to pan · ctrl+scroll to zoom
Blue = intra-chunk attention (masked QKᵀ within a chunk of C tokens). Violet = inter-chunk recurrent state S. Orange = per-channel gates Diag(αₜ). Research mode adds tensor shapes and the exact update equations. Click a pathway chip to isolate it.
KDA state update — fine-grained diagonal gating [SOURCE: Kimi Linear, arXiv:2510.26692]
Reported results [SOURCE]With an identical training recipe, Kimi Linear (48B total / 3B active, hybrid 3:1 KDA:MLA) outperforms full MLA attention across evaluated tasks, cutting KV cache by up to 75% and reaching up to 6× decode throughput at 1M context.
09 / KIMI'S ATTENTION
Hybrid attention — why not one mechanism?
Full attention retrieves any token exactly but pays a growing KV cache; recurrent layers decode in O(1) but must compress. Kimi's answer is a hybrid stack: most layers run KDA, every fourth runs MLA. Build your own stack below and watch the memory profile change.
FIG 09LAYER STACK COMPOSER
Decode memory per layer at N = 8192, d_h = 128 [ILLUSTRATIVE] — click a layer to toggle its type.
Pattern used by Kimi Linear and Kimi K3: 3 KDA layers + 1 MLA layer per macrocycle [SOURCE]. The memory bar shows per-layer decode footprint: MLA grows with N; KDA stays fixed at d_h².
Global retrieval
MLA layers keep (compressed) keys and values for every past token — the model can still look anything up exactly, through the latent.
↔
Efficient recurrence
KDA layers carry a constant-size state through the whole sequence — O(1) decode, no cache growth, but information must be evicted to fit.
10 / KIMI'S ATTENTION
Multi-head Latent Attention — compressing the cache
At long context, the KV cache itself becomes the model. MLA compresses keys and values into a shared latent vector per token and up-projects them on use — trading exact storage for a much smaller cache.
FIG 10KV → LATENT COMPRESSION
Press compress: the K and V stacks collapse into one latent row cₜ per token, up-projected by WUK / WUV when attention needs them. In Kimi K3, MLA is further extended with query LoRA, output gating, and gated MLA [SOURCE].
d = 4096, d_c = 512 per token [ILLUSTRATIVE ratios]. Reported for Kimi Linear: KV cache reduced by up to 75%, up to 6× decode throughput at 1M context [SOURCE].
11 / SCALING & DEPTH
Mixture of Experts — capacity without compute
A learned router sends each token to a few experts, so total capacity can grow far beyond per-token compute. Kimi K3 has 898 experts: 2 shared (every token) + 896 routed, of which the router selects 16 per token [SOURCE].
FIG 11ROUTER → TOP-K EXPERTS (8 OF 896 SHOWN)
Each token's router scores are real dot products over a 6-dim feature; top-4 of 8 shown here (top-16 of 896 in K3) [SOURCE for counts, ILLUSTRATIVE for demo size]. Shared experts (S1, S2) receive every token.
Sparse routing — token x activates only its top-k experts
\[
y \;=\; \sum_{e \,\in\, \mathrm{top}\text{-}k}\;\htmlClass{kx-symGe}{g_e(x)}\;E_e(x)
\qquad
\text{K3: } k=16 \text{ of } 896 \text{ routed}
\]
2.8Ttotal parameters [SOURCE]
~104Bactive per token [SOURCE: reporting]
898experts · 2 shared · 16 routed/token
≈3.7%of capacity active per token
The router must also balance load across experts during training — otherwise a few experts absorb most tokens and the rest of the capacity is wasted. In K3 the experts also operate in a compressed latent space, which nearly halves their FLOPs [SOURCE].
12 / SCALING & DEPTH
Attention Residuals — attending over depth
In a standard residual stream, every layer's output is added with equal weight — so each layer's relative share shrinks as the network deepens, and later layers must learn ever-larger outputs to matter. AttnRes lets each layer retrieve from earlier representations instead of receiving one lossy sum.
FIG 12STANDARD RESIDUAL vs ATTENTION RESIDUAL
Left: equal weights — layer 8's share of the stream is 1/9 no matter what it computed. Right: a learned query scores each earlier block; softmax weights decide what the current layer reads. Press new query to see the weighting re-shuffle.
Each αᵢ is a query-key dot product over earlier residual states.
Kimi K3 blockwise AttnRes [SOURCE]Applied every 12 layers → 8 AttnRes blocks across 23 macrocycles. Adds roughly 2% inference latency; provides selective retrieval of earlier representations (mitigating residual dilution and hidden-state growth) and a 1.25× compute advantage in scaling-law comparisons.
13 / SCALING & DEPTH
The Kimi K3 architecture
Everything assembled: 23 four-layer macrocycles — three KDA layers plus one MLA layer each — latent MoE feed-forward, blockwise AttnRes every 12 layers, and a 1M-token context. Click any component to open its explanation.
2.8Ttotal params
23macrocycles (3×KDA + 1×MLA)
8AttnRes blocks (every 12 layers)
898experts · 16 active / token
1Mtoken context
~104Bactive params / token
FIG 13KIMI K3 · EXPANDABLE MAP
Component detail
Click any block in the map — KDA, MLA, MoE, AttnRes, the LM head — to read what it does and jump to its full section.
Structure per the source: layer 1 uses a dense FFN; every remaining layer uses latent MoE. KDA supplies constant-state recurrent memory; periodic MLA layers retain full softmax retrieval over the context; AttnRes retrieves from earlier depth-wise representations.
Engineering notes [SOURCE]SiTU replaces the SiLU activation in the experts (β·tanh(g/β)·sigmoid(g), with a learnable β); without a fused kernel it is almost 3× slower than the original path — offset by experts operating in compressed latent space, which nearly halves their FLOPs.
14 / SYNTHESIS
The architecture evolution map
The whole argument on one map. Each node is a wall hit and a fix purchased — with a price. This is the navigation spine of the site: expand any node, then jump to its section.
15 / SYNTHESIS
Complexity explorer
Asymptotics on one chart — and where they lie. Drag N and compare compute and memory for every mechanism; then see why the FLOP-optimal chunk size is not the wall-clock-optimal one.
FIG 15COMPUTE & MEMORY vs N (LOG–LOG)
Curves show asymptotic shapes with illustrative constants (d_h = 128, L = 32, d_c = 32, C = 64). Asymptotic complexity ≠ real-world GPU performance: decode is memory-bandwidth bound, and tensor cores prefer C = 64–128 even though C = 1 minimizes FLOPs.
FIG 15bCHUNK SIZE C — FLOPs vs GPU SWEET SPOT
Total chunked cost = fixed state work 2Ld² + growing within-chunk work 2LCd [SOURCE]. C = N recovers standard O(N²) attention; C = 1 is cheapest in FLOPs but maps poorly onto matrix-multiply hardware; C = 64–128 hits the tensor-core / UMMA sweet spot.
16 / SYNTHESIS
Prefill vs decode playground
Prefill processes the whole prompt at once; decode advances one token at a time against whatever memory the mechanism keeps. Switch mechanisms and phases to see how the same model behaves completely differently.
FIG 16INFERENCE MECHANISM VIEWER
17 / SYNTHESIS
Mathematical playground
No equation without its computation. Both playgrounds run real numbers on tiny tensors — step through, hover symbols, and watch each symbol light up the part of the pipeline it belongs to.
FIG 17STEP-THROUGH COMPUTATION
Hover any symbol in the equations above (§02, §06) to highlight the matching stage here. All numbers are computed live from a seeded 4-token, d_k = 4 example.
18 / SYNTHESIS
Before vs after — seven verdicts
Each comparison answers the same four questions: what changed, why, what got better, and what got harder.