OTAKAR.AI
Both are, technically, "the Transformer." One is a 65M-parameter machine translation model that ran on 8 GPUs for half a day. The other is a 284B-parameter, MIT-licensed model serving 1M-token context at $0.22/M tokens. Same skeleton, everything else rebuilt. Here's where the delta actually lives, in numbers.
The honest comparison isn't total params — it's active compute per token, since V4 Flash only ever "uses" 4.6% of itself at once. See Section 3.
| Spec | Original Transformer (base) | DeepSeek V4 Flash |
|---|---|---|
| Total parameters | ~65M | 284B |
| Active parameters / token | 65M (100% — fully dense) | ~13B (4.6% of total) |
| Layers | 6 encoder + 6 decoder = 12 | 43, all MoE (no dense FFN layers) |
| Hidden dimension (d_model) | 512 | 4,096 |
| Attention heads | 8 (d_k = d_v = 64) | 64 query heads; 64 indexer heads |
| Feed-forward design | 1 dense FFN, d_ff = 2,048 | 256 routed experts + 1 shared expert per layer, d_ff = 2,048 each; top-6 routed activated |
| Vocabulary | ~37,000 (byte-pair) | 128,000 |
| Context window (practical) | ~512 tokens | 1,000,000 tokens native |
| Positional signal | Fixed sinusoidal, absolute | Learned indexer + compression-relative position (no fixed sinusoid) |
| Optimizer | Adam (β1=.9, β2=.98) | Muon (majority of weights) + AdamW (embeddings/head/norm) |
"Active parameters" is the number that governs latency and inference cost — it's the real point of comparison against a dense model, not total parameters.
Attention(Q,K,V) = softmax(QKᵀ/√d)V — every query scores every key. Cost scales as O(n²·d). At n=512 that's ~262K pairwise scores per head; nobody trained past a few thousand tokens because the cost curve makes it impractical.m=4 raw KV tokens into 1 entry, then a lightning indexer picks the top k=512 compressed entries per query for full attention; a 128-token sliding window covers local detail. HCA compresses even harder (m'=128) and runs dense attention over what's left. Cost approaches O(n·k) — flat instead of exploding.
Net effect for V4 Flash at 1M-token context: 10% of the single-token inference FLOPs and 7% of the KV-cache footprint of DeepSeek's own prior generation (V3.2) — a same-lab, same-architecture-family baseline, not the 2017 Transformer. (V4-Pro, the larger sibling model, posts 27% FLOPs / 10% KV-cache on the same comparison — a different model, not this one.)
| Dimension | Original Transformer (base) | DeepSeek V4 Flash |
|---|---|---|
| Training data | ~4.5M sentence pairs (WMT'14 En-De) | 32 trillion tokens — web, code, math, long documents, multilingual |
| Tokens processed | ~5B (100,000 steps × ~25K tokens/batch) | 32,000,000,000,000 |
| Hardware | 8× NVIDIA P100 GPUs | Multi-thousand-GPU cluster (undisclosed exact count) |
| Training time | 12 hours (base model) | Not disclosed publicly |
| Est. training compute | ~3.3×10¹⁸ FLOPs (paper's own figure) | ~2.5×10²⁴ FLOPs (6·N_active·D estimate) |
| License / distribution | Research paper, no released weights | Open weights, MIT license, Hugging Face + API |
| Inference pricing | N/A — not a served product | $0.22 / $0.66 per M tokens (in/out, off-peak) |
The compute estimate uses the standard 6·N·D approximation (6 × active params × training tokens) — a rough order-of-magnitude, not a disclosed figure. Even generously discounted, it lands north of 100 million times the original's training compute.
Total parameters is the wrong headline number for a MoE model. 284B total but only ~13B active means V4 Flash costs closer to a 13B dense model to run, and needs closer to a 284B model's worth of memory to host.
The attention mechanism from 2017 didn't survive intact. MLA, then CSA/HCA, replaced full dot-product attention specifically because O(n²) makes long context economically impossible, not just slow.
Almost every number moved in the same direction: bigger, sparser, and more compressed simultaneously. V4 Flash is both far larger than the original Transformer and, per active token, far more parameter-efficient than a dense model its total size would suggest.
What didn't change: it's still stacked self-attention + feed-forward blocks with residual connections, exactly as the 2017 paper specified. Every 2026 innovation here is an engineering layer added on top of, not a replacement for, that core idea.