LM-Head GKQ Router Validation
- model:
Qwen/Qwen2.5-0.5B-Instruct - vocab: 151936
- hidden: 896
- positions evaluated: 111
- prompts: 8
- ranks tested: [3, 32, 64, 128, 256]
- K_verify for bandwidth budget: 50
Recall table
| rank | recall@1 | recall@5 | recall@10 | recall@50 | recall@100 |
|---|---|---|---|---|---|
| 3 | 0.00% | 0.90% | 0.90% | 4.50% | 8.11% |
| 32 | 2.70% | 4.50% | 4.50% | 8.11% | 9.91% |
| 64 | 1.80% | 2.70% | 5.41% | 9.01% | 11.71% |
| 128 | 0.90% | 0.90% | 2.70% | 9.01% | 10.81% |
| 256 | 0.00% | 2.70% | 6.31% | 11.71% | 14.41% |
Bandwidth model (option 3 = GKQ-router + sparse verify)
- per-step fetch: K_verify × hidden × 4 bytes = 50 × 896 × 4 = 175.0 KB/step
- router-resident estimate ≈ rank × (vocab + hidden) × 1.5 bytes:
- rank 3: ≈ 671.6 KB
- rank 32: ≈ 7164.0 KB
- rank 64: ≈ 14328.0 KB
- rank 128: ≈ 28656.0 KB
- rank 256: ≈ 57312.0 KB
Verdict
ROUTER NOT VIABLE — recall too low at all tested ranks (best: rank=256 recall@50=11.71%, need ≥95%)
The hypothesis is falsified: rank-K SVD reconstruction of lm_head does NOT
preserve argmax ordering for Qwen2.5-0.5B-Instruct. Recall@50 plateaus near
~10% across ranks 32→256, and recall@1 is essentially noise (≤3%) at every
rank. Adding components does not help because the dominant singular vectors
of embed_tokens encode high-frequency statistical structure shared across
tokens, not the discriminative directions that separate the next-token argmax.
Worse: recall@K' is non-monotone in rank (rank 64 sometimes beats rank 128), strongly suggesting uint8 quantization noise dominates the bottom singular components rather than reconstruction approximation error. Even a full-rank (896) SVD with uint8 quantization would likely fail the threshold.
Pivot recommendation
Boot-amortize the lm_head. The dense lm_head weight is ~519 MB at fp32 / ~260 MB at fp16 / ~74 MB at Q4_K. At Q4_K it streams in seconds over any reasonable link, sits resident for the entire session, and serves exact argmax with zero per-step routing overhead. The "router + sparse verify" plan only ever made sense if (a) recall@K' was high AND (b) the resident sketch was small enough to dwarf the dense weight — neither holds here.
If boot-amortize is unacceptable for a target deployment, the next best candidate is a trained router (small dense classifier learned end-to-end to predict the dense top-K' candidates from the hidden state), not an SVD sketch. Cascaded vocab (frequency-binned dense fetches) is the worst option because it still pays per-step bandwidth and has no recall guarantee on rare-but-correct tokens.