Whisper-tiny Compression Methods Benchmark

Date: 2026-05-24
Script: bench_compression_methods.py
Model: whisper-tiny-hf-root/model.safetensors (fp32, 67 2D matrices, 37.2M weights)
Runtime: 518s on Apple Silicon
Prior finding: GKQ/SVD fails on Whisper (high-rank spectra). See GKQ_WHISPER_HIGH_RANK_FINDING.md.

Question

Is there a high-rank-friendly compression that beats q4 (4.5 bpw) for Whisper without crossing into unsafe reconstruction error? Or is bits-per-weight the entropy floor?

Method Summary

All methods are rank-agnostic (exploit value structure, not singular vectors):

method class	variants	bpw range
k-means scalar codebook	b=2,3,4 bits	2.0 – 4.0
Product quantisation (PQ)	d=2,4 subvector dim, 256-entry codebook	3.4 – 4.3
Entropy floor	Shannon entropy of q8/q4 index streams	3.9 / 6.8
Actual entropy coding	zlib/deflate of packed q4 nibbles	3.9
Outlier-aware low-bit	top 0.5/1/2% fp16, rest q3 or q4	4.6 – 5.7
Baselines	fp32, q8 per-row, q4 block-32	4.5 / 8.1 / 32

Metrics: mean relative Frobenius error ||W-W'||/||W|| and p95, weighted by n_weights.
Safety threshold: mean err < 2% (interpolated — q8 at 1.07% is WER 0.00 known safe; GKQ at 64.8% is garbage; 2% is conservative; see caveat below).

Results Table

Sorted by bits/weight ascending:

Method	bits/w	mean rel-err	p95 rel-err	SAFE?	notes
kmeans_b2	2.000	38.853%	48.553%	NO
kmeans_b3	3.000	21.745%	29.005%	NO
pq_d4	3.397	20.873%	34.015%	NO
q4_zlib	3.885	10.864%	12.524%	NO	actual zlib/deflate on packed nibbles
entropy_q4	3.901	10.864%	12.524%	NO	Shannon floor (theory min)
kmeans_b4	4.001	12.175%	18.312%	NO
pq_d2	4.297	11.306%	14.155%	NO
q4_block32	4.500	10.864%	12.524%	NO	baseline
outlier_0.5pct_q3	4.565	20.050%	21.219%	NO	hurts: q3 base is too lossy
outlier_1.0pct_q3	4.627	18.999%	19.548%	NO
outlier_2.0pct_q3	4.753	17.343%	17.675%	NO
outlier_0.5pct_q4	5.560	9.377%	9.937%	NO	slightly better err, more bits
outlier_1.0pct_q4	5.617	8.883%	9.148%	NO
outlier_2.0pct_q4	5.732	8.101%	8.275%	NO
entropy_q8	6.781	1.071%	1.542%	YES	Shannon floor for q8
q8_per_row	8.075	1.071%	1.542%	YES	proven WER 0.00
fp32	32.000	0.000%	0.000%	YES

Entropy Analysis

q4 raw bits/weight:                4.500
q4 Shannon entropy floor (bpw):    3.901   (theory min via Huffman/ANS on index stream)
q4 actual zlib bpw:                3.885   (zlib beats per-symbol entropy floor because
                                            it exploits cross-block positional correlations)
Gap q4_raw -> theory floor:        0.599 bpw
Gap q4_raw -> zlib actual:         0.615 bpw

The q4 index distribution is NOT uniform — it is approximately Gaussian-shaped (bell curve: indices near 8 are most frequent, tails sparse). Shannon entropy ~3.6 bits/sym vs raw 4 bits/sym → ~0.4 bits/sym gain with entropy coding per tensor. The aggregate gap is ~0.60 bpw including fp16 scale overhead.

Per-tensor spot-check confirms zlib is ~0.04 bpw ABOVE Shannon entropy (as expected — zlib has stream overhead). The aggregate table shows zlib slightly below Shannon floor due to weighting artifacts from embed_tokens (20M weights, low entropy, dominates mean). Correct interpretation: zlib ≈ Shannon floor; neither beats it.

Entropy coding on top of q4 saves ~0.6 bpw (to ~3.9 bpw), with error unchanged at 10.9%. This is lossless but requires an ANS/Huffman decoder at runtime.

VERDICT

No method achieves mean rel-err < 2% at fewer bits/weight than q4.

The safe boundary is at ~6.8 bpw (q8 Shannon floor) → 8.1 bpw (q8 raw), with 1.07% error. Everything below 4.5 bpw crosses into 10%+ error territory. No exotic codebook structure (k-means, PQ, outlier-aware) escapes this — the error floor at sub-q4 rates is structurally determined by the high-rank nature of the weights (same root cause as GKQ failure).

Is bits-per-weight the entropy floor?

Mostly yes, with one caveat: entropy coding of q4 can reach ~3.9 bpw (saving 0.6 bpw) without additional reconstruction error. This is not "free" — it requires an ANS or Huffman coder at decode time — but it is lossless on top of q4. Whether that 0.6 bpw saving justifies the decoder complexity is a separate question.

Outlier-aware q4 slightly reduces error (9.4% vs 10.9% at comparable bits, 5.6 bpw) but at the cost of more bits, not fewer. It does not break the floor.

Outlier-aware q3 is strictly worse than q4 across all tested fractions — protecting a few fp16 outliers does not compensate for the increased base error of q3.

Safety Threshold Caveat

The 2% mean Frobenius threshold is conservative. All lossy quantisation below q8 reports

10% mean error by this metric. However:

Whisper-tiny in GGUF q4 format is empirically near-lossless in published WER tests.
Frobenius error penalises all weights equally; WER only cares about errors that shift the argmax of the logit distribution.
q4 at 10.9% Frobenius error may be perfectly safe in practice — or not. The Frobenius number cannot tell you. Only a WER gate can.

Implication for Knot Strategy

The search for a compression scheme that beats q4 on Whisper is closed:

SVD/GKQ: dead (high-rank spectra, 64.8% error at 10 MB — see prior finding).
k-means scalar / PQ: dead (10–38% error below 4.5 bpw, worse than q4).
Outlier-aware: does not reduce bits vs q4; only marginally reduces error at higher bpw.
Entropy coding of q4: saves 0.6 bpw (to ~3.9 bpw), same error as q4. Worth considering only if 0.6 bpw matters for the target deployment (browser WASM, mesh transfer).

q4_block32 at 4.5 bpw is the minimum-bits representation whose WER is not yet known to be catastrophic. The next and only remaining question is WER, not bpw.

Next Exploration (NE-COMP-1)

Run q4_block32 through the voice-stt gate for real WER on whisper-tiny. If WER is acceptable, q4 is the answer — invest no further in compression search.

Secondary: if entropy coding of q4 (ANS/Huffman, ~3.9 bpw) is needed for size targets, validate that the decoder overhead is acceptable in the WASM/knot runtime, then gate WER.

Reproduce

cd open-source/gnosis/distributed-inference
python3 bench_compression_methods.py

forkjoin-ai/gnosis