Mesh bandwidth constraints (gnosis-openai-mesh, Cloud Run L4)

Decode speed is governed by bandwidth, not FLOPs. There are FOUR distinct bandwidth domains; each binds a different phase. Numbers measured 2026-06-21 on the live L4 service unless marked "spec". Knowing which domain binds which phase is the whole game — optimizing the wrong one (we did, repeatedly) wastes builds.

The four domains

Which domain binds which phase

• Cold start (per fresh Cloud Run instance): R2 download (7.7GB Q8 knot, domain 4) → then host dequant + PCIe upload to VRAM if GPU-resident (domain 2). The one-time fill: e.g. a [14336×4096] f16 weight logged dequant=99.9ms + htod=27.3ms (117MB ÷ 27.3ms = 4.3 GB/s PCIe). Full f16 residency fill ≈ 14GB ÷ 4.5 GB/s ≈ 50s+ of htod alone — acceptable once, catastrophic per token.
• Prefill (batched, the prompt): big GEMM, compute-leaning, already fast.
• Decode on CPU (prod today): bound by domain 3 (~82 GB/s system DRAM). ~7.55GB Q8 ÷ 82 GB/s ≈ 92ms/token ≈ 10 tps. NOTE the subtlety: lowering to Q4 did NOT help on CPU — at fewer bytes the scalar K-quant dequant compute becomes the new bottleneck, so CPU is at the DRAM/compute crossover, not purely bandwidth.
• Decode on GPU: bound by domain 1 (HBM). The weights live in VRAM; each token re-reads all of them through the SMs.

The decode-critical constraint: HBM (domain 1), in detail

A single-token GEMV reads the entire weight matrix once and does almost no compute (out[r] = Σ W[r,k]·x[k], one MAC per weight). So time ≈ weight_bytes ÷ HBM_BW. The L4's 300 GB/s is the ceiling; what we actually get:

• cuBLAS f16: ~183 GB/s (61% of peak). A 1-column GEMV (m×k · k×1) can't fill the L4 like a wide GEMM — limited occupancy + fixed per-launch cost. 61% is good for GEMV.
• Naive custom Q8 kernel: ~86 GB/s (29%). Memory-inefficient: for K=4096 rows only 128 Q8-blocks but 256 threads (half idle); byte-by-byte int8 loads; 34B-misaligned blocks; one tiny CUDA block per row. Reading half the bytes (Q8 vs f16) was erased by hitting only 29% of peak → net slower than cuBLAS.
• Optimized Q8 kernel (target): ≥250 GB/s (warp-per-row, float4 vectorized loads, no shared-mem reduce). For 15 tps on Q8 we only need 135 GB/s (see budget doc), so even a mediocre optimized kernel clears it; ~250 → ~20-25 tps.
• Why fusion/sync-batching gave 0%: they don't change bytes read from HBM. At GEMV size the per-call sync latency is a minority of the per-GEMV time; the weight read dominates. Only fewer bytes (lower-bit resident) or higher %-of-peak helps.

The PCIe corollary (domain 2): RESIDENCY IS MANDATORY

PCIe here is ~4-5 GB/s (virtualized Cloud Run, far below the ~25 GB/s bare-metal Gen4). Re-uploading weights per token = 7.55GB ÷ 4.5 GB/s ≈ 1.7s/token = 0.6 tps. This is exactly the failure modes we hit:

• f16 cache fragmenting at ~12GB (cudaMalloc) → overflow layers re-upload → 2.34 tps.
• The MESH_GPU_SLAB contiguous-slab fix exists PRECISELY to make ALL weights stay resident (one alloc, no eviction) so PCIe is paid ONCE at cold start, never in the decode loop. Any GPU decode path MUST keep 100% of weights resident or PCIe kills it.

System-DRAM corollary (domain 3): the 8-vCPU cap

CPU decode is ~82 GB/s on an 8-vCPU instance. Cloud Run GPU instances cap at 8 vCPU (Must be one of 1,2,4,6,8) — can't add cores/memory-channels — and 4≈8 threads already saturate it, so CPU decode is pinned near ~10 tps. AVX2 gave +3% (compute isn't the bottleneck). This domain has NO headroom left → the GPU (domain 1, 300 vs 82 GB/s ≈ 3.7×) is the only way up.

Virtualization tax (the easy-to-miss part)

Cloud Run's GPU is a shared/virtualized L4. Two domains are taxed vs bare metal:

• PCIe ~4-5 GB/s (vs ~25 spec) — makes residency non-negotiable.
• Per-call host↔device sync latency is elevated → favors FEWER, BIGGER device ops and resident state (no per-token host round-trips for weights). HBM (domain 1) and system DRAM (domain 3) are near their normal ceilings; the tax is on the crossings (PCIe + sync), not the on-device/on-host streams.

Bottom line for the 15 tps stretch

The binding constraint is HBM bandwidth × bytes-read. Levers, in order of leverage:

1. Fewer resident bytes (lower-bit in-kernel): Q8 7.55GB needs 135 GB/s for 15; Q4_K 4.0GB needs just 71. ← the real lever.
2. Higher %-of-peak (kernel quality: coalesced/vectorized loads, occupancy): 83→183→250+ GB/s. ← in flight (optimized Q8 kernel).
3. Residency (domain 2): mandatory enabler, not a speed lever itself — without it PCIe caps you at <1 tps. f16 cuBLAS is a dead end for 15 (needs 254 GB/s, hardware gives 183). See MESH_DECODE_PERF_BUDGET.md for the full tps × dtype requirement matrix.