Mesh quality reporting framework (decode grade: AAA → F)

A pipeline-wide quality framework: every stage grades itself against a measured budget and self-reports, so the parts that suck light up red and the parts that shine (after an upgrade) light up green — in aeon-forge over OTel. Single source of truth: mesh-decode-budget.json. Decode is the first instrumented stage; the pattern generalizes to any pipeline component with a measurable budget.

1. The grade scale (tok/s → letter; correctness is prerequisite)

1a. Quant ≠ decode grade — Q4_K is a SIZE win, not a tok/s win (measured 2026-06-29)

Q4_K and Q8 knots grade the same. Warm two-length decode (Δtokens ÷ Δelapsed_ms, which cancels prefill and is immune to buffered SSE) on gnosis-openai-mesh L4:

The Q4_K K-quant dequant cost gives back exactly what the smaller bytes save, so dropping Q8→Q4_K does NOT raise the decode grade. Q4_K's real win is on-disk size + cold-load + transfer (~1.7× smaller knots: qwen3-8b 8300→4789 MB, qwen3-4b 4076→2375, gemma3-4b-it 3932→2368). Opt-in <model>-q4k variants (knotUrl→.q4km.knot, MESH_GPU_Q4K=1) exist for storage/bandwidth-bound deploys, not throughput. So re-grading after a Q8→Q4_K swap is no change to this table; only the load-time/size budgets move. Caveat: measure WARM on a stable instance — a cold scale-to-zero L4 sweeping multiple multi-GB models thrashes (evict/reload, child restarts) and yields fake 0/divergent reads.

2. Three reporting surfaces (all driven by the one budget JSON)

(a) Post-hoc admission gate — admit-decode-profile.py [BUILT]

Zero runtime cost: grades a bench-decode-next profile and ADMIT/REJECTs a candidate revision before it earns prod. python3 admit-decode-profile.py prof.json --min-grade AA → exit 0/1. Prints the per-component where-the-time-goes + achieved GB/s + "to reach the next grade, shave N ms." Use it in the deploy loop (runbook §4) so a regression or a sub-grade candidate can't be promoted.

(b) In-mesh self-report (observation gate) — [TO BUILD, rides next build]

The fat-station decode loop already times every token (forward_us). Add a rolling window (e.g. last 32 decode tokens) → sustained tok/s → grade. ZERO-cost: one running mean + a compare + a throttled emit; no hot-path branching beyond that.

• Establishes a grade per loaded model (the warm steady-state grade).
• GRADE-DIP alarm: when the rolling tok/s drops below the floor needed to HOLD the established grade (e.g. was AA=15, dips < 15), emit a WARN mesh.decode.grade_dip with from/to grade + the offending component (the matvec stage that slowed). This is the "self-reports when it sucks."
• UPGRADE event: when rolling tok/s crosses UP a band (e.g. A→AA after a kernel ships), emit INFO mesh.decode.grade_up. This is the "shines when it upgrades."
• Flag-gated MESH_QUALITY_REPORT=1 (default OFF → identical behavior).

(c) aeon-forge OTel dashboard — [DESIGN; forge already speaks OTel]

apps/aeon-forge/src/telemetry/tracing.ts already exports OTLP (HTTP, via OTEL_EXPORTER_OTLP_ENDPOINT). The mesh emits the grade as OTel so forge renders a per-stage grade tile (green=shining/upgraded, amber=holding, red=dipped/F). Same OTLP backend the forge CLI uses → one pane shows which pipeline parts suck and which shine.

3. OTel metric/attribute schema (the contract between mesh → forge)

Emit one gauge + events per decode stage. Resource attrs: service.name=gnosis-openai-mesh, mesh.model=mistral-7b, mesh.revision=<rev>.

• gauge mesh.decode.tps (double) — sustained rolling tok/s.
• gauge mesh.decode.grade_ordinal (int: AAA=6,AA=5,A=4,B=3,C=2,D=1,F=0) + attr grade (string).
• gauge mesh.decode.component.gbs{component=ffn_matvec|qkv_matvec|o_matvec|logit_matvec} — achieved GB/s.
• gauge mesh.decode.component.ms{component=...} — per-component ms/token.
• counter/event mesh.decode.grade_dip{from,to,cause_component} and mesh.decode.grade_up{from,to}.
• hard: attr mesh.decode.divergent_tokens (int) — any >0 ⇒ grade F. forge maps grade_ordinal → tile color; grade_dip/grade_up → timeline annotations.

4. Why this generalizes ("parts self-report when they suck")

Any pipeline stage with (a) a measurable per-unit cost and (b) a budget can adopt the same shape: define its budget tiers in a *-budget.json, grade the live metric, emit grade/grade_dip/grade_up over the same OTel schema. forge then shows the WHOLE topology graded — a stage that regresses (slow kernel, cold cache, quota eviction) goes red and names itself; a stage that ships an upgrade goes green. The decode stage here is the reference implementation. Budget is data (mesh-decode-budget.json), so re-grading after a hardware/quant change is a JSON edit, not code.

5. Status / next

• ✅ grade scale + budget JSON + post-hoc report-card gate (built, tested).
• ⏳ in-mesh self-report (grade + dip/up + OTel emit) — additive, flagged; build with the next kernel iteration.
• ⏳ aeon-forge tile/timeline for mesh.decode.* — consumes the schema in §3. See MESH_DECODE_PERF_BUDGET.md (the numbers), MESH_BANDWIDTH_CONSTRAINTS.md (why), MESH_BUILD_DEPLOY_RUNBOOK.md (how to ship).