Mesh concurrency & federation (ack-and-hold · hold-through-reload · peers · Q4_K)

How gnosis-openai-mesh serves multiple requests and multiple models on scarce GPUs, and the exact ops to build, cut over, and federate it. Companion to MESH_BUILD_DEPLOY_RUNBOOK.md (build/deploy), MESH_QUALITY_REPORTING_FRAMEWORK.md (decode grades), MESH_DECODE_PERF_BUDGET.md (the numbers).

0. Serving topology

client ──> gnosis-openai-mesh (coordinator, Cloud Run L4, min-instances=1 warm)
              │  mesh-supervisor.ts  (model routing + swap + proxy)
              │     ├─ local model? → child openai-server.ts → fat-station (Rust, :8000)
              │     └─ a PEER owns it? → proxyToPeer → gnosis-mesh-peer-* (its own L4)

• Coordinator holds ONE model resident on its GPU at a time (the "swap lane"). It serves the default daily-driver (mistral-7b) + any model with no peer.
• Peers are full mesh instances (same image, scale-to-zero) that each own a model family, so different models run concurrently on different GPUs instead of thrash-swapping one GPU. See §4.
• The child openai-server runs ONE shared pipeline; concurrent requests to it are serialized by a promise-chain mutex (withPipelineMutex) — one GPU does one generation at a time. Concurrency within a model = a queue, not parallelism (§2).

1. Q4_K knot variants (size win, NOT a speed win)

Opt-in <model>-q4k registry entries (mesh-model-registry.ts) mirror the base model with knotUrl → .q4km.knot + env:{ MESH_GPU_Q4K: '1' }; Q8 stays the default. Built: mistral-7b-q4k, qwen3-8b-q4k, qwen3-4b-q4k, gemma3-4b-it-q4k.

• Decode is parity with Q8 — measured warm two-length (Δtokens÷Δelapsed_ms): qwen3-8b ~11/~11 tok/s, mistral 12.7/12.6. The Q4_K K-quant dequant costs back what the smaller bytes save. The win is on-disk size + cold-load + transfer (~1.7×: qwen3-8b 8300→4789 MB, qwen3-4b 4076→2375, gemma3-4b-it 3932→2368).
• Build a Q4_K knot from a genuinely-Q8 model only (content-length ÷ params: Q8 ≈1.06 B/param, Q4_K ≈0.56). Pipeline: cloudbuild-q4k-knots.yaml (HF Q4_K_M gguf → tsx encode → r2-mpu → verify); mirror next to the gguf with cloudbuild-knot-to-hf.yaml (_HF_REPO=forkjoin-ai/<model>-gguf, NOT a dump repo).
• Knots live in R2 distributed-inference/models/<id>.q4km.knot + HF forkjoin-ai/<model>-gguf beside the source gguf.

2. ack-and-hold (queued requests stay connected)

Problem: a request queued behind the pipeline mutex used to sit on a silent socket (headers were deferred until the first token) → clients time out → "the mesh won't take multiple requests."

Fix (openai-server.ts streamBareChatCompletion): flush SSE headers + an : accepted; queued position N comment immediately on accept, then a 10 s : keep-alive heartbeat while queued/prefilling, cleared on first content / in the finally. writeHeadersOnce is split into ensureHeaders (header flush) + the telemetry comment so the early flush doesn't swallow prefill-hit info. pipelineQueueDepth() exposes the queue position.

Verified: two concurrent streams → the queued one shows accepted; queued position 1 + heartbeats, then its full response; both complete. (Smoke: scripts/bench-mesh-tps.mjs is throughput; the ack-and-hold smoke fires 2 concurrent streams and greps the queued one for : accepted / : keep-alive.)

3. hold-through-reload (survive a model swap)

Problem: a model swap does ensureModel → killChild + spawnChild, so the child is momentarily unreachable; proxyToChild 502'd (mesh proxy error / fetch failed).

Fix (mesh-supervisor.ts proxyToChild, now async): retry the child connect on a transient error (ECONNREFUSED|ECONNRESET|EPIPE|socket hang up|ETIMEDOUT) with backoff — guarded: only while (a) no bytes have streamed yet (a streamed response can't be replayed) and (b) currentId is still the model this request was routed to (never retry across a swap → would serve the wrong model). Tunables MESH_PROXY_CONNECT_RETRIES (8) / MESH_PROXY_CONNECT_BACKOFF_MS (500).

Limit (by design): a concurrent different-model swap that pulls the model out from under an in-flight request 502s safely (no mis-route) rather than retrying — that case is a federation concern (§4), not a retry bug.

4. Federation (multi-model concurrency without thrash)

Different models can't share one GPU without swap-thrash. Federation puts each model family on its own scale-to-zero peer; the coordinator forwards by model instead of swapping.

Routing (mesh-supervisor.ts ~L518): on currentId !== requested, resolveFederatedPeerForModel(model, FEDERATION_PEERS) picks the peer whose models[] lists the model (or *), lowest priority then highest weight; if found → proxyToPeer. Empty MESH_FEDERATION_PEERS ⇒ behaviour identical to a lone node.

Config — MESH_FEDERATION_PEERS (JSON array, or compact CSV) on the coordinator:

[{"id":"peer-a","baseUrl":"https://gnosis-mesh-peer-a-…run.app",
  "models":["qwen3-8b","qwen3-8b-q4k","qwen3-4b","qwen3-4b-q4k"],
  "priority":0,"weight":1}]

Deploy a peer (a copy of the coordinator service; replicate its ~20 env vars):

gcloud run deploy gnosis-mesh-peer-a \
  --image …/moonshine:mesh-fast-v7 \
  --region us-central1 --args mesh --cpu-boost \
  --cpu 8 --memory 32Gi --gpu 1 --gpu-type nvidia-l4 \
  --no-gpu-zonal-redundancy --no-cpu-throttling \
  --min-instances 0 --max-instances 1 --concurrency 1000 --timeout 3600 \
  --allow-unauthenticated --set-env-vars "^@^KEY1=v1@KEY2=v2@…"

Then point the coordinator at it (§5 traffic note applies):

gcloud run services update gnosis-openai-mesh --region us-central1 \
  --update-env-vars "^@^MESH_FEDERATION_PEERS=$PEERS_JSON"

Verified live: alternating mistral (coordinator) / qwen3-8b-q4k (peer) after warm = all sub-second (0.64–1.1 s), zero reload; pre-federation each flip reloaded a multi-GB model (~minutes).

Gotchas (paid for in blood):

• --args mesh is REQUIRED on the peer (selects supervisor mode). Without it the container exits 1 with Permission denied (os error 13) and never binds :8080.
• L4 GPU quota nvidia_l4_gpu_allocation_no_zonal_redundancy (us-central1) gates total concurrent L4s — was 3. Each peer needs a slot. Free slots by deleting stale GPU services, or raise the quota (console → IAM & Admin → Quotas → "L4").
• The CPU-only gnosis-openai-<model> services (gpu=0, old baked images) are not viable fast peers.
• One model-family per peer; a peer that receives >1 thrash-prone model still swaps.

5. Build & cutover runbook (the safe path)

1. Build the image from the monorepo root: gcloud builds submit . --config open-source/gnosis/distributed-inference/cloudbuild-moonshine-mesh-fast.yaml (bump the mesh-fast-vN tag per build; .gcloudignore excludes firefox + the regenerable chunks/test-caches that else vanish mid-archive and crash the upload — the 2.8 GB context needs a stable window, it dies if the machine reboots mid-upload).
2. Canary: gcloud run deploy … --no-traffic --tag q4kvN → a 0 % revision with a direct https://q4kvN---…run.app URL.
3. Verify on the tag (serve a model + the relevant smoke) BEFORE routing.
4. Cut over: gcloud run services update-traffic … --to-revisions <rev>=100.
5. Keep the prior revision as rollback.

Traffic-pin gotcha: after an explicit update-traffic --to-revisions X=100, a later services update --update-env-vars makes a NEW revision but traffic STAYS pinned to X. You must update-traffic --to-revisions <new>=100 to activate it.

min-instances gotcha: --min-instances at the service template applies to the NEXT deploy's serving revision; you can't pin an already-running revision warm without redeploying its image (or cutting over).

6. Tunables (env)

7. Current live topology (2026-06-30)

• Coordinator gnosis-openai-mesh rev 00138-dch = mesh-fast-v7 + federation env, min-instances=1 warm. Carries Q4_K variants + ack-and-hold + hold-through- reload. Rollback chain: v7 00222-reg ← v6 00220-tox ← v5 00218-yey ← 00214-roy.
• Peer gnosis-mesh-peer-a = mesh-fast-v7, L4, min-instances=0 ($0 idle, ~3 min cold-start), owns the qwen-8B family.
• Source on GitHub: gnosis forkjoin-ai/gnosis (ack-and-hold + hold-through-reload).