forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Mesh (`gnosis-openai-mesh`) build / deploy / profile runbook

distributed-inference-host/MESH_BUILD_DEPLOY_RUNBOOK.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Mesh (gnosis-openai-mesh) build / deploy / profile runbook

Known-good invocation, validated 2026-06-21. All commands run from the monorepo root (/Users/buley/Documents/Code/monorepo). Project neutral-418500, region us-central1, service gnosis-openai-mesh, image us-central1-docker.pkg.dev/neutral-418500/gnosis/moonshine:mesh-v1, public URL https://gnosis-openai-mesh-366749842679.us-central1.run.app.

The build needs NO GPU (Cloud Build E2_HIGHCPU_32, nvcc in the cuda:12.4-devel rust-builder). Only the deploy needs a transient GPU slot. ~10 min/build. CUDA (#[cfg(feature="cuda")]) code is NOT checkable by local cargo check (no toolkit on the arm Mac) — it only compiles in the cloud build. See memory moonshine-mesh-build-file-wiring.

0. Before you build: wiring new files

Any NEW source file the build needs must be added to BOTH (else silently absent → confusing failures like error: environment variable OUT_DIR not defined):

  1. cloudbuild.moonshine-mesh.gcloudignore — add !<path> (whitelist-style ignore; it only un-ignores distributed-inference/{Cargo.toml,build.rs,src/**,examples/**}).
  2. Root Dockerfile.moonshine — add a COPY line (rust-builder stage, ~line 41-45, copies Cargo.toml/build.rs/src/examples individually). Custom CUDA kernels: compile to CUBIN (nvcc --cubin -arch=sm_89), NOT PTX — the L4 runtime driver rejects CUDA-12.4 PTX (CUDA_ERROR_UNSUPPORTED_PTX_VERSION); cubin (native sm_89 SASS) loads via cuModuleLoadData with no PTX-version JIT.

1. Build (→ new image digest)

cd /Users/buley/Documents/Code/monorepo
gcloud builds submit --project=neutral-418500 \
  --config=cloudbuild.moonshine-mesh.yaml \
  --ignore-file=cloudbuild.moonshine-mesh.gcloudignore .
# get the result + digest:
gcloud builds list --project=neutral-418500 --limit=1 \
  --format='value(id,status,results.images[0].digest)'
# (optional) confirm a cuda kernel compiled:
gcloud builds log <BUILD_ID> --project=neutral-418500 | grep -iE 'compiled to CUBIN|error'

2. GPU quota gate (MANDATORY before deploy)

mesh + flux + ltx = the full 3-GPU L4 quota; a rollout transiently needs a 2nd mesh slot, so flux AND ltx must be idle (scaled to 0). Verify:

START="$(python3 -c 'import datetime;print((datetime.datetime.now(datetime.UTC)-datetime.timedelta(minutes=10)).strftime("%Y-%m-%dT%H:%M:%SZ"))')"
for svc in flux-render-gpu ltx-render-gpu; do
  v=$(gcloud monitoring time-series list --project=neutral-418500 \
    --filter="metric.type=\"run.googleapis.com/container/instance_count\" AND resource.labels.service_name=\"$svc\"" \
    --interval-start-time="$START" --format=json 2>/dev/null | jq -r '[.[].points[].value.doubleValue]|max // 0')
  echo "$svc: ${v:-0}"   # both must be 0
done

3. Deploy (pin digest; --no-traffic, then route)

Keep maxScale=1, minScale=0 (do NOT raise maxScale — quota). --update-env-vars adds/changes only the listed keys; use --remove-env-vars to drop experimental ones.

gcloud run services update gnosis-openai-mesh --project=neutral-418500 --region=us-central1 \
  --image=us-central1-docker.pkg.dev/neutral-418500/gnosis/moonshine@sha256:<DIGEST> \
  --update-env-vars=MESH_BUILD=<label>[,KEY=VAL,...] \
  [--remove-env-vars=KEY1,KEY2] \
  --no-traffic
# route 100% to the new revision (printed above), or the named rev:
gcloud run services update-traffic gnosis-openai-mesh --project=neutral-418500 --region=us-central1 \
  --to-revisions=LATEST=100
# scale audit (must show maxScale 1, minScale null/0):
gcloud run services describe gnosis-openai-mesh --project=neutral-418500 --region=us-central1 --format=json \
  | jq -r '{serving:[.status.traffic[]|select(.percent==100).revisionName][0], maxScale:.spec.template.metadata.annotations["autoscaling.knative.dev/maxScale"], minScale:.spec.template.metadata.annotations["autoscaling.knative.dev/minScale"]}'

NOTE: traffic does NOT auto-move to a new revision if it was previously pinned to a specific revision — you must update-traffic explicitly.

4. Profile (token-gated bench-decode-next)

PROFILE_TOKEN="$(gcloud run services describe gnosis-openai-mesh --project=neutral-418500 --region=us-central1 \
  --format=json | jq -r '.spec.template.spec.containers[0].env[]|select(.name=="MESH_PROFILE_TOKEN").value')"
curl -sS -X POST \
  'https://gnosis-openai-mesh-366749842679.us-central1.run.app/__mesh/bench-decode-next?model=mistral-7b&warmup=2&iterations=100&token=8&position=0' \
  -H "X-Mesh-Profile-Token: $PROFILE_TOKEN" --max-time 580 > /tmp/prof.json
# the response JSON embeds raw newlines → parse non-strict:
python3 -c "import json,re;d=json.load(open('/tmp/prof.json'),strict=False);s=d.get('stdout','')or '';print('code',d.get('code'));[print(l) for l in s.splitlines() if l.startswith(('mesh-decode-next:','split','micro'))];m=re.search(r'divergent_tokens=(\d+)',s);print('divergent_tokens:',m.group(1) if m else '?')"
  • warmup=1&iterations=4 = quick smoke (correctness/divergent_tokens); warmup=2&iterations=100 = the sustained tps number.
  • First call after a deploy is a COLD start (knot download + any GPU residency fill) — can take minutes; allow --max-time 580 and retry warm if it returns no code.
  • ACCEPT a candidate only if faster than baseline AND divergent_tokens=0.

5. Rollback / config-only A/B (NO rebuild)

  • Roll back / switch prod instantly: update-traffic --to-revisions=<known-good-rev>=100. Known-good stable coord as of 2026-06-21: gnosis-openai-mesh-00068-f45 (~10 tps, all experimental flags OFF, includes committed AVX2). The proven w11 baseline: gnosis-openai-mesh-00059-d75.
  • A/B experimental flags WITHOUT rebuilding: update ... --update-env-vars=KEY=VAL creates a new revision on the same image (env-only), then update-traffic. This is how to test the flag-gated GPU paths from ONE build:
    • MESH_GPU_SLAB=1 (+MESH_GPU_SLAB_GB=16) — contiguous f16 residency slab (full GPU residency).
    • MESH_GPU_FUSE=1 — fuse FFN gate+up (no win — bandwidth-bound).
    • MESH_GPU_Q8K=1 — custom in-kernel Q8 dequant-GEMV (fewer VRAM bytes).
    • FLUX_GPU_MIN_TOKENS=1 — engage any GPU path for single-token decode (default gates ≥2 → CPU).
    • MOONSHINE_CONTINUOUS_BATCH=1 (+MOONSHINE_BATCH_LANES=N, default 8) — fat-station continuous batching: a scheduler thread batches concurrent /generate decodes through decode_step_batch (one weight read per wave). Llama-path models only (qwen-dense/llama/tinyllama; gemma/MoE fall back to serial). Default OFF → byte-identical to today. Requires a rebuild that includes the fat-station continuous-batch code (gnosis ≥ a6a18504); NOT a pure env-only A/B on pre-a6a18504 images (the flag is a no-op without the code). CPU parity gate green (cargo test --bin fat-station batch_scheduler_tests); throughput is read-bound so the win shows on the L4, not CPU (gate with §4 bench-decode-next at concurrency, and/or scripts/continuous-batch-throughput-gate.mjs).
    • All default-OFF → behavior identical to the CPU baseline.

Gotchas

  • Redeploys are flaky under quota pressure — only when flux/ltx idle (see §2).
  • A model SWITCH restarts the child (cache lost); profile warm, same model.
  • MESH_PROFILE_TOKEN is rotated periodically; always re-read it (§4), don't cache.
  • Build context is ~466 MiB; the gcloudignore is a strict whitelist — see §0.