Distributed Inference — Onboarding Guide

Parent: distributed-inference

This guide exists so the next engineer can become productive in this package without archaeology. It maps the territory: what the project does, the vocabulary it uses, how a model travels from HuggingFace weights to a served token, where every kind of code lives, how to build/test/deploy, and which of the ~100 markdown files in this directory to read for any given question.

• What: the complete onboarding map for open-source/gnosis/distributed-inference.
• Why: the package spans a 219-module Rust crate, ~174 binaries, a Python encode/validate toolchain, four deployment targets, and a physical radio mesh; no single existing doc covers how the pieces relate.
• How: read Day One first, then the section matching your task. Use the Documentation Map to route into the deeper per-topic docs.
• Next: README.md for full package-level narrative, ARCHITECTURE.md for the serving topology.

Table of Contents

1. Day One
2. What This Project Is
3. Vocabulary — the project-specific jargon, decoded
4. Repository Layout
5. The Artifact Pipeline — HF weights → .knot → R2 → serving
6. The Rust Engine
7. The Python Toolchain
8. Serving Topologies
9. Build, Test, Verify
10. Deployment and Operations
11. Physical Hardware: R1 Fleet and SkyMesh
12. Current Status and Known Failure Modes
13. Documentation Map
14. Related Packages
15. First-Week Checklist

1. Day One

The five facts that orient everything else:

1. The product is compressed model serving. Large model weights are encoded into a custom binary container (.knot / .rknot), pushed to object storage (Cloudflare R2, fronted by https://edgework.ai/api/v1/r2/), and served by a Rust engine that reconstructs weights on demand. Headline number: Llama-3.3-70B compresses from 280 GB to 153.5 MB (≈1,824x) via the GKQ spectral pipeline, with sub-100 ms GPU reconstruction.
2. One crate, many surfaces. distributed-inference (Rust, MPL-2.0) builds as a native rlib, a cdylib for Node (NAPI), and wasm. The same kernels run in native "fat stations", Cloudflare Workers (via the wasm build consumed by apps/distributed-inference-worker), Android phones (the R1 fleet), and GPU containers on Cloud Run/Kubernetes.
3. Python encodes, Rust serves. Model encoding (HF → .knot) is Python (encode-knot.py and friends). Inference, mesh networking, and benchmarks are Rust. The Python inference-server-gkq.py exists for GPU-side GKQ reconstruction serving; the Rust fat-station binary serves the native mesh path.
4. Sovereign tooling. Work in the monorepo via pnpm run a0 -- run distributed-inference:<target> (Nx targets in project.json). Raw cargo is used inside this package for builds/benches where targets don't exist, but deploys go through the repo's a0 surface — never raw wrangler. Commit/push with a0 flow from the monorepo root.
5. Many docs are point-in-time. Most root-level *.md files are status snapshots, investigation logs, or falsification ledgers — valuable history, not current truth. The Documentation Map marks which is which. When a status doc and the code disagree, trust the code.
6. Do not over-name the carrier. This package is distributed inference until live obligations justify a stronger label. Read SARDIS_GUARD.md before promoting any runtime, standing-wave, or mesh claim into "distributed intelligence."

Smallest honest smoke test (no model download needed):

cd open-source/gnosis/distributed-inference
cargo build --release --bin fat-station          # ~full engine build
cargo test --lib cat_map_route                   # fast unit-test lane
pnpm run a0 -- run distributed-inference:lint    # clippy via Nx

2. What This Project Is

distributed-inference implements the Gnosis program's model-serving layer: weights, execution, and monitoring transported through one graph/runtime topology. In engineering terms it combines three threads:

1. Model compression science. GKQ (Gnosis Knot Quantization) spectral compression, K-quant (Q4_K/Q5_K/Q6_K/Q8_0) dequantization kernels, E8 and Leech lattice quantizers, saturation masks (frozen-neuron skipping), low-rank FFN/KV factorizations, and standing-wave dimension pinning.
2. Distributed systems engineering. A native mesh of "fat stations" (HTTP/UDS/UDP-flow transports) that split a model's layers across nodes; a Cloudflare Workers "cobordism" topology that shards the embedding vocabulary across Durable Objects; gossip membership for a fleet of Android R1 devices; and an RF/teleport path that broadcasts cache keys over software-defined radio.
3. Formal grounding. Many runtime gates mirror Lean 4 theorems in gnosis-math (E8 well-order, Fano incidence, FOIL loser-policy bounds, teleport admission). Modules carry theorem-lineage strings; certificates emitted by benchmarks cite the proofs that back each gate. The convention: runtime code mirrors or implements a Lean contract — claims beyond the proven boundary stay marked experimental.

Sibling packages it works with (see Related Packages): distributed-inference-host (TypeScript coordinator), gnosis-frf (fork/race/fold primitives), bitwise (binary encoding), gnosis-uring (native transport server), and apps/edge-workers (Cloudflare Worker surfaces that consume the wasm build and serve R2 knots).

3. Vocabulary

The codebase names things with deliberate metaphors. Engineering translations:

Style note you will see everywhere: modules carry theorem-lineage strings referencing Lean proofs in gnosis-math, and emitted reports are certificates (JSON with theorem refs + checksums). Claims not backed by a proof are deliberately labeled experimental/research — preserve that discipline when editing.

4. Repository Layout

distributed-inference/
├── Cargo.toml             # crate manifest: ~90 [[bin]] targets, feature flags
├── project.json           # Nx targets (the a0-facing command surface)
├── src/
│   ├── lib.rs             # 219 public modules, cfg-gated by target/feature
│   ├── README.md          # module-by-module map (maintained; read it)
│   ├── bin/               # 174 binaries: stations, probes, benches, agents
│   ├── rknot/             # .rknot format: encoder, decoder, backends, manifest
│   └── topology_mask/     # imaging topology masks
├── tests/                 # 22 Rust integration tests + 2 Python tests
├── docs/                  # curated deep-dive docs (teleport, R1, skymesh, …)
├── scripts/               # ops scripts (r1/, preconditioning/, probes, SDR)
├── examples/              # sample data + synth example
├── fixtures/              # test fixtures (e.g. raw-capture JSONL)
├── rknots/                # local encoded artifacts (gitignored *.knot)
├── probe-logs/            # preserved big-LLM probe logs
├── tools/gpu-hidden-embed # CUDA-side hidden-embed tool
├── *.py                   # encode/validate/bench toolchain (root level)
├── *.md (~100 files)      # docs: concepts, runbooks, status logs (see §13)
├── Dockerfile.*           # container images (skymesh, paris-probe, moonshine, quantized-weights)
├── cloudbuild-*.yaml      # GCP Cloud Build pipelines
├── k8s-deployment-quantized.yaml
└── deploy.sh / DEPLOY_NOW.sh / cloud-run-deploy.sh / deploy-to-cdn.sh

Sibling directory: ../distributed-inference-host/ — the TypeScript coordinator package (see §14).

5. The Artifact Pipeline

How a model goes from HuggingFace to served tokens:

HuggingFace weights (e.g. 280 GB safetensors)
   │
   ▼  [1] ENCODE (Python)
encode-knot.py --model <hf-id-or-dir> --output models/<name>.knot --hf-token $HF_TOKEN
   │     streaming tensor writes (no full-RAM accumulation); FLUX bundles
   │     transformer + VAE + CLIP + T5 into one knot
   ▼  [2] ADMIT (Rust quality gates)
bowl-probe  →  per-model admission JSON (dev-fast / dev-balanced / dev-strict
   │            profiles + gamma frontier), aggregated into a runtime certificate
   ▼  [3] ENRICH (optional)
profile-saturation-all-models  →  frozen-neuron bitmasks
encode-and-upload-rknots       →  .rknot with masks/spectral blocks embedded
   ▼  [4] DISTRIBUTE
upload-to-r2.py / deploy-to-cdn.sh
   │     R2 key shape: models/<name>.knot
   │     proxy: https://edgework.ai/api/v1/r2/distributed-inference/models/<name>.knot
   │     CDN:   https://cdn.forkjoin.ai/gnosis/weights/<model>/
   ▼  [5] SERVE (one of four topologies — see §8)
fat-station mesh │ Cloudflare cobordism workers │ K8s/Cloud Run GPU │ R1 fleet
   ▼  [6] VALIDATE
paris-probe-rknot --dense <url> --tokens ... --expected <token-id>

Reference encode + probe commands (current as of 2026-05-25 status doc):

# Encode
python3 encode-knot.py \
  --model Qwen/Qwen2.5-72B-Instruct \
  --output models/qwen2.5-72b-instruct.knot \
  --hf-token "$HF_TOKEN"

# Validate the served artifact end-to-end
target/release/paris-probe-rknot --dense \
  https://edgework.ai/api/v1/r2/distributed-inference/models/qwen2.5-72b-instruct.knot \
  --dense-only --tokens 785,6722,315,9625,374 --expected 12095

Model support matrix (maturity as of the May 2026 status docs — re-verify against BIG_LLM_DISTRIBUTED_INFERENCE_STATUS_2026_05_25.md):

Quantization formats handled by the loader/kernels: F32, F16, Q4_K (144 B/256 elems), Q5_K, Q6_K (210 B/256 elems), Q8_0, plus GKQ2/GKQ3 spectral blocks and E8/Leech lattice codes.

6. The Rust Engine

Crate distributed-inference v0.1.0 (MPL-2.0). Library targets: rlib + cdylib (NAPI). Release profile: opt-level=3, codegen-units=1, lto="thin", panic="abort".

Feature flags

Module families (full map in src/README.md)

• Kernel core: model.rs (NativeLlamaPipeline; Qwen2.5 default; FFN leakage/guard modes via GNOSIS_FFN_* env vars), math.rs (SIMD f16/Q-quant dequant, matvec, RoPE, softmax, norms), loader.rs (KNOT parser; mmap backend for local files, HTTP-Range backend for R2), gkq_backend.rs (GKQ2/GKQ3 spectral reconstruction with LRU cache).
• Caching: kv_cache.rs (+ with_base fluid-memory mode), amplituhedron*.rs (prefill volume cache + wire codec), matvec_memo.rs, prefill_replay.rs.
• Compression research: standing_wave_*.rs, resonant_ffn.rs, e8_quantizer.rs, leech_quantizer.rs, kv_cache_e8.rs, pcar_codec.rs, residual_dict.rs, tau_bowl_dequant.rs, inference_saturation.rs.
• Mesh & transport: mesh_gossip.rs (UDP membership; DIRECT heartbeats mark alive, RELAYED lists are discovery-only), swarm_net.rs, protocol69.rs, teleport.rs, rf_beacon.rs, gnosis_foil.rs (FOIL scheduler + RF substrate engines), fano_runtime.rs, mesh_probability_admission.rs.
• Models: model_phi3*.rs, model_gemma4.rs, model_qwen35moe.rs, model_mamba.rs, model_rwkv7.rs, model_hybrid.rs, model_whisper_{encoder,decoder}.rs, model_pneuma_tts_*.rs, model_qwen3_tts_*.rs, model_flux*.rs, model_ltx_*.rs, model_molmoact2*.rs, model_glm_ocr_vit.rs, model_qwen2vl_vit.rs.
• Imaging/3D: depth_*.rs, sfm_two_view.rs, mesh_depth_field.rs, holographic_*.rs, mesh_residual_deblur.rs, leech_visual_sieve.rs.
• Telemetry/formal: pleromatic_*.rs, scribal_standing_wave.rs, spectrometer.rs (JSONL via GNOSIS_MESH_SPECTROMETER_JSONL), semantic_authority_boundary.rs, neurosymbolic_tool_markov.rs.
• Wasm surface: wasm_bindings.rs (WasmLlamaPipeline, WasmHostBackend), wasm_bindings_whisper.rs, wasm_bindings_tts.rs, wasm_topology_mask.rs, wasm_sfm.rs, wasm_mesh_depth_field.rs.

Binary taxonomy (174 in src/bin/; declared in Cargo.toml)

Hot-path environment variables

GNOSIS_FFN_LEAKAGE_MODE=observe|off|mask-low-N|guard-tail-low-43|...   # FFN skip experiments (observe = no logit change)
GNOSIS_FFN_GUARD_ADMIT_WEATHER_CELLS=3332          # calibrated guard-cell admissions (empty by default)
FOIL_RACE_LOSER_POLICY=integration|cancel|adaptive|drain
FOIL_ASYNC_LOSER_WORKERS=1..8                      # default 1; >1 measured slower, opt-in
FOIL_NAIVE=0|1                                     # 1 disables HELIX
GNOSIS_MESH_SPECTROMETER_JSONL=<path>              # residual-capture JSONL lane
GNOSIS_FOIL_PROBABILITY_ADMISSION=0                # opt out of Flow/UDP admission cache

/decode-next also accepts request-scoped header overrides (X-FFN-Leakage-Mode, X-FFN-Guard-*) so experiments don't require station restarts — bench-decode-next drives these.

7. The Python Toolchain

The Python scripts at the package root form the encode → validate → serve → distribute pipeline. They share logic deliberately: the streaming encoders and sketch generator import their math from encode-real-model.py via importlib, so there is a single source of truth for quantization.

Two artifact formats, two encoder families

• KNOT (b"KNOT" magic): raw container — JSON metadata + tensor payload. Large matrices Q8_0-quantized (32-elem blocks), small tensors F32. Consumed by the Rust loader (mmap or HTTP Range).
• GKQ v2/v3 (b"GKQ2" magic): spectral format — per-tensor rank-K SVD blocks (U/σ/Vᵀ as offset-128 uint8 with delta encoding) plus dense blocks (norms, embeddings, biases). v3 widens dense byte_length to u64 for ≥4 GiB tensors (e.g. Qwen-72B's token embedding). Reader/writer: gkq_v2_format.py; Rust consumer: src/gkq_backend.rs.

Encoders

Sketches, witnesses, masks

• generate-sketches.py — <model>-sketches.knot (rank-K lm_head/projection sketches, SK32 payload, HTTP-Range friendly — deliberately not delta-encoded, unlike GKQ). Format spec: sketch_knot_format.py.
• generate-escape-witnesses.py — does a rank-K lm_head sketch preserve argmax ordering for SSM/recurrent architectures? Emits witness JSON (see escape-witnesses.json, falcon-mamba-witnesses.json, recurrence-witnesses.json).
• generate-rknots.py, quantize-rknots.py — rknot metadata generation/compaction for the four headline models.
• scripts/saturation_mask_encoder.py / _decoder.py / test_saturation_masks.py — the Python side of saturation bitmasks.

Validators (run these before trusting an artifact)

Servers

• inference-server-gkq.py — FastAPI server keeping quantized GKQ weights memory-resident, reconstructing layers on demand with an LRU GPU cache. Env: WEIGHTS_PATH, GPU_CACHE_SIZE_MB (default 8000), MAX_AGE_SECONDS (default 3600). This is what the K8s/Cloud Run image runs.
• inference-server-gkq-gpu.py — monkey-patch wrapper swapping the CPU reconstruction for the NVRTC CUDA kernel (gkq_gpu_binding.py, reconstruct-weights-gpu.cu); falls back to CPU on any GPU error; adds /gpu-stats.
• paris-server.py — OpenAI-compatible shim (/v1/chat/completions, /v1/completions, /health) that tokenizes with an HF tokenizer and forwards to a fat-station /generate. Env: FAT_STATION_URL (default http://127.0.0.1:8000), PARIS_PORT (8080), HF_MODEL, MODEL_ID.
• openai-proxy.mjs — the Node equivalent used in the SkyMesh image (port 8081).

Distribution

upload-to-r2.py — multipart upload through the edgework proxy. Env: PROXY_BASE (e.g. https://edgework.ai/api/v1/r2), R2_TOKEN. 16 MB parts, 6 retries with backoff.

Dependencies

No requirements.txt is checked in; the working set is: numpy, torch, transformers, safetensors, huggingface_hub, ml_dtypes, scikit-learn (randomized_svd), fastapi + uvicorn (servers), gguf (converter), requests, sgp4 (NOAA pass scheduling), diffusers>=0.27 (FLUX/LTX tests), psutil (validation harness).

scripts/ highlights

• scripts/spawn-rknot-mesh.sh — launch N local fat-stations over contiguous layer windows against one rknot+dense pair (pids in /tmp/fat-station-mesh-pids.<start_port>).
• scripts/r1/ — R1 fleet benchmarking: mesh-bench.py (latency/throughput against /chain entries), two-node-forward.py (Mac entry + R1 exit over USB), mesh-shootout.py, mesh-timeprofile.py, mass-equip.sh.
• scripts/preconditioning/ — prompt-preconditioning experiments; headline finding: per-request word-swap is niche (~2%) but amortized prefix compression pays ~11% recurring prefill savings (compress-precond.mjs is the valuable one).
• scripts/aeon_*.sh + aeon_noaa_pass_schedule.py — SDR/satellite lanes (NOAA APT pass scheduling via sgp4, pager/IoT resolvers, live SDR).
• scripts/skymesh-linux-ap.sh — bring up the SkyMesh WiFi access point.
• scripts/submit-paris-probe-cloud.sh, scripts/submit-knot-reencode-cloud.sh, scripts/run-big-llm-probe-containers.sh, scripts/monitor-big-llm-probes.sh — see §10.

8. Serving Topologies

Four ways the same artifacts get served:

8.1 Native fat-station mesh

Each station serves a contiguous layer range (e.g. trisplit 0–9 / 9–18 / 18–28); KV cache runs in fluid-memory mode (kv_cache::with_base) so a node holds only its own range's cache. Handoff between stations is HTTP, UDS (fat-station-uds), or 10-byte Aeon Flow UDP frames (fat-station-flow-udp). mesh-elect queries a seed's /mesh/peers and recommends which stage a joining node should take. fat-station also exposes GET /.aeon/attention-closure-benchmark (live optimizer-admission certificate) and /decode-next (single-step decode with experiment headers).

8.2 Cloudflare cobordism mesh (edge)

Documented fully in ARCHITECTURE.md. Embedding lookup is sharded across 8 worker-cobordism shards, each owning a vocab range and a CobordismWardenDO Durable Object (L1 = 256 MB DO RAM LRU, L2 = Workers KV F32 rows with 24 h TTL, L3 = R2). Layer compute runs on worker-node1 stations (entry: layers [0,14); exit: layers [14,28) + lm-head), each with a layer-warden-do (L1 = 1 GiB DO RAM). The TypeScript coordinator (distributed-inference-host) fans out, merges partial embeddings by elementwise addition, and pipelines prefill. Known sharp edge: a single shard timeout poisons Promise.all merges — the fix direction is allSettled.

8.3 GPU containers (K8s / Cloud Run)

Dockerfile.quantized-weights (CUDA 12.2.2, sm_80/A100) serves GKQ reconstruction + inference on port 8000 (/health, /ready, /infer, metrics on 9090). Kubernetes manifest: k8s-deployment-quantized.yaml (namespace gnosis-distributed, 3–10 replicas via HPA, A100/H100 node affinity). Cloud Run service: gnosis-quantized-inference (24 GB / 8 CPU, concurrency 10, timeout 3600 s). See §10.

8.4 R1 fleet + SkyMesh radio

Five Rabbit R1 Android devices on WiFi "Hawk" each serve the full qwen2.5-0.5b knot (373 MB) as data-parallel lanes — data port 8090, gossip 9090/UDP. Dockerfile.skymesh packages fat-station + protocol69-{embed,gen}-agent with an OpenAI-compatible proxy on 8081 (MODEL_NAME=gnosis, SKYMESH_URL=https://skymesh.forkjoin.ai). The radio layer broadcasts protocol69 frames over 433.92 MHz ISM via ADALM-Pluto, with two RTL-SDR receivers as witnesses. See §11.

9. Build, Test, Verify

Nx targets (preferred surface — pnpm run a0 -- run distributed-inference:<target>)

The top-level test target is currently a deferred echo (aeon-monitor fields drifted) — run module-scoped test-* targets or cargo test --lib <module> instead.

Integration tests (tests/)

Most are #[ignore]-gated because they need real .knot artifacts from R2 (Qwen-Coder-7B, Gemma4-31B, TinyLlama); several fall back to synthetic Gaussian tensors when the artifact is missing. Highlights:

• gkq_resident_knot_integration.rs — GKQ2/GKQ3 round-trip + cosine parity (synthetic; runs anywhere)
• amplituhedron_attention_real_tensor_parity.rs — real-vs-synthetic Q/K/V divergence
• phase1_standing_wave_deployment.rs, standing_wave_*_validation.rs — coverage/speedup gates
• {phi3_lowrank,hybrid,mamba,rwkv7}_integration_test.rs — per-model pipelines
• saturation_integration.rs, resonant_ffn_integration.rs — optimization gates
• mesh_probability_admission.rs, e2e_gnexec_handshake.rs, skymesh_wifi_pluto_integration.rs — mesh/transport
• Python: tests/test_encode_knot_flux.py, tests/test_flux_knot_diffusers.py (need diffusers>=0.27, safetensors, numpy)

Verification ethos

Benchmarks here are falsification instruments, not marketing: gates are explicit (bench-decode-next --gate speedup|tail-latency|semantic|none), divergence counts are first-class outputs, and several documented results are negative (mycelial cache 0.41x, full FFN fusion 0.55x, Variant C MMLU regression). Keep that standard: report the losing numbers.

10. Deployment and Operations

Identities and endpoints

Deploy paths

# Master pipeline: verify → build → bench → package → upload → deploy → validate
DEPLOYMENT_ENV=production TARGET_DEPLOYMENT=k8s bash deploy.sh
# (flags: SKIP_BENCHMARKS, SKIP_UPLOAD, FORCE_REBUILD)

# One-shot
bash DEPLOY_NOW.sh                          # K8s default
DEPLOYMENT_TARGET=cloud-run bash DEPLOY_NOW.sh

# Weights to CDN
bash deploy-to-cdn.sh                       # → gs://gnosis-distributed-weights + cdn.forkjoin.ai

# Health
kubectl rollout status deployment/quantized-inference -n gnosis-distributed
gcloud run services describe gnosis-quantized-inference --region us-central1
curl http://localhost:8000/health

Reminder: Cloudflare Worker deploys anywhere in the monorepo go through a0 xr deploy [--env <env>] — never raw wrangler.

Cloud Build pipelines

• cloudbuild-paris-probe.yaml — E2_HIGHCPU_32, 200 GB disk, 4 h timeout. Steps: assert build context → build paris-probe-rknot/verify-rknot → fetch dense knot + rknot from R2 → probe → optional dense baseline / verify / residual diff → upload to gs://<project>_paris5b/. Submit via scripts/submit-paris-probe-cloud.sh [max-layers] with env toggles PARIS_RUN_DENSE_BASELINE, PARIS_DUMP_RESIDUALS, PARIS_KERNEL_DEBUG, PARIS_RKNOT_R2_KEY.
• cloudbuild-skymesh.yaml — builds the SkyMesh node image from Dockerfile.skymesh (build context = open-source/, parent of gnosis + bitwise).
• cloudbuild-build-probe-only.yaml — probe binary only → gs://neutral-418500_cloudbuild/probe-binaries/.
• scripts/submit-knot-reencode-cloud.sh [model] [layers] — cloud re-encode with a minimal source tarball.

Big-LLM probe operations (the current live workflow)

# Start/restart the three local probe containers (Gemma4-31B, Qwen72B, Llama70B)
./scripts/run-big-llm-probe-containers.sh
# Watch them (poll every 300 s; logs in probe-logs/)
INTERVAL_SECONDS=300 ./scripts/monitor-big-llm-probes.sh

Verdict states surfaced per model: pending → loading → forwarding → logits → pass/fail. Container fallback note: dockerized probes hit a local containerd storage error in May 2026; screen sessions (gemma4_probe, qwen72_probe, llama70_probe, monitor bigllm_monitor) are the proven fallback.

Troubleshooting quick table (full detail: troubleshooting_guide.md)

Operating targets: p99 < 100 ms, error rate < 0.1%, GPU mem < 80%, cache hit rate > 95%.

11. Physical Hardware: R1 Fleet and SkyMesh

• R1 fleet: 5 live Rabbit R1 devices (+1 to flash), WiFi "Hawk" (10.0.0.0/24), mixed Android 13/16, hardware-derived node IDs (r1-<hash>). Each serves qwen2.5-0.5b in role both. Provisioning: scripts/r1/mass-equip.sh (uses mesh-elect against a seed's /mesh/peers). Docs: docs/R1_FLEET.md, docs/R1_RESILIENT_MESH.md, docs/R1_MESH_PROTOCOL69.md.
• Radios: 2× Nooelec NESDR SMArt v5 receivers (RTL2832U; SN 94092559 → index 0, SN 78922015 → index 1) and one ADALM-Pluto transmitter (USB 0x0456:0xb673). Frequencies: 433.92 MHz ISM (protocol69 framed key), 102.49 MHz FM (witness anchor — venue-specific, scan first), 137 MHz NOAA APT (pass-timed demo). Mars-loopback sample rate 2.4 MS/s; start TX gain at −20 dB on the bench. Setup: docs/HARDWARE_SETUP.md.
• Open item: RTL-SDR over USB-OTG directly on R1 is not yet qualified; the proven configuration is bridge-host (radio on a desktop, IQ fed to the R1).
• No radio? gnosis-foil-control --raw-engine ambient-host provides a safe no-hardware witness engine, and fixtures/gnosis-foil-raw-capture.jsonl exercises the raw-capture path.

12. Current Status and Known Failure Modes

Snapshot from the 2026-05-25 status doc (verify before relying on it):

Working and validated: GKQ pipeline (compression ratios ~1,664–1,824x across Phi-3/Qwen/Gemma/Llama with 45–95 ms reconstruction); Phi-3-mini production with saturation masks (1.89x); cobordism mesh + Frobenius merge; fluid-memory stations; Hope Jar E8; bowl-probe admission suite; FFN fusion+saturation combined 1.89–2.12x.

In flight: Gemma4-31B / Qwen2.5-72B / Llama-3.3-70B full-gate probes (encodes complete, layer-by-layer probes running; interrupted repeatedly by workstation crashes — sessions restart via the §10 probe workflow); MolmoAct2 VLA serving (697 s end-to-end, dominated by 100-step depth autoregression — needs per-depth-token timing before any optimization).

Falsified / negative results (do not re-try without new evidence):

• FFN low-rank Variant C (50% all-layer): violates the McNally Cliff criterion (σ₁/σ₂ < 8 on late layers) → 10.1% MMLU loss. Rejected.
• Full gate+up+down FFN fusion: 0.55x (memory layout). Partial fusion wins.
• Mycelial Fano cache: 0.41x in smoke — research path only.
• FOIL_ASYNC_LOSER_WORKERS=4: slower than 1 in saturation probe.
• Predictive 43-row-skip FFN: instrumented, semantically clean, but fails tail-latency gate — not promotable yet.
• Guard weather cells 3321,3322: replicated raw-divergence misses → on the explicit deny list.

Environmental gotchas: big encodes need disk headroom (a crash series traced to ~12 GiB free); Cloud Build retains only the latest 3 probe logs (older ones manually preserved in probe-logs/); gcloud account taylorbuley@gmail.com, project neutral-418500.

13. Documentation Map

~110 markdown files live here. Classification key: CONCEPT (design/format, still authoritative), RUNBOOK (operational how-to), CONTRACT (interface spec between components), STATUS (point-in-time snapshot — history, not current truth), INVESTIGATION (root-cause/falsification log).

Start here

By family

Knot/RKNOT format (CONCEPT + CONTRACT + RUNBOOK): RKNOT_RUST_CONTRACTS.md (format.rs is the SSOT), RKNOT_PHASE2_CONTRACTS.md, RKNOT_PHASE3_CONTRACTS.md, RKNOT_DEPLOYMENT_GUIDE.md, RKNOT_FAT_STATION_RUNBOOK.md (boot a station from .rknot + dense fallback), RKNOT_INTEGRATION.md, RKNOT_ENCODING_STATUS.md (STATUS), src/rknot/README.md.

GKQ (CONCEPT): GKQ_TECHNICAL_GUIDE.md, GKQ_POLYGLOT_DEPLOYMENT.md (Python-train / native-serve / wasm-edge tiers), GKQ_WHISPER_HIGH_RANK_FINDING.md (INVESTIGATION — Whisper is high-rank; spectral sketching fails on it, q8 block quant is the floor).

Saturation masks (CONCEPT + RUNBOOK): SATURATION_MASKS_README.md (entry point), SATURATION_BITMASKS.md, SATURATION_MASK_INTEGRATION.md, SATURATION_MASKS_INDEX.md, SATURATION_IMPLEMENTATION_SUMMARY.md, SATURATION_PROFILER_SUMMARY.md, docs/PHI3_SATURATION_PROFILER.md.

Adaptive compression / McNally Cliff (CONCEPT): ADAPTIVE_COMPRESSION_SPEC.md (CliffAtlas: compress a layer only when σ₁/σ₂ ≥ 8; 376-byte L1-resident table), then _INTEGRATION, _TESTS, _DELIVERABLES, _IMPLEMENTATION_SUMMARY, docs/ADAPTIVE_COMPRESSION_GUIDE.md.

FFN optimization (CONCEPT + STATUS): docs/FFN_OPTIMIZATION_INNOVATIONS.md (the full 10-optimization writeup), FFN_OPTIMIZATION_FINAL_SUMMARY.md, PHI3_LOWRANK_README.md + PHI3_LOWRANK_ANALYSIS.md (variants A/B/C; C was falsified), BENCH_FFN_Q4K_README.md, FOIL_HOTPATH_MERSENNE_PLAN.md.

The Five Deaths (CONCEPT/CONTRACT — the optimization roadmap): DEATHS_CONTRACT.md (module contracts for all five), AMPLITUHEDRON_PLAN.md (Death #2), AMPLITUHEDRON_BITWISE_WIRE.md (fp48 wire), AMPLITUHEDRON_LIQUID_MEMORY.md (eviction), OCTONION_ROUTING_PLAN.md (Death #4: Fano-plane hop routing), DEATH5_CROSS_PREFIX_COMPRESSION.md, AETHER_FLOW_INTEGRATION_SCOPE.md (Death #3 scope), TOPOLOGY_CEILING_BREAKERS.md.

Standing wave (STATUS series — read newest-first): STANDING_WAVE_STATUS.md (the big ledger), _SHIPPING, _2026_05_03, _2026_05_03_AFTER_FALSIFICATION (PCA-only speculative demoted here).

Mesh falsification (INVESTIGATION — the debugging masterclass files): COLTRANE_MESH_FALSIFICATION_LEDGER.md (14 numbered failures F-mesh-1..15, 11 resolved — the single best document for learning how this team debugs), F_MESH_3_BYTE_FALLBACK_ROOT_CAUSE.md, F_MESH_5_THROUGHPUT_CLIFF_ANALYSIS.md (per-hop wire serialization, not matmul, dominates), F_MESH_15_CLIENT_SUBSTRATE_INVESTIGATION.md, KV_CACHE_OOM_ROOT_CAUSE.md (F-mesh-1), SPLIT_A/B/C_BOTTLENECK_INVESTIGATION*.md (post-SIMD per-layer bottlenecks), COBORDISM_LM_HEAD_SPEEDUP.md, BOWL_MESH_FALSIFIABILITY.md (prediction ledger protocol).

Pneuma voice (CONTRACT + STATUS): PNEUMA_INTEGRATION_PLAN.md, PNEUMA_CONTRACTS.md, PNEUMA_PHASE_2.md (TTS), PNEUMA_BULK_TRANSCRIPTION_TRACKER.md, WHISPER_COMPRESSION_METHODS_BENCHMARK.md, WHISPER_RESIDENT_KNOTS.md.

Teleport / RF / mesh protocol (CONCEPT, in docs/): docs/TELEPORT.md + _BENCHMARKS, _LIMITATIONS, _SCALING, docs/MARS_DELAY_TOLERANT_TELEPORT.md, docs/SKYMESH.md + docs/SKYMESH_DEMO.md, docs/HARDWARE_SETUP.md, docs/R1_FLEET.md, docs/R1_RESILIENT_MESH.md, docs/R1_MESH_PROTOCOL69.md, docs/R1_MESH_SCALING_PLAN.md, docs/R1_INAPP_NODE.md, docs/R1_GOLDEN_STAR.md, PROTOCOL69_HIDDEN_EMBED.md, docs/GLOBAL_CACHE_DESIGN.md, docs/MONITOR_BRIDGE_DESIGN.md, docs/MONITOR_CLIENT_DESIGN.md.

Deployment (RUNBOOK + STATUS): DEPLOYMENT_RUNBOOK.md and HOT_PATH_DEPLOYMENT.md are the operational ones; DEPLOYMENT_INDEX.md, DEPLOYMENT_READY.md, DEPLOYMENT_COMPLETE.md, DEPLOYMENT_SUMMARY.txt, OPERATIONAL_CHECKLIST.md, integration_checklist.md, performance_dashboard.md (Prometheus/Grafana queries), cost_impact_report.md are point-in-time completion claims — treat as STATUS.

Imaging / topology / misc concepts: TOPOLOGY_IMAGING.md, JIT_SPECULATIVE_DEQUANT.md (τ-bowl gate), STRUCTURED_SPECTRAL_QUANTIZATION.md, TOPOLOGICAL_SPECTROMETER_DUAL_TRACK.md, HELIX_COLUMNAR_EMBED_PROPOSAL.md, PLEROMATIC_RESIDUAL_ATLAS.md, GLOSSOLALIA_V2_INTEGRATION.md, MOLMO_ACTION_SERVING.md, ACKERMANN_CERTIFICATION_PROTOCOL.md, AETHER_GNOSIS_URING_INTEGRATION.md, docs/PILLAR2_Q4K_TAU_BOWL_NATIVE.md (2026-06-06 — among the newest).

Status snapshots (STATUS — newest first as of this writing): BIG_LLM_DISTRIBUTED_INFERENCE_STATUS_2026_05_25.md, PROJECT_STATUS_MAY_18_2026.md, MESH_CONSUMPTION_WAVE_STATUS.md, SESSION_NARRATIVE_2026_05_03.md (six waves of parallel agent dispatch — good for understanding how work happens here), BENCHMARK_WINS_COMPREHENSIVE.md, WEIGHT_QUANTIZATION_STATUS.md, COLUMNAR_BANDWIDTH_REPORT.md, LM_HEAD_ROUTER_REPORT.md.

A reading caution: several STATUS docs make completion claims ("production-ready", "all tests passing") that reflect their moment, and the compression headline varies between docs (1,824x in the deployment set, 14,000x in GKQ_TECHNICAL_GUIDE — different model/method baselines). When numbers matter, re-run the cited benchmark rather than quoting a doc.

14. Related Packages

../distributed-inference-host (TypeScript: @a0n/distributed-inference-host)

The host shim that wraps the wasm build of this crate and orchestrates multi-station inference from JS land. It owns: knot header parsing and R2 range fetching on the JS side (host-shim.ts), layer prefetch + LRU eviction (6-layer default), the async-fetch ↔ sync-wasm bridge, station transports (station-uds.ts, station-ws.ts — the Death #3 persistent substrate, station-flow.ts, station-flow-udp.ts), pipelined prefill/generation with amplituhedron replay/capture (pipeline.ts, prefill-window.ts), and an OpenAI-compatible mesh server (openai-server-mesh.ts). It also carries the imaging (deblur, style-transfer) and TTS host paths. Build/test via the package's pnpm monster scripts/open-source-repo-target.mjs {build,lint,test,typecheck} scripts. Key docs there: README.md, README-DEBLUR.md, README-STYLE-TRANSFER.md, AMPLITUHEDRON_COORDINATOR_PROTOCOL.md, FRAME_COALESCING_PROTOCOL.md, ZEDGE_CONSUMER_MAP.md.

Monorepo consumers (forkjoin-ai/monorepo)

• apps/distributed-inference-worker — the Cloudflare Worker entry that calls the wasm exports.
• apps/worker-node1, apps/worker-node2 — layer-station workers.
• apps/layer-warden — the layer-warden Durable Object.
• apps/edge-workers — hosts the R2 proxy and the integration test apps/edge-workers/src/__tests__/integration/distributed-inference.integration.test.ts (coordinator health/node discovery, layer-node forward pass, end-to-end inference, DO caching, across several model configs).

Sibling crates/packages in this repo

• gnosis-frf — fork/race/fold primitives and prelude (direct dependency).
• bitwise (open-source/bitwise) — binary encoding (bwDense, BWF2 frames).
• gnosis-uring — native transport server; recognizes certified FANO Flow payloads.
• gnosis-math / lean/ — the Lean 4 proofs that runtime certificates cite.
• aeon-monitor (repo root) — the native mesh visualizer (feature-gated bin here; fields have drifted — see project.json test target note).

15. First-Week Checklist

Day 1 — orient:

• Read this file, then README.md top-to-bottom (long; skim benchmarks) and ARCHITECTURE.md.
• Build: cargo build --release --bin fat-station --bin bowl-probe.
• Run a unit lane: cargo test --lib cat_map_route && cargo test --lib loader.

Day 2 — touch an artifact:

• Fetch or encode the small knot (qwen2.5-0.5b) and run native-pipeline-smoke against it.
• Run bowl-probe with --quality-profile dev-fast and read the admission JSON it emits.

Day 3 — serve:

• Start a local fat-station, hit /decode-next and /.aeon/attention-closure-benchmark.
• Run bench-decode-next --gate semantic against it; read the summary.

Day 4 — the edge path:

• Read docs/TELEPORT.md and src/protocol69.rs; run pnpm --dir open-source/gnosis run protocol69:foil-probe -- --self-test.
• Skim distributed-inference-host and the worker integration test in apps/edge-workers.

Day 5 — operations:

• Read troubleshooting_guide.md and the latest BIG_LLM_*STATUS*.md; check whether probes are running and learn the restart workflow (§10).
• Read the falsified-results list (§12) so you don't re-fight settled battles.

Rules of the road (from repo policy):

• Sovereign tooling: a0 / monster / gnode / a0 flow / a0 xr deploy. Raw wrangler, raw cross-submodule git, and raw test runners are task failures.
• No paid AI bindings in worker code (no env.AI, no [ai]).
• State relationships precisely ("implements", "mirrors", "maps to") — no emphatic identity claims; if an identity is real, it gets proven in Lean.
• Benchmarks must be falsifiable and report negative results honestly.
• Maintain the README tree when adding or materially changing directories.