Golden Star: quality-confirmed distributed inference on the Rabbit R1

The aeon-shell runtime foothold — proven end-to-end on a USB-connected, rooted Rabbit R1 (GSI aarch64, Linux 4.19, ~4 GB RAM, 104 GB disk).

An admitted (or soon-to-be) moonshine-ready agent model confirms its own quality through our admission gates (monster-guard qspec), running on the R1's distributed inference. If it passes the gate, we trust it works.

The stack

swerve cline           Arch Linux ARM sliver on the R1 (mount+UTS ns + chroot)
  └─ monster-guard      lease + capability gate; admission gated on the qspec quality probe
      ├─ moonshine      FRF shell / distributed-inference client (--oneoff mesh generation)
      └─ fat-station     house NativeLlamaPipeline inference SERVER (KNOT model, HTTP)
         + paris-probe-rknot   the quality probe (argmax check) the gate runs

All four are pure-Rust, cross-compiled to aarch64-unknown-linux-musl with cargo-zigbuild (zig as the cross C-compiler/linker — no Android SDK/NDK): monster-guard 826 KB · fat-station 3.1 MB · paris-probe-rknot 2.6 MB · moonshine 49 MB. (moonshine needs reqwest on rustls-tls, not native-tls/openssl.)

What was proven on-device

1. fat-station serves a real model on the R1. Dense qwen2.5-0.5b-instruct.knot (374 MB, mmap demand-paged): pipeline ready in 6 ms, listening on http://0.0.0.0:8080, ~480 ms/token on the ARM CPU.
2. Client-server inference. moonshine --oneoff (client) → fat-station (server): prompt "The capital of France is" → tokens [785,6722,315,9625,374] → first generated token id 12095 = " Paris".
3. The quality gate passes. paris-probe-rknot --dense qwen05-dense.knot --dense-only --tokens 785,6722,315,9625,374 --expected 12095: max finite logit +14.34 at token_id 12095 → PASS: argmax 12095 matches.
4. Admission is gated on quality (the golden star). monster-guard delegate qwen-0.5b-agent net,fs --qspec paris.json → quality spec passed → delegated exec/run on plugin:qwen-0.5b-agent. The grant is minted ONLY because the model argmaxed " Paris".
5. Admitted model runs under a lease. monster-guard run --name qwen-0.5b-agent --caps net,fs --lease 60 -- paris-probe-rknot … → lease + revocation still enforced → PASS.
6. The whole stack runs INSIDE a swerve sliver. swerve cline base:scratch --rootfs <arch> -- bash golden-star.sh (chroot + mount/UTS ns, swerve braid-cert admitted) runs steps 4+5 from inside the Arch sliver: host: aarch64 / "Arch Linux ARM" → quality spec passed → delegated → lease enforced → PASS: argmax 12095. The full nest is proven: swerve sliver → monster-guard gate → quality-confirmed model.
7. The FOIL entropy collector runs alongside, client-server. gnosis-foil-control --host 127.0.0.1 --port 8886 is the collector server (/.aeon/model-forward-cache, /.aeon/health); the inference client posts its residue via GNOSIS_FOIL_CACHE_URL. On the R1: the PARIS probe with that env set logs foil-cache-admit: status=200 key=qwen05-dense:dense:785,6722,… while still PASS: argmax 12095 — entropy collected (status 200), keyed by model+tokens, in the same run as the quality confirmation. (Query the cache back with a matching ?key=…; a bare query is bad_request by design.)

Capstone: the whole foothold in one swerve cline

scripts/r1/foothold.sh, run via swerve cline base:scratch --rootfs <arch> -- bash /foothold.sh, does it all in a single invocation on the R1:

=== aeon-shell foothold inside a swerve sliver: aarch64 / "Arch Linux ARM" ===
1) monster-guard: quality spec passed -> delegated      (admission gated on PARIS)
2) The capital of France is ->  Paris                    (admitted model, as text)
3) cache_hit:true argmax 12095 logit 14.34               (FOIL entropy collected)

swerve sliver -> monster-guard quality gate -> admitted model answers " Paris" -> FOIL collector holds the residue. Sovereign, quality-gated distributed inference on a $200 device, end to end.

Sharded inference is bit-perfect (2-station parity) — proven on the R1

parity-station-handoff qwen05-dense.knot on the R1 splits the model across two stations (station1 = layers 0..12, station2 = layers 12..24) and compares the chained residual against the single all-24-layer pipeline:

TEST 1 (4 tokens):           cos=1.000000  max-abs-delta=0.000000  -> PASS
TEST 2 (8 tokens, 4+4):      cos=1.000000  max-abs-delta=0.000000  -> PASS
OVERALL: PASS

The sharded handoff is bit-identical to the monolithic pipeline. This is the correctness proof underneath "as big as we can get away with": a model can be split across stations and chained exactly. The two stations are in-process here (one device); real mesh distribution is the same wire (/embed → /forward with X-Layer-Range → /lm-head) across separate nodes — only a second physical node is missing, not the mechanism.

Real two-node distribution across separate machines (Mac + R1) — proven

The second physical node is now supplied: the Mac mini (Apple M1, aarch64) runs the entry half and the R1 runs the exit half, over the wire, for real.

tokens -> /embed (Mac entry) -> /forward 0-12 (Mac entry, native build)
       -> [residual crosses the USB wire] ->
       -> /forward 12-24 (R1 exit, aarch64-musl) -> /lm-head (R1 exit) -> argmax

Both stations load the full qwen-0.5b knot and each runs only the layer range named in X-Layer-Range (entry 0-12 on the Mac, exit 12-24 on the R1). Result on the actual two machines (scripts/r1/two-node-forward.py, R1 tunneled via adb forward tcp:8090 tcp:8090):

[mac]  /embed         -> 4480 f32  (seq=5 hidden=896)
[mac]  /forward 0-12   -> 4480 f32
[r1]   /forward 12-24  -> 4480 f32   (over the USB wire; R1 logs layers=[12,24) 455 ms)
[r1]   /lm-head        -> vocab=151936 (190 ms)
argmax=12095 logit=14.3434  expect=12095   -> PASS (" Paris")

The compute is genuinely split across two real machines. Versus running all 24 layers on the Mac alone (monolithic), the two-node residual is numerically equivalent — cos=1.00000000, max-abs-delta=2.67e-05 (just Mac-native vs R1-musl floating-point rounding), and both paths argmax 12095 at the identical logit 14.3434. This is the wire-level confirmation of the in-process parity above, now across separate nodes. (This distributes compute; both nodes still hold the full weights — per-node memory sharding is the open path below.)

Three nodes (Mac + 2× R1) — proven

With a second R1 cabled (data cable + USB-debug authorized), the same wire extends to three nodes, splitting the 24 layers evenly and crossing the USB wire twice (scripts/r1/three-node-forward.py):

[mac]  /embed          -> embed
[mac  entry] /forward 0-8   -> residual
[r1#1 mid ] /forward 8-16   -> residual  (over the wire; R1#1 logs [8,16) 402 ms)
[r1#2 exit] /forward 16-24  -> residual  (over the wire; R1#2 logs [16,24) 304 ms)
[r1#2 exit] /lm-head        -> vocab=151936 (195 ms)
argmax=12095 logit=14.3433  expect=12095  -> PASS (" Paris")   (1326 ms across 3 nodes)

Three separate machines each computed a third of the network; the answer is the same (" Paris", logit 14.3433 — matching the 2-node/monolithic 14.343x to FP rounding). Adding nodes is now mechanical: push fat-station + the knot, start it with a layer range, adb forward, and add a hop to the driver.

The gate discriminates (admit good, reject lesser) — proven on the R1

The admission gate is not a rubber stamp; it has teeth. Two models, same PARIS qspec, opposite verdicts:

monster-guard delegate --qspec refused to admit two models — including a genuinely different architecture/tokenizer (SmolLM2, Llama-arch) — because their top token wasn't " Paris". The gate filters real quality across model families; it does not wave models through. That is what "we can trust it actually works" means.

Gating distinct-tokenizer models (the earlier blocker — the in-tree /tokenize is a Qwen-only smoke table): tokenize the probe prompt with the model's OWN tokenizer on the host (its cached HF tokenizer.json) and feed the raw token ids to paris-probe-rknot --tokens … / the qspec prompt_tokens. No GGUF, no download, no third-party engine. SmolLM2's "The capital of France is" → [504,3575,282,4649,314], " Paris" → 7042; the qspec then runs entirely on our own kernel. (SmolLM2-360M is chat-tuned — BOS <|im_start|> — so on a raw completion it argmaxes a newline, not " Paris"; it fails this raw-completion bar.)

Reproduce

# host: cross-build (zig + aarch64-musl target + cargo-zigbuild installed)
for B in fat-station paris-probe-rknot; do
  ( cd open-source/gnosis/distributed-inference &&
    CARGO_PROFILE_RELEASE_DEBUG=0 cargo zigbuild --release --bin $B --target aarch64-unknown-linux-musl ); done
( cd open-source/gnosis/monster-guard && cargo zigbuild --release --target aarch64-unknown-linux-musl )

# push binaries + the dense model + the qspec
T=aarch64-unknown-linux-musl/release
adb push open-source/gnosis/distributed-inference/target/$T/fat-station        /data/local/tmp/fat-station
adb push open-source/gnosis/distributed-inference/target/$T/paris-probe-rknot  /data/local/tmp/paris-probe-rknot
adb push open-source/gnosis/monster-guard/target/$T/monster-guard             /data/local/tmp/monster-guard
adb push ~/.edgework/models/qwen2.5-0.5b-instruct.knot                        /data/local/tmp/qwen05-dense.knot
adb shell 'chmod 755 /data/local/tmp/{fat-station,paris-probe-rknot,monster-guard}'

# qspec.json — the PARIS quality spec the admission gate runs:
#   {"probe":"/data/local/tmp/paris-probe-rknot","knot":"/data/local/tmp/qwen05-dense.knot",
#    "prompt_tokens":[785,6722,315,9625,374],"expect_token":12095,"expect":"Paris"}

# the golden star: admit only if the model confirms quality
adb shell 'cd /data/local/tmp && HOME=$PWD/mghome MONSTER_GUARD_HOME=$PWD/mghome \
  ./monster-guard delegate qwen-0.5b-agent net,fs --ttl 3600 --qspec paris.json'

"As big as we can get away with"

The R1 (~2.7 GB free) runs qwen-0.5b comfortably; ~Gemma2-2B is the local ceiling. Bigger models come from distributed sharding across the R1 + mesh stations (fat-station --role mid --layers a..b chained over UDS/flow-UDP), the same client-server wire — the device holds a slice, the mesh holds the rest. Larger dense knots are already staged on the build host (llama-3.2-3b, mamba-2.8b, qwen2.5-7b).

8. Human-readable answer. With TOKENIZER_GGUF_PATH pointed at the Qwen2.5 GGUF, fat-station's /detokenize resolves the full vocab, so moonshine --oneoff -c "The capital of France is" prints, on the R1: The capital of France is -> Paris.

Performance: where the time goes, and the optimizations

Measured with scripts/r1/mesh-shootout.py (latency + throughput) and scripts/r1/mesh-timeprofile.py (per-stage, attributed via fat-station's X-Compute-Ms header) on the Mac mini (M1) + 2x R1 mesh.

Where the time goes (single forward, qwen-0.5b, 5-token prompt):

• Wire/USB overhead is negligible — ~15 ms/hop, 1-6% of total. The residual is 17,920 B; USB + adb-forward + HTTP are not the bottleneck. Compute is 94-99%.
• Compute splits into two regimes:
  • Prefill (new prompt): the transformer layers dominate (~85%). R1 ARM is a stable ~41 ms/layer (the quantized matvec is already NEON + rayon over 8 cores — at the kernel ceiling). The Mac M1 is erratic (14 -> 148 ms/layer under sustained load; thermal/memory) — an unreliable mesh node.
  • Decode (KV-cached steady state): the lm-head dominates (~188-225 ms) — a memory-bound projection of the residual against the 144 MB tied embedding over the 151,936-token vocab. (Same-prompt "warm" runs hide the layer cost via the KV cache; mesh-timeprofile.py --vary defeats that to show true compute.)

Optimization results (all argmax-verified " Paris"/12095):

Channels (USB vs WiFi vs BTLE vs FOIL): per-request the channel is cheap (residual 17,920 B, USB RTT ~11 ms) — a liquid/SSM model makes the handoff small and latency-tolerant. But post-microopt the shared USB bus (both R1s on one hub) caps aggregate throughput — independent per-node channels (WiFi P2P) are the unlock. BTLE = discovery only (too slow for streaming); FOIL = an RF entropy source, not a node-to-node data channel. For true peer-to-peer (R1<->R1, no coordinator) the R1s must join the same WiFi LAN; the forward chain then points the next-hop URL at the peer's IP instead of the Mac.

Takeaways: the biggest win was a microopt (last-position lm-head, 2-3.5x); after it, the shared USB bus is the throughput limiter -> WiFi/P2P is the next unlock. The Mac mini is an unreliable sustained node; the sovereign R1 mesh is the stable target. Per-node memory sharding (fitting a bigger model across nodes) remains the open mechanism below.

Correctness: bowl_q_filter + qwen3 arch (2026-05-24)

Two correctness fixes landed after the numbers above were first taken:

• bowl_q_filter was silently corrupting attention for every model. A BowlMeshHead Q-mutation was Some(default) by default and applied to Q before attention scoring in all four forward paths. Robust models tolerated it; deeper ones (qwen2.5-3b, mistral, llama, qwen3) did not. Root-caused by HF-reference bisection (qwen2.5-3b: HF Paris rank0 → knot rank7; encoder faithful within the Q8 step; numpy with the same weights cos 0.99997 vs the runtime 0.957). Fixed to None by default. Effect: mistral-7b now answers "Paris" / "cold" (rank 0, both wrong before); qwen-0.5b still argmaxes 12095 but the logit sharpened 14.34 → 16.82 (the golden-star number above was the degraded value).
• qwen3 arch support (q_dim = num_heads·head_dim ≠ hidden_dim; per-head QK-norm; tied lm-head): fixed a diagnostic OOB panic, the attention-output stride (written hidden_dim-strided while the O matmul reads qkv_dim → only some heads computed, Q didn't propagate; now qkv_dim-strided in all four paths, batch_xb sized max(hidden,qkv)), and the missing per-head QK-norm in the serving path. All no-ops for qwen2/llama (q_dim==hidden), so the golden star is unaffected.

Remaining (mesh-gated / incremental)

• Per-node memory sharding (so a model too big for one node fits across several, each holding only its slice). The compute split is proven on two real machines (above); loading only a slice per node is the open part. Two precise gaps found running shard-knot → fat-station end-to-end:
  1. shard-knot residency — FIXED (gnosis 784234be). It checked contains_key("token_embd_weight") but source keys carry a .bin suffix, so the entry shard shipped without the embedding. Added resolve_key (.bin tolerant) + tied-embedding handling (qwen ties output_weight->token_embd, so the exit shard now carries token_embd for its /lm-head). Entry shard now has the embedding (145 tensors); exit has output_norm + tied embed (146). Unit-tested.
  2. No partial-layer/role loading — the remaining blocker. fat-station's full-model constructor resolves EVERY component (embed + all num_layers layers + final_norm + lm_head) and panics on any missing — so even a correct entry shard panics on output_norm_weight (then blk_12). Needs a role-aware partial constructor: entry loads embed + layers [0,split) (skip norm+lm_head); exit loads layers [split,end) + norm + lm_head (skip embed); all load only [start,end) at their ORIGINAL indices (naive renumbering is unsafe — masks_layer(layer, num_layers) is index-dependent). Touches ResolvedLocs (Option embed/final_norm/lm_head, sparse layers) + the role/layers wiring in fat-station. Verify any sharded run argmaxes 12095 / matches the 14.343x logit. This is what unblocks the larger models (mamba-2.8b / qwen3-4b / qwen2.5-7b all OOM a single R1's ~2.95 GB). Bigger dense knots (llama-3.2-3b, mamba-2.8b, qwen2.5-7b) are staged on the host; once memory sharding lands they distribute across nodes.
• More nodes: two R1s now enumerate (Mac + 2× R1 = the 3-node chain above). A third R1 was reported connected but does not yet present a USB data device (charge-only cable / dead hub data port) — swap to a data cable + authorize USB-debug and it adds a fourth hop the same way.
• Widen the qspec gate to every admitted moonshine-ready agent model (each with its own qspec). On the R1 alone this is limited to small models.
• Executor refinement: PID/NET namespaces via the second-fork-to-PID-1 + veth increment (today: mount + UTS ns).