forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

R1 Mesh Scaling Plan — node counts, the bus, and the plug→equip→release lifecycle

distributed-inference/docs/R1_MESH_SCALING_PLAN.md
forkjoin-ai/gnosis
Repo overview Open in workbench

R1 Mesh Scaling Plan — node counts, the bus, and the plug→equip→release lifecycle

How many Rabbit R1s to cover every model we run, up to 70B+, and the connectivity architecture to get there. Grounded in measured numbers from the aeon-shell foothold (see R1_GOLDEN_STAR.md).

Measured per-node constraints (one R1)

resource value source
RAM total / free 3.95 GB / ~2.95 GB free /proc/meminfo
usable RAM-resident weights ~2.4 GB/node (free − KV − activations − OS headroom) budget
disk 104 GB df
CPU MT6765, 8 cores; ~41 ms/layer (NEON + rayon, at the kernel ceiling) mesh-timeprofile
per-hop residual hidden_dim × 4 B/token = 3.6 KB (0.5B) … 32 KB (70B) wire
current data bus USB2 to host, ~11 ms RTT; shared bus saturates under load shootout

Sizing rule: a KNOT is ≈ 1.0 GB per 1B params at our mixed Q5/Q6/Q8 (qwen-0.5b 0.75, qwen3-4b 1.0, mistral-7b 1.03). Q4_K ≈ 0.6 GB/1B.

Nodes needed (fast / RAM-resident) = ceil(model_GB / 2.4). A model can also be disk-paged (mmap from the 104 GB disk) to exceed RAM, but each token then re-reads the shard from eMMC (~0.3 GB/s) → only sane when the per-node shard is small.

Coverage table

model params knot (mixed) nodes (mixed) knot (Q4) nodes (Q4)
qwen2.5-0.5b 0.5B 0.37 GB 1 0.3 GB 1
qwen3-4b 4B 4.0 GB 2 2.4 GB 1–2
qwen2.5-7b 7B 4.4 GB 2 4.0 GB 2
mistral-7b 7B 7.2 GB 3 4.0 GB 2
Llama-3 8B 8B ~8 GB 4 ~5 GB 2
~13B 13B ~13 GB 6 ← 6-R1 ceiling (mixed) ~8 GB 4
32B 32B ~32 GB 14 ~19 GB 8
70B 70B ~70 GB 30 ~42 GB 18
120B (e.g. MoE) 120B ~120 GB 50 ~72 GB 30

(MoE caveat: for sparse-MoE 120B only the active experts compute per token, but ALL expert weights must be resident across the mesh — size by total params, not active.)

What your 6 R1s cover

6 × 2.4 GB = ~14.4 GB RAM-resident.

  • Everything we have, fast, with room to spare. mistral-7b (the biggest, 7.2 GB) needs 3 nodes → 6 R1s run it plus a second model, or run mistral-7b as two 3-node replicas (data-parallel) for ~2× throughput.
  • Largest single model, fast: ~13–14B at mixed quant, or ~22–24B at Q4_K.
  • 70B: not fast on 6. It needs ~18 R1s (Q4) to ~30 (mixed) RAM-resident. On 6, 70B is only reachable disk-paged (~12 GB/node on disk) at tens of seconds per token — fine for a correctness demo, not interactive.

Buy-ladder for a fast 70B mesh: ~16–18 R1s at Q4_K (≈2.6 GB/node incl. the entry/exit embedding+lm-head overhead), ~24–30 at mixed quant. Round to 18 for a comfortable Q4 70B with a spare; 6 is the right size for "everything ≤13B + replicas."

The bus problem (6+ devices on one host)

USB only needs to carry the residual at runtime (≤32 KB/token/hop — trivial), but 6 R1s on one host's USB controller share one bus and contend (measured: throughput regressed 17→9 req/s when 2 R1s shared the bus once compute was fast). So USB does not scale as the runtime fabric. Two clean options:

  1. Provisioning hub (wired, transient): one powered 7–10-port USB 3.0 hub handles all 6 for equip + charge. Bandwidth is irrelevant for a one-time push, so even sequential single-cable provisioning works. This is the equip bus, not the run bus.
  2. Runtime = wireless (below). USB after equip is charge-only (the charge-only cables already on hand are perfect here).

Target lifecycle: plug → equip → release → talk (WiFi/BTLE)

This is the architecture that actually scales past a USB hub.

  1. PLUG — R1 connects to the host over USB (hub or one-at-a-time).
  2. EQUIP — push, once: the fat-station musl binary, this node's model shard (shard-knot output for its layer range), and a node manifest {node_id, model_id, role, layers[a,b), successor_endpoint}. Install fat-station as a boot service bound to the WiFi IP.
  3. RELEASE — unplug from data; the R1 runs on battery (or charge-only USB). It joins WiFi and advertises itself.
  4. TALK (P2P, no coordinator in the data path):
    • WiFi = data plane. Each node forwards its residual directly to its successor's IP:8090 (/embed → /forward X-Layer-Range → … → /lm-head). 32 KB/hop over WiFi is ~ms; a liquid/SSM model's constant-size state makes it even more latency-tolerant.
    • BTLE = control/discovery plane. Beacon {node_id, model_id, layer_range, ip}; used to find each other + health + (re)assign shards. Too slow for data (~1–2 Mbps → 32 KB ≈ 0.26 s/hop), perfect for control.
    • FOIL/RF = entropy only (gnosis-foil-control + SDR), not a comms channel.
    • Coordinator is ephemeral: at startup one node (or a phone/Mac) elects shard assignments and builds the ring, then steps out — steady-state is peer-to-peer.

Mesh forms: discover peers (BTLE/mDNS) → elect ring + assign [a,b) ranges by node RAM → each node loads only its shard (fat-station --sharded --role … --layers a..b) → chain over WiFi. Adding a node = plug, equip, release; the ring re-forms.

Where we are vs. this plan

  • Done: per-node memory sharding (the partial loader, cos=1.0 vs monolithic on qwen-0.5b); the same /embed→/forward→/lm-head wire proven across Mac+2×R1; the bowl_q_filter + qwen3 arch correctness fixes (models now answer correctly).
  • Gaps for this plan: (a) WiFi/P2P transport (R1s currently USB-tethered, not yet joined to a LAN); (b) discovery (mDNS/BTLE beacon — not built); (c) the ring coordinator/elector; (d) a boot service for fat-station; (e) RAM-aware shard assignment. None are large; the kernel + sharding + wire are already proven.

Bottom line

  • Everything we have today (≤7B): 3 R1s (mistral-7b); your 6 gives full coverage ≤13B plus replicas.
  • 70B fast: ~18 R1s (Q4) — beyond 6; on 6 it's disk-paged and slow.
  • The plug→equip→release→WiFi-P2P lifecycle (BTLE for discovery, WiFi for the residual, FOIL for entropy) is the right fabric for 6→18+ nodes and removes the shared-USB-bus ceiling.