forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Sovereign Object Detection & Labeling — Plan

distributed-inference/docs/SOVEREIGN_VISION_PLAN.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Sovereign Object Detection & Labeling — Plan

Goal. A forkjoin-owned, commercial-clean model that does what NVIDIA LocateAnything-3B does — open-vocabulary object detection, dense labeling, and point localization — built on our own vision stack (ViT encoder + scattering-amplitude attention), trained by distilling LocateAnything as a training oracle, then run free of it ("break the scaffold").

Status: PLAN (2026-06-09). Owner: primary agent + Taylor.

1. Strategy

LocateAnything-3B  ──run as oracle──►  (image, prompt) → {label, box, points}
   (NVIDIA, NC)         (batch)                    │
                                                   ▼
                                       distillation dataset
                                                   │
                                   train ──────────▼──────────
                          Sovereign detector (our ViT + scattering attn
                          + grounding head)  ── eval standalone ──► drop oracle

Teacher → student distillation into a different architecture, then the student stands alone. The oracle is a one-time/batch data generator, never a served dependency.

2. License posture (the gate)

NVIDIA OneWay Non-Commercial License on LocateAnything-3B:

  • Work + derivative works = non-commercial only (NVIDIA itself excepted).
  • Silent on outputs — no clause restricts use of what the model generates.

The scaffold-break rides on that silence: running the oracle to generate labels is permitted research/eval use; a model trained from scratch on those generated coordinates (functional data, not the weights) is plausibly not a "derivative work of the Work." This is the contested frontier of distillation licensing, not a settled win. Harden the position:

  1. Different architecture — ours is ViT + scattering-amplitude attention, not MoonViT + Qwen2.5. ✅ already true.
  2. Independent signal — mix in human-verified and/or other-source labels so the student is demonstrably not a pure functional clone.
  3. Counsel review before any commercial ship. Research/eval is clear now.

3. Assets we already have (reuse, don't rebuild)

Asset File Role in student
MolmoAct2 ViT encoder src/model_molmoact2_vit.rs 27-layer ViT, 378², 14px patches→729; Molmo lineage is grounding-native. Default encoder.
Qwen2-VL ViT encoder src/model_qwen2vl_vit.rs native-resolution ViT alternative (better for small objects / large images)
Scattering-amplitude attention src/scattering_attention.rs + src/scattering_amplitude.rs Lean-formalized Plücker/KPlane softmax replacement; the sovereign differentiator (A/B vs softmax)
Amplituhedron volume cache src/amplituhedron.rs prefill-volume reuse (inference-time speedup, not training)
KNOT encode + streaming Q8 encode-knot.py (add_q8_streamed) quantize the trained student to a servable knot
fat-station serving src/bin/fat-station.rs serve the sovereign detector once trained
Federated training FedMime* (lean + runners) distributed student training across the mesh

The only genuinely new component is the grounding head (§5.3).

4. Oracle host (LocateAnything-3B)

Local GTX 1080 Ti is architecture-blocked (Pascal sm_6.1: no bf16, no flash-attn, PyTorch dropping sm_61 — not a driver issue; driver 566/CUDA 12.7 is current). The oracle needs one of:

  • Cloud Ampere+ CUDA (GCP L4/T4/A10, spot) — transformers bf16, trust_remote_code. Recommended (we're GCP-authed; batch job, then tear down).
  • Apple Silicon + MLX (mlx-vlm ≥ 0.6.1, 4-bit 2.96 GB) — when the M1 (10.0.0.27) is back online.

Oracle is batch, so a transient cloud GPU is cheapest and unblocks today.

5. Sovereign detector architecture

5.1 Encoder

Start with MolmoAct2 ViT (model_molmoact2_vit.rs, fixed 378², simplest path; grounding-native lineage). Keep Qwen2-VL ViT as the native-resolution upgrade for small-object / high-res regimes.

5.2 Attention

Default softmax first (get parity), then A/B-swap encoder + head attention to scattering-amplitude attention (scattering_attention.rs) — the sovereign, Lean-backed geometric path. This is the differentiator that makes it ours, not a MolmoAct re-skin.

5.3 Grounding head (NEW — the real build)

DETR-style set prediction:

  • N learned object queries cross-attend to ViT patch tokens.
  • Per query: class/label (open-vocab via a small text-embedding match, or a fixed label vocab in v1) + box (normalized cx,cy,w,h in [0,1000] to match the oracle) + optional point.
  • Training loss: Hungarian-matched set loss (class CE + L1/GIoU box), DETR-style.
  • Output format mirrors LocateAnything (<box>x1,y1,x2,y2</box>, <box>x,y</box>, [0,1000]) so distillation targets line up 1:1.

v1 simplification: fixed label vocab (COCO/Objects365 classes). v2: open-vocab text-prompted (small text encoder → query conditioning).

5.4 Servability

Train in PyTorch (or our trainer), export weights → encode-knot.py (add_q8_streamed handles the ViT tables) → fat-station pipeline model_sovereign_detector.rs (new, mirrors the molmoact2_vit forward + head).

6. Distillation dataset

  • Corpus: start with an open image set (COCO / Objects365 / LVIS images) + forkjoin's own images. (Decision needed: which + how many — target ~100k–1M images for a strong v1.)
  • Labeling: batch-run the oracle with detection/grounding prompts → (image_id, prompt, [{label, box[4], points[]}]) JSONL.
  • Quality: spot-check + confidence filtering; reserve a human-verified slice for §2.2 independent signal and for honest eval.

7. Phases & exit criteria

Phase Work Exit criteria
0 License/corpus confirm NC-on-outputs posture; pick corpus counsel note filed; corpus chosen
1 Oracle stand-up cloud GPU; mlx_vlm/transformers smoke test 1 image → correct boxes, verified
2 Distill set batch-label corpus → JSONL N images labeled, QC passed
3 Student build encoder reuse + grounding head + scattering-attn option forward runs; overfits a tiny batch
4 Train FedMime distributed train on distill set train/val loss converges
5 Break scaffold eval student standalone on COCO/LVIS val (mAP) mAP within target of oracle; oracle dropped
6 Serve (opt) knot + model_sovereign_detector.rs + fat-station admitted via a detection qspec

8. Risks

  • License (§2) — the gate; resolve before commercial ship.
  • Grounding head is the unknown — set-prediction training is finicky (matching instability, slow convergence). Mitigate: start from DETR/Deformable recipes; overfit-a-batch sanity gate before full training.
  • Scattering attention is un-wired/additive — A/B vs softmax may show divergence; treat softmax as the fallback so the head trains regardless.
  • Oracle label noise — LA isn't ground truth; cap with confidence filtering
    • human-verified eval slice.
  • Compute — distributed training cost; FedMime mesh vs a cloud run TBD.

9. Immediate next actions

  1. Pick corpus (own images vs COCO/Objects365/LVIS, and scale).
  2. Stand up the oracle on a GCP Ampere+ GPU; smoke-test one image → boxes.
  3. Spec the grounding head concretely (query count, label vocab v1, loss) and stub model_sovereign_detector.rs against model_molmoact2_vit.rs.
  4. Wire a batch-labeling harness (oracle → JSONL) reusing the corpus.

(Heretic-admission track continues in parallel: qwen35moe ext4 probe → verdict, then E4B encode via the new streaming-Q8 path.)