KAIJU the whole model zoo — generalization + execution plan

Goal (user, 2026-07-10): generalize the KAIJU encode pipeline off its qwen2.5-7b hardcoding so we can produce a KAIJU1 knot for any qspec-covered model, then lay out the full ordered execution plan. This doc is the SSOT for that rollout. Companion state file: IQ_ROLLOUT_LOG.md (sibling in this dir). Commands below assume the repo root as CWD (the cloudbuild yaml lives there).

Status: LAUNCHED (self-draining) 2026-07-10. The zoo-watcher LaunchAgent is loaded and the campaign is live. Two encodes are still in flight (Codestral IQ-ladder 7c829a70, KAIJU2 role-mix 09452e34); the one-encode-at-a-time gate holds, so the watcher DEFERS every tick until a slot frees. The FIRST action it will take when a slot opens is qwen2.5-0.5b KAIJU encode (the one READY model); it then walks the queue, auto-submitting IQ-ladder feeders for not-ready models and KAIJU-encoding each once its f16+imatrix land — one build at a time, unattended. No encode was force-started during go-live.

1. What KAIJU needs per model (the prerequisite set)

A KAIJU1 knot is built by scripts/kaiju-knot-driver.py: it re-encodes every 2D attn/ffn weight tensor (those with an imatrix H entry) as KAIJU1 from f16 truth, and copies everything else (lm_head, token_embd, attn_v, norms) verbatim from an existing base knot. So the real inputs are:

Key consequence: the IQ-ladder is the feeder. It's the only thing that produces f16 + imatrix. The base knot already exists for every ADMITTED model. The gate already exists in every argmax-gated qspec. So "KAIJU a model" reduces to "run its IQ-ladder (once, to mint f16+imatrix), then run the KAIJU encode."

The IQ2_M base is NOT required — the qwen reference happened to use its IQ2_M knot as the base, but the driver copies whatever types the base carries for the non-KAIJU tensors, and all of Q4_K/Q5_K/Q6_K/Q8_0 are natively servable. So the model's production .knot is a perfectly good base. This drops IQ2_M from the critical path for the fleet.

2. Zoo inventory

State observed 2026-07-10 (gcloud storage ls for f16; R2 proxy HTTP range probes for imatrix/knots; ADMITTED_MODELS.md for base-knot/qspec status). ✓ present, ✗ absent, ~ partial/caveat.

Argmax-gated transformer models (KAIJU-eligible), small → large

Big-disk (do last — need diskSizeGb 400 + longer timeouts)

Out of scope for the KAIJU argmax gate

• No argmax token (generative / gateway): flux-1-schnell, ltx-video, unlimited-ocr, openai-gateway, codestral-22b-openai (the gateway qspec).
• SSM / hybrid (mamba-2.8b, falcon-mamba-7b, jamba-1.5-mini, rwkv7-2.9b): the KAIJU encoder targets 2D attn/ffn matmuls; SSM/conv/MoE layers need their own encode+serving path. The KAIJU serving arms panic (by design) on MoE expert-block slicing → qwen35moe-heretic excluded too.

Headline: only qwen2.5-0.5b and qwen2.5-7b have cached f16 today. Every other model is blocked on its IQ-ladder feeder run (which mints f16 + imatrix and persists them). Base knots + qspec gates already exist fleet-wide.

qspec reconciliation (2026-07-10, go-live) — codestral corrected

Audited monster-swarm/qspecs/*.qspec.json directly. Every admitted dense transformer carries a REAL argmax qspec (prompt_tokens + a genuine f16 expect_token), so the gate is data-ready fleet-wide:

The one correction: codestral is NOT gate-ready. Its only qspec is codestral-22b-openai.qspec.json, a gateway sentinel — prompt_tokens: [1], expect_token: 1, knot: /dev/null, modality: coding-chat-completions, shape-identical to openai-gateway ([1]→1). That gates a live HTTP endpoint's coding quality, not an f16-argmax reproduction. There is no codestral-22b.qspec.json, so kaiju-gate-params.py correctly errors (no qspec for codestral-22b). The earlier "no argmax qspec" read was right — the assumption that a qspec's mere existence makes a model gate-ready does not hold for the gateway sentinels. So codestral stays HOLD on two counts: no dense .knot base and no argmax gate (both must be minted first: a dense base knot from f16, and an f16-derived expect_token). SSM/image/gateway qspecs (mamba/rwkv/jamba/falcon-mamba, flux/ltx/ocr, openai-gateway) are out-of-scope by architecture, not by a missing gate.

3. The qwen-hardcoding audit (cloudbuild-kaiju-gate.yaml)

Every qwen2.5-7b-specific assumption in the original gate yaml, with the parameter that replaces it in the generalized cloudbuild-kaiju-encode.yaml:

No hardcoded arch params (hidden/layers/vocab) exist in the yaml — those are read from the knot/gguf at runtime by the driver and the native pipeline. The gate was the only genuinely qwen-specific logic, and it turned out to be data, not code: it lives in the qspec.

4. The generalized pipeline

cloudbuild-kaiju-encode.yaml (new)

Parameterized 1:1 with the IQ-ladder's style. 16 substitutions; qwen2.5-7b defaults preserved, so a bare gcloud builds submit --config=cloudbuild-kaiju-encode.yaml reproduces the reference case. Steps: fetch-f16 (GCS) → fetch-r2 (imatrix + base knot) → build-bins → encode-model (the ~8-12h/7B leech encode) → gate → ship. fetch-f16 fails loud with a "run the IQ-ladder first" message if f16 is absent (honest DAG — the ladder is the feeder, not a silent fallback).

The gate mechanism (the linchpin)

The " Paris" gate was never really about Paris — it's "does the KAIJU dequant reproduce the full-precision argmax on a fixed prompt?" Every admitted model already carries that reference in its qspec:

• prompt_tokens — the fixed prompt, pre-tokenized (so the gate needs no per-model tokenizer at runtime — it feeds token IDs straight to native-pipeline-smoke --tokens).
• expect_token — the argmax the full model produces on that prompt. This is literally how the model was admitted at full precision, so it is the f16 argmax reference. Requiring argmax == expect_token = requiring the KAIJU dequant to reproduce f16's completion.
• reject_token — a plausible-but-wrong token (anti-rubber-stamp; recorded in telemetry).

This directly fixes the Codestral failure mode logged wake-172 ("Codestral is a CODE model; the qwen-tuned factual battery does not port; paris_pass=FALSE") — the gate no longer demands " Paris" of anyone; it demands that model's own f16 token, whatever it is.

No hand-picking. scripts/kaiju-gate-params.py reads monster-swarm/qspecs/<id>.qspec.json and emits every substitution mechanically (gate tokens + all default keys + family tokenizer). Verified across families: qwen (12095), mistral (6233), llama (12366).

Automatic derivation without a qspec (for a new model, or to cross-check an existing token): run the model's own f16 through llama.cpp greedily on the prompt and take the argmax as expect_token — the IQ-ladder already builds llama.cpp and holds the f16, so a one-line llama-cli -n 1 --top-k 1 there mints the gate. The qspec path is primary because it's already validated fleet-wide; f16-derivation is the fallback for models lacking an argmax qspec (documented, not yet wired — it's the natural next feeder-side addition if a non-qspec model needs KAIJU).

ppl anchor (secondary, best-effort)

The ctx-128 perplexity check (the wake-147 lesson: single-token margin ≠ distributional quality) runs only when _TOKENIZER resolves to a loadable HF tokenizers json and gates only when _PPL_MAX is set; otherwise it's informational or skipped. We carry qwen/gemma/tinyllama tokenizers in-repo; tiktoken families (llama3) skip ppl gracefully — the argmax gate still holds. This keeps the pipeline universal without pretending we have every tokenizer.

Helpers (new, all in distributed-inference/scripts/)

• kaiju-gate-params.py — qspec → substitutions (json / shell / gcloud-subst).
• kaiju-zoo-submit.sh <id> [--check|--submit] [--writer-re RE] — checks the 3 prereqs (f16/imatrix/base knot), assembles substitutions, dry-runs by default, refuses to submit if any encode is in flight.

Verified: qwen2.5-7b → READY, emits a valid submit command; mistral-7b → NOT READY (no f16 yet), correctly points at the IQ-ladder feeder.

5. Dependency DAG

  HF weights ─┐
              ▼
        [IQ-ladder run]  (cloudbuild-iq-ladder.yaml, ~2-8h)
              │  ├─ convert → f16 ──────────────► GCS models/<id>/<id>-f16.gguf   (FEEDER)
              │  ├─ imatrix ───────────────────► R2  models/iq-ladder/<id>/<id>.imatrix (FEEDER)
              │  └─ IQ rungs + eval (informational; not on the KAIJU path)
              │
  prior admission ──► R2 models/<id>.knot  (BASE KNOT — already exists fleet-wide)
  qspec  ───────────► prompt_tokens + expect_token (GATE — already exists)
              │
              ▼
        [KAIJU encode]  (cloudbuild-kaiju-encode.yaml, ~8-12h/7B)  ◄── one-at-a-time
              │  driver: f16 + imatrix + base knot → KAIJU1 knot
              ▼
        [gate]  native argmax == qspec expect_token  (+ optional ctx-ppl)
              ▼
        R2 models/<id>-kaiju1.knot  + .telemetry.json + .gate.log

The two feeder edges (f16, imatrix) are the whole gating constraint. Base knot + gate are always satisfiable. So the drain order is: for each model — ladder (if no cached f16) → KAIJU encode → gate → ship, one encode at a time.

6. Orchestration — serial queue watcher

Mirrors the proven mistral-iq-ladder-watcher.sh launchd pattern (idempotent, single-flight, one-encode-at-a-time), but drains an ordered queue instead of a single follow-on.

• scripts/kaiju-zoo-queue.txt — ordered small→large model list; comments mark ready-now vs auto-feed vs EXTERNAL vs HOLD vs out-of-scope.
• scripts/kaiju-zoo-watcher.sh — now two-stage (feeder-aware). Each hourly tick, if no encode is in flight, it walks the queue and takes AT MOST ONE action on the first not-yet-.done model:
  • READY (f16+imatrix+base knot) → submit the KAIJU encode (.done marker).
  • NOT READY + a mapped HF source (hf_source_of()) + no .feeder marker → submit its IQ-ladder feeder (cloudbuild-iq-ladder.yaml, .feeder marker). A later tick then finds it READY and KAIJU-encodes it.
  • EXTERNAL (mistral-7b) → skip; its own watcher owns the feeder.
  • HOLD (unmapped: loki / gemma4 / codestral) → skip with a loud log. Feeder and KAIJU are gated identically by the one-encode-at-a-time check, so the zoo self-drains across the whole two-stage DAG, one build at a time, with no hand-holding. A failed feeder is skipped (not looped) — bounded by the .feeder marker. Exit 11 when every non-HOLD model has a .done marker.
• one-at-a-time gate hardening (go-live): the ongoing-build detector now keys on _OUT_KEY too (kaiju/.knot), not just _HF_SOURCE/_KNOT_NAME. The role-mix kaiju2 build (09452e34) carries ONLY _OUT_KEY, so the old detector would have gone blind to it once the codestral ladder cleared — a latent double-encode hole. Same fix applied to kaiju-zoo-submit.sh.
• scripts/com.forkjoin.kaiju-zoo-watcher.plist — hourly LaunchAgent (enable with the two launchctl lines in the script header).

Parallelism note: Cloud Build would happily run several encodes at once, but the campaign's one-encode-at-a-time rule exists to keep the laptop-side submit + staging serial and the R2/GCS budget bounded. The encodes themselves are fully independent (different models, different keys) — if the user lifts the rule, the watcher's ongoing-gate is the single line to relax, and N encodes cost N× but finish in max(N) wall-clock instead of sum(N). Recommendation: keep serial for the ladder feeders (they're the contended, staging-heavy step), allow 2-3 parallel KAIJU encodes once feeders are cached (they only fetch, then compute).

7. Execution plan + cost

Wall-clock scales linearly with params (encode = nearest_leech per bulk block). Cost = build-minutes × E2_HIGHCPU_32 ($0.064/min ≈$3.84/h; treat as rough — Cloud Build machine rate). Encode-step timeout in the yaml is 16h, global 22h → 12B is near the ceiling; 22B+ need a timeout bump (as the ladder did 4h→8h).

Rough totals (serial, one-at-a-time):

• Small/mid tier (0.5B–12B, 11 models): encodes ≈ $320, feeders ≈ $140 → ~$460, ~5-6 days wall-clock.
• • 22B/31B: +~$260, +~3 days (after timeout/disk bumps).
• • 70B-class (2): +~$700, +~1 week — recommend deferring until the hexacode fast decoder lands (turns ~90h into ~single-digit hours).

Ready to fire on a green light

1. qwen2.5-0.5b — the cleanest first real fleet KAIJU (validation-size, both prereqs cached). One command: bash open-source/gnosis/distributed-inference/scripts/kaiju-zoo-submit.sh qwen2.5-0.5b --submit
2. qwen2.5-7b — already has a shipped kaiju1; re-run only for v2/role-mix experiments (the in-flight 09452e34 is exactly that).
3. The queue watcher — launchctl load the plist and it drains the rest as each model's IQ-ladder feeder completes. Nothing else fires until a feeder mints f16+imatrix, so it's self-pacing and safe to enable now.

Blocked / needs a decision

• All models except qwen0.5b/7b: need an IQ-ladder feeder run first (mints f16+imatrix). That's ~10 ladder runs, serial, ~1-2 days of the budget. The Mistral one is already auto-following Codestral.
• codestral-22b: (a) its ladder failed at eval-sweep (timeout — since fixed 4h→8h) so f16 was never persisted; needs a clean ladder rerun. (b) It has no dense .knot base (only the OpenAI gateway qspec) and no argmax qspec — a base knot must be built and a gate derived from f16 before it can KAIJU. Flagged for a decision, not auto-queued.
• 22B/31B: bump the encode-step timeout (57600s→115200s) and, for 31B, diskSizeGb; otherwise the encode step times out mid-run.
• 70B-class: diskSizeGb 400 + the encode is ~90h serial — defer to the hexacode fast-decoder v2.
• loki-8b-erotica: confirm the HF safetensors source exists before queuing (it's a custom finetune; ADMITTED shows a knot but not a clean HF f16 source).
• gemma "4" sources: the repo's gemma4-* ids map to google/gemma-4-* GGUFs (ggml-org) in ADMITTED; confirm the convert path for the encode's f16 (the ladder needs safetensors, not GGUF — may require a different f16 source than the admission GGUF).

Do not launch the wave yet — two encodes are in flight and this is prepare-only. On a green light: enable the watcher (drains as feeders land) or fire qwen0.5b directly.