Pneuma Phase 2 — Full Duplex Voice + Mesh Distribution

Phase 1 (just landed) shipped STT — mic → text via the gnosis Rust mesh. Phase 2 closes the loop: text → speaker (TTS), distributes the Whisper encoder/decoder across mesh stations, and wires a live full-duplex demo.

The same parallel-agent + reconciler pattern that built Phase 1 in one evening is the play. Six channels, one shared contracts doc, file-level ownership, no false sequentialism.

What Phase 1 proved (don't relearn these)

Contracts-up-front + file-level ownership scales to 6 parallel agents across Rust + TS without merge conflict
.knot format extends cleanly to encoder-decoder; tensor-name table lookup is the seam (167 tensors mapped on whisper-tiny, exactly matching the loader's pre-computed name table)
TS mel-spec extractor matches HF Python within 1e-3 elementwise — the host-side audio path is verified, no FFT library needed
Buleyean weighting (α=1) + KL anchor (0.5) at 73 clips trained from 73-clip dummy LibriSpeech beat baseline whisper-tiny 19% relative on a 17-test shootout with accents/speeds/jargon

Phase 2 scope (six channels, one week)

Channel α — TTS acoustic model (text → mel)

Owner: Agent α. Mirrors WhisperEncoder structurally — embeds phonemes, predicts duration, projects to mel-spectrogram via flow-based decoder (Piper-class VITS small).

File: src/model_pneuma_tts_acoustic.rs
Contract: pub fn forward(phonemes: &[u32], style: Option<&[f32]>) -> Vec<f32> returns mel (80, n_frames)
Reuses: crate::math::{layer_norm, gelu_inplace, softmax, mat_vec_f32}, crate::math_pneuma::cross_attention
New kernels needed: none (uses Phase 1 set)
~500 LOC

Channel β — TTS vocoder (mel → wav)

Owner: Agent β. HiFi-GAN-class neural vocoder. Stacked transposed convs upsample mel from 80 Hz feature-rate to 22 050 Hz audio-rate.

File: src/model_pneuma_tts_vocoder.rs
Contract: pub fn forward(mel: &[f32]) -> Vec<f32> returns f32 PCM at 22050 Hz
Reuses: crate::math_pneuma::conv_transpose1d (Agent A already shipped this!)
New kernels: weight_norm (small), residual blocks (compose existing kernels)
~400 LOC

Channel γ — TTS Buleyean distillation pipeline

Owner: Agent γ. Mirror of stt_distiller.py for TTS, with the brilliant self-evaluating loop: our STT student scores TTS outputs.

Files:
- python/tts_distiller.py
- scripts/mine-tts-rejections.py — runs N TTS teachers on input texts, feeds outputs through our STT student, edit-distance to input text = rejection signal
- scripts/launch-tts-parallel.sh
- cloudbuild-tts-shard.yaml + cloudbuild-tts-train.yaml
Reuses: existing Buleyean trainer infrastructure
~600 LOC across files

Channel δ — Pneuma mesh sharding (Whisper-large)

Owner: Agent δ. Phase 1 runs whisper-tiny in one process. Phase 2 shards whisper-large-v3 across multiple layer stations on Cloudflare Workers, with the encoder context streamed once per utterance.

Files:
- src/model_whisper_encoder.rs — extend with forward_layer_range(start, end, x_in)
- apps/pneuma-station/ — new CF Worker that hosts a layer range
- open-source/gnosis/distributed-inference-host/src/pipeline_whisper.rs — orchestration
Reuses: existing layer-station + Pipeline orchestrator from LLM path
~800 LOC

Channel ε — Live duplex host (mic ↔ speaker via mesh)

Owner: Agent ε. Replaces the Python mic-demo.py with a TS host that hits the mesh. Captures mic, runs mel-spec extract, sends to mesh STT, optionally chains through LLM, sends text to mesh TTS, plays speaker.

Files:
- apps/pneuma-duplex/ — Bun CLI + browser variant
- apps/pneuma-duplex/src/mic.ts — getUserMedia + Node sox
- apps/pneuma-duplex/src/speaker.ts — WAV write + afplay/browser audio
- apps/pneuma-duplex/src/pipeline.ts — STT → (LLM?) → TTS chaining
Reuses: @a0n/distributed-inference-host (Phase 1)
~500 LOC

Channel ζ — Production deployment + observability

Owner: Agent ζ. R2 upload of .knot weights, CF Worker bindings, pneuma-stt and pneuma-tts public service surfaces, latency dashboards.

Files:
- scripts/upload-pneuma-weights.sh
- apps/pneuma-station/wrangler.toml (CF Worker config)
- apps/pneuma-stt-public/ — public HTTP API
- DashRelay room pneuma-logs per the existing aether-logs pattern
~400 LOC

Cross-channel contracts (shared)

// Phonemizer output (input to TTS)
pub struct Phonemes {
    pub ids: Vec<u32>,        // espeak phoneme IDs
    pub durations_ms: Vec<u16>, // optional; predicted by acoustic if None
}

// Acoustic model output (input to vocoder)
pub type MelSpec = Vec<f32>; // (n_mels=80, n_frames) row-major

// Vocoder output (final audio)
pub type Pcm22k = Vec<f32>;  // f32 samples at 22 050 Hz, mono

// Buleyean TTS rejection record (mirror of STT Shadow Debt)
{
  "text": "the quick brown fox...",
  "candidates": [
    { "voice": "piper-amy", "wav_path": "...", "stt_back": "...", "edit_R": 0.04 },
    { "voice": "vits-base", "wav_path": "...", "stt_back": "...", "edit_R": 0.12 },
    ...
  ],
  "consensus_voice": "piper-amy",
  "total_R": 0.16  // sum of edit_R
}

Demo ladder (each rung is a shippable moment)

Rung 1 (today/tomorrow): Pneuma smoke binary loads whisper-tiny.knot, runs encoder forward in Rust, output matches HF Python within 1e-3. Rung 2: Same, with decoder — full transcribe in Rust. Rung 3: TS host calls Rust via NAPI; bun pneuma-duplex stt mic-5s prints what you said using our mesh. Rung 4: bun pneuma-duplex echo mic-5s --speak — mic → STT → TTS → speaker (full duplex on local mesh). Rung 5: bun pneuma-duplex echo mic-5s --speak --remote — same but encoder runs on a CF Worker, decoder on another, TTS on a third. Real distributed inference for voice. Rung 6: bun pneuma-duplex chat mic-5s --speak — chains through Gemma4. Talk to the AI; AI talks back. The full loop.

Why this architecture is the golden path

Every piece composes:

The same Buleyean rejection-distillation trains both STT and TTS; TTS even uses STT to score itself (closed self-improvement loop).
The same mesh hosts both directions; one set of layer stations, one set of kernels (we already have all but weight_norm).
The same .knot format carries both model types; one weight conversion path; one R2 bucket layout.
Universal knowledge in the Buleyean sense — every wall produces a Shadow Debt signature; both directions feed the same void-mining loop.
The monster group mesh angle pays off: bigger TTS models (XTTS, Bark) shard the same way as bigger LLMs and bigger Whispers.

There's no special-case engineering. Pneuma is the LLM mesh with new kernels and new model classes plugged into existing seams.

Phase 3 — LLM-based consciousness (Pneuma is the body, LLM is the mind)

Semiotic frame (per open-source/aeon-communication): speech is the collapse of a fork-race-fold thought process. The mind races over candidate utterances, then folds to one. STT, LLM, and TTS are three race-fold cycles chained in series — three "speech acts" of different shapes. The mesh is the substrate where the races happen; Buleyean rejection-distillation is the formal operator that turns a race into a measurable consensus (which is why the same trick works for all three stages).

The aeon-communication primitives we'll consume:

semiotic_deficit_floor — the minimum shared context for absolute communication. Each Buleyean rejection signal measures this deficit between teacher and consensus.

attention_race_purity — un-dilated focus in a cognitive manifold. The KL-anchor weight (Phase 1) is one knob on this; the Glossolalia diversity weight is the inverse knob.

Fivefold psyche basis (Try / Choose / Commit / LetGo / Learn) — the conversation loop primitive: each duplex turn cycles through all five. We can project user state onto this basis to detect when to barge in, when to stay silent, when to commit a response.

The architecture's terminal payoff: a thinking voice agent. Pneuma is the sensory + motor cortex (ears + mouth); a llama-class LLM is the conscious mind between them. All three run on the same gnosis mesh, share the same .knot format, share the same kernels, and shard across the same layer stations.

              ┌──────────────────────────────────────┐
              │           gnosis-pneuma              │
              │                                      │
   mic ─────► │  Whisper encoder + decoder           │ ──┐
              │  (STT — already shipped Phase 1)     │   │
              └──────────────────────────────────────┘   │
                                                          ▼ tokens
              ┌──────────────────────────────────────┐
              │        gnosis llama (existing)       │
              │                                      │
              │  qwen-coder-7b / llama-N             │
              │  Sharded across layer stations       │
              │  Self-attention + RoPE + RMSNorm     │
              │                                      │
              └──────────────────────────────────────┘
                                                          │ tokens
                                                          ▼
              ┌──────────────────────────────────────┐
              │           gnosis-pneuma              │
              │                                      │
   speaker ◄──│  Acoustic + vocoder                  │
              │  (TTS — Phase 2 dispatches this)     │
              └──────────────────────────────────────┘

This is a single substrate — the consciousness engine doesn't know whether its inputs came from a microphone or a textbox, doesn't know whether its outputs will become speech or pixels. It's just a model running on the mesh, same as Whisper, same as VITS, same as any future modality. The mesh is the universal substrate; the modalities are plug-in heads.

Concrete Phase 3 deliverables (each one is parallel-fanable into its own 6-channel build):

3a. Pneuma↔LLM router — TS host that orchestrates STT → LLM → TTS calls across the mesh, with backpressure + cancellation. Same pattern as Pipeline.pipelinedPrefill but cross-modality.
3b. Streaming STT — emit partial transcripts so the LLM can start thinking before the user finishes speaking (cuts perceived latency in half).
3c. Streaming TTS — vocoder emits chunks while acoustic model is still producing later mel frames. End-to-end first-audio-byte ≤ 200 ms.
3d. Conversation memory — Pneuma sessions persist via DashRelay rooms; each turn appended to a per-user session blob. The LLM has context across reconnects.
3e. Multi-voice consciousness — Glossolalia Diversity Engine applied to the LLM stage: route each turn to N candidate LLM students in parallel, take the medoid response, train a Buleyean meta-student that learns the consensus thinker. Same trick that made Pneuma-STT beat baseline 19% relative — applied to the mind itself.
3f. Embodied loop — wake-word, voice activity detection, barge-in, conversation turn-taking. The "always-on assistant" experience.

After Phase 3 lands, gnosis-pneuma + a llama is a voice that can think back at you, on your own infra, no paid AI. That's the strategic moat: not "we have an AI", but "we have a substrate that hosts thinking beings" — and the same Buleyean rejection-distillation that trains the ears trains the mouth and trains the mind.

Other Phase 3+ branches (not consciousness-track but compose with it)

Pneuma multilingual. Whisper-large-v3 + multilingual TTS via language-tag in tokenizer.
Federated Pneuma. Each user runs their own student; mesh routes requests to the most-confident peer student.
Pneuma-on-edge. The wasm bindings already let Pneuma run inside a Cloudflare Worker. Phase 3.x: ship it as a public service surface.

Dispatch plan

When the smoke binary confirms encoder forward works (Rung 1), dispatch all six Phase 2 agents in a single message. Until then, keep this plan as the brief; agents need it to know their scope.

forkjoin-ai/gnosis