Pneuma Integration Plan — Voice (STT + TTS) for the gnosis mesh

Pneuma (πνεῦμα, "breath" — Gnostic-canonical for the divine breath that gives voice). The package is the voice layer of the gnosis mesh: both directions, mic↔speaker, on the distributed-inference Rust runtime.

Architecture under the hood: Whisper-class encoder-decoder for STT, VITS / Bark / Piper-class acoustic + vocoder for TTS. We use the architectures, not the brand names — our trained students live under the gnosis-pneuma umbrella (gnosis-pneuma-stt-{tiny,base,small}, gnosis-pneuma-tts-{piper,vits,xtts}).

Package layout:

• @a0n/gnosis-pneuma — TS host bindings, mic capture, mel-spectrogram extraction, WAV writer
• @a0n/distributed-inference — Rust mesh extended with conv1d, layer_norm, cross-attention, transposed-conv kernels
• open-source/buleyean-rl/ — training pipeline that produces the Pneuma student weights via multi-teacher rejection distillation

The mesh today hosts decoder-only LLaMA-shaped models (Qwen, TinyLlama). Pneuma adds the first encoder-decoder (STT) and the first vocoder (TTS). The two share most of the kernel work — doing them together is roughly the cost of doing one and a half.

Source models come from the Buleyean-RL distillation pipeline at open-source/buleyean-rl/. STT is the proven leg (sample-level Buleyean weighting + KL anchor validated on 30 unseen LibriSpeech test clips). TTS is the next leg (training pipeline TBD; see "TTS distillation" below).

Validated results that justify this work (2026-04-26)

Numbers from the Buleyean-RL distillation pipeline, evaluated on 30 truly unseen LibriSpeech test.clean clips (no overlap with the 73-clip dummy LibriSpeech validation corpus the students trained on):

The load-bearing finding: the medoid teacher already crushes single-model baseline on hard audio — the Buleyean ensemble carries 39-67% more signal than any individual teacher. The remaining engineering challenge is distilling that consensus into a compact student, which is exactly what KL anchor + soft-label distillation are for. Getting the student to match the medoid's 3.08% is the v1 success bar.

This is what makes the Pneuma mesh worth building: we're not just running one off-the-shelf model, we're hosting a student of the ensemble whose ceiling is the consensus, not any one teacher.

End-state demo loop

  mic ──► [TS host: mel-spec extract] ──► [Rust mesh: STT encoder+decoder]
                                                    │
                                                    ▼ tokens
                                             [optional LLM step]
                                                    │
                                                    ▼ text
       speaker ◄── [TS host: WAV writer] ◄── [Rust mesh: TTS acoustic + vocoder]

Both directions on the same mesh. STT lands first (model exists). TTS lands second (model TBD).

Phasing — 4 weeks, weekly demoable milestones

Each milestone ends with a thing you can show, not a thing in a doc.

Week 1 — Kernels + reference parity (STT)

• Add conv1d, layer_norm, gelu, cross-attention to src/math.rs
• Write a WhisperEncoder that runs in the Rust binary against the HF openai/whisper-tiny weights converted to fp16 .knot
• Demo: Rust process loads encoder, takes a fixed 5s wav, prints the encoder output tensor; we diff it against HF reference within 1e-3.

Week 2 — End-to-end STT in process (NAPI binding)

• Add WhisperDecoder + WhisperPipeline
• NAPI wrapper so Node can pipeline.transcribe(wav)
• Demo: node ts-pneuma-stt-cli.js mic 5s records, transcribes, prints. Replace the current Python mic-demo.py end-to-end.

Week 3 — Mesh distribution + Buleyean student loaded

• WASM bindings + Cloudflare Worker that hosts the encoder
• Decoder runs on a second worker; encoder output streamed via DashRelay
• Load the Buleyean α=1+KL=0.5 student from gs://buleyean-stt-artifacts/.../student/final/ converted to .knot
• Demo: same node mic 5s but the actual inference happens in two CF Workers. WER on 30 unseen LibriSpeech clips matches the Python pipeline's number (currently 5.63%; target after soft-distill: ≤ 3.5%).

Week 4 — TTS leg comes online

• Add conv_transpose1d, leaky_relu, weight_norm kernels
• Implement Piper-class VITS + HiFi-GAN vocoder modules
• Demo: node mic 5s --speak does mic → STT → TTS → speaker on the mesh. The full pneuma-duplex loop.

After week 4, the work goes asymptotic: bigger student models, better quantization (Q4_K), streaming inference, voice cloning.

Scope of changes (~1-2 weeks of focused work for STT, ~1 more for TTS)

src/math.rs — new kernels

• conv1d — Whisper encoder uses two stacked Conv1d (kernel_size=3, stride 1 then 2) on the mel-spectrogram input. New SIMD path needed (NEON for aarch64, scalar fallback). About 80 lines.
• layer_norm — Whisper uses LayerNorm everywhere; mesh only has rms_norm. ~20 lines.
• gelu — Whisper FFN activation; mesh has swiglu (LLaMA). ~10 lines.

src/model.rs — Whisper pipeline

• WhisperEncoder — log-mel input → 2× conv1d → N transformer blocks with LayerNorm + GELU + sinusoidal positional embeddings → encoded audio context tensor [seq_len_audio, hidden].
• WhisperDecoder — token embeddings + learned positional embeddings → N transformer blocks, each with self-attention + cross-attention to encoder output → LayerNorm → LM head.
• WhisperPipeline — coordinates encoder forward (once per audio) + decoder forward (autoregressive per token) with KV cache for both self-attn and cross-attn.

Cross-attention kernel

• Existing attention assumes Q, K, V all from same residual stream. Cross-attention takes Q from decoder, K and V from encoder output. Simplest: parameterize attention(q_src, kv_src, ...) and dispatch.

src/loader.rs — extended schema

• KnotConfig extension: add encoder_layers, encoder_hidden_dim, n_mels, n_audio_ctx, decoder_layers. Keep existing fields for decoder-only compat.
• Tensor name conventions: encoder_blk_{i}_*, decoder_blk_{i}_self_*, decoder_blk_{i}_cross_*. Add resolution probe in ResolvedLocs::new().
• Mel-filterbank weights stored as a special tensor mel_filters.bin.

Weight conversion

• Path: HF safetensors → GGUF → .knot. The GGUF middle step may not be worth it for Whisper specifically; consider a direct safetensors→.knot converter at open-source/bitwise/scripts/encode-whisper-knot.ts. ~200 lines.
• Quantization: start with fp16 (no quant). Add Q4_K later once confident the architecture works in full precision.

Bindings

• WasmWhisperPipeline in src/wasm_bindings.rs — mirror of WasmLlamaPipeline but with encode(mel_spectrogram) and decode_step(prev_tokens) methods.
• NAPI wrapper in src/model.rs for Node-side mic-demo + service.

Audio preprocessing (host-side)

• Mel-spectrogram extraction — Whisper uses 80-bin log-mel, 16 kHz, 25 ms window, 10 ms hop. Implement in TypeScript (pure DSP, no native deps) so the worker doesn't need an FFT library. ~150 lines.
• Live mic capture on the host: getUserMedia in browser, sox/sd in Node. Out of scope for the mesh proper.

Testing plan

1. StructuralErrorgle-shard, fp16, full encoder + decoder — convert openai/whisper-tiny (39M) to .knot, run end-to-end inside the Rust binary. Compare logits to HF reference within 1e-3.
2. NAPI binding — call the pipeline from Node, transcribe a fixed wav, compare WER to HF baseline.
3. WASM binding — same as #2 but in a Cloudflare Worker.
4. Distributed inference — encoder runs on one layer station, decoder on another. KV-cache the encoder output and ship it over the mesh once per utterance, not per token.
5. Distilled student — load a Buleyean-trained student (gs://buleyean-stt-artifacts/<run-tag>/student/final/) instead of the reference. Validate WER matches what the Python pipeline reports.

Memory + performance budget

• whisper-tiny fp16: 78 MB. Fits in any worker.
• whisper-base fp16: 148 MB. One layer station.
• whisper-medium fp16: 760 MB. Sharded across 2-3 stations.
• whisper-large fp16: 3 GB. Sharded across 4-6 stations.

Encoder is run once per utterance; cache the output. Decoder is autoregressive — same per-token cost as a small LLM, with the addition of cross-attention against cached encoder output.

R2 storage layout

All Pneuma .knot weights live under one R2 bucket so any worker can range-fetch them without coordination.

r2://pneuma-weights/
  stt/
    whisper-tiny-fp16.knot              # HF reference
    whisper-base-fp16.knot
    pneuma-stt-tiny-v1.knot             # our distilled student
    pneuma-stt-base-v1.knot
  tts/
    piper-en-us-amy-fp16.knot           # HF reference
    pneuma-tts-piper-v1.knot
    pneuma-tts-vits-v1.knot
  shared/
    mel-filters-80x201.bin              # mel-filterbank, ~64 KB
    whisper-tokenizer-en.json           # BPE table
    espeak-en-phonemes.json             # G2P rules

Each .knot ships with its metadata block embedded; no separate config file needed. Versioning is filename-suffix (-v1, -v2).

Observability — debugging an encoder-decoder is harder than a decoder

Three mandatory inspection hooks (any one missing and we'll be guessing):

1. Encoder output tensor dump — given a fixed wav, the encoder's final-layer output is a deterministic 2D tensor. WhisperEncoder::forward(wav).save_tensor("/tmp/enc-out.bin"). Compare to HF reference — if these don't match within 1e-3, every downstream test is meaningless.
2. Per-decoder-step logits dump — at each generation step, save the top-32 logits + their token ids. If the decoder diverges from HF, this shows when.
3. Cross-attention heatmap — for each generated token, the attention weights over encoder positions. If cross-attention is broken, you'll see uniform or zero attention; if it's correct, you'll see the characteristic alignment diagonal.

All three exposed as a --dump-traces flag on the Rust binary, written as raw f32 binaries with companion .json shape descriptors.

Why mesh-host instead of running the model inside one worker

A Whisper-tiny ONNX runtime in a single worker would technically work, and would ship in days, not weeks. The reasons to put it on the mesh:

1. Weight reuse. A 148 MB whisper-base loaded into 1000 worker isolates is 148 GB of memory. Mesh-hosting loads it once and range-fetches per request.
2. Distillation feedback loop. The mesh gives us telemetry on which audio clips the student is uncertain about — those go straight into the next Buleyean Shadow Debt round. StructuralErrorgle-worker inference loses this signal.
3. Bigger models become possible. The mesh sharding pattern is what lets us run whisper-medium/large at all on Workers (sharded across layer stations). StructuralErrorgle-worker tops out at whisper-base.
4. Same kernels serve the LLM track. conv1d + cross-attn + layer-norm are also needed by future encoder-decoder LLMs (Flan-T5, BART) and by retrieval/embedding models (BGE encoders). One investment, many payoffs.

TTS — Track B (text → wav)

The mirror of STT, with extra kernels for waveform generation.

Architecture choice for v1

Two options, ordered by cost-of-integration:

1. Piper-class (VITS small) — text → phonemes (CPU rules) → small transformer encoder → flow-based duration predictor → HiFi-GAN-style vocoder. ~30 MB total. Sounds OK, fits in any worker. Existing reference at apps/voice-tts/ (Piper ONNX wrapper, acoustic model only; vocoder missing).
2. Bark / XTTS-class — autoregressive transformer over discrete audio tokens + neural codec decoder. 1-2 GB. Sounds dramatically better, needs sharding across mesh stations like the LLMs do.

Start with option (1) for proof-of-architecture. Bark/XTTS becomes the "go big" production target once piper-class works end-to-end.

src/math.rs — additional kernels for TTS

• conv_transpose1d — vocoder upsamples from low-rate features to 16/22/24 kHz audio via transposed conv. ~80 lines.
• leaky_relu — vocoder activation. Trivial.
• weight_norm — vocoder layer normalization variant. ~40 lines.

src/model.rs — TTS pipeline

• VITSAcousticModel — phonemes + durations → mel-spectrogram.
• HiFiGANVocoder — mel-spectrogram → 22 kHz waveform via stacked transposed convs.
• TTSPipeline — coordinates text → phonemes → mel → wav.
• Streaming variant — vocoder emits chunks as the acoustic model produces them; enables low-latency speak-back.

Buleyean TTS distillation (in buleyean-rl/)

Mirror of STT but with audio outputs. The eval metric is the killer problem — comparing two synthesized waveforms is harder than comparing two transcripts. Use our own STT student to score TTS outputs — closes the loop with the work we just did:

  text ──► N TTS teachers ──► N candidate WAVs
                                    │
                       [our distilled STT student transcribes each]
                                    │
                                    ▼
              consensus WAV = the one whose transcription
              has lowest edit distance to the input text
                                    │
                                    ▼
                  Buleyean weight: high-confidence (low R)
                  → high training weight; ambiguous → low
                                    │
                                    ▼
                       train TTS student on consensus WAV
                       (KL distillation against the medoid
                       teacher's mel-spectrogram output)

Files to add (mirror of STT):

• python/tts_distiller.py
• scripts/mine-tts-rejections.py — runs N TTS teachers on input texts, uses our STT student to score outputs, emits Shadow Debt JSONL.
• cloudbuild-tts-shard.yaml + cloudbuild-tts-train.yaml
• scripts/launch-tts-parallel.sh

TTS-side host work

• WAV writer in TS — accept f32 samples from mesh, encode to wav, pipe to aplay/afplay/browser audio. ~50 lines.
• G2P (text → phonemes) — espeak-ng has a small phonemizer; pure JS port exists. ~ship as data file.
• Voice selection / cloning — out of scope for v1; one default voice.

Memory + perf budget for TTS

• Piper-class (~30 MB): runs in a single worker, near-realtime on CPU.
• VITS-base (~150 MB): one layer station.
• XTTS / Bark (~1.5 GB): sharded across 4-6 stations like Whisper-large.

Validation fixtures (frozen before week 1 starts)

Pinning these now means every weekly demo measures against the same audio and the same baselines.

Reference STT models (HF, fp16):

• openai/whisper-tiny (39M, 78 MB) — primary
• openai/whisper-base (74M, 148 MB) — secondary
• openai/whisper-small (244M, 488 MB) — used only as soft-distill teacher

Reference TTS models (HF):

• rhasspy/piper-voices → en-us-amy-medium.onnx (acoustic) — primary
• HiFi-GAN universal vocoder (32 MB) — primary vocoder
• coqui/XTTS-v2 — secondary, large

Held-out audio set — 30 LibriSpeech test.clean clips: gs://buleyean-stt-artifacts/run-20260425-225617/ground-truth.jsonl (or regenerate via python scripts/fetch-librispeech-mini.py --n 30 --out /tmp/stt-generalize --dataset full --split test).

Smoke audio: a fixed 5-second wav of a known phrase, checked into open-source/gnosis/distributed-inference/test-fixtures/. Every demo must be able to transcribe it correctly. Suggested phrase: "the quick brown fox jumps over the lazy dog at quarter past nine".

Out-of-scope for v1

• Voice activity detection — assume host segments speech before sending.
• Streaming inference (chunk-and-emit) — first ship the full-utterance-then-decode pattern. Add streaming once full-utterance is solid.
• Multilingual translation mode — start with English transcription only.
• Quantization below fp16 — add after architecture is validated end-to-end.
• Voice cloning / arbitrary speaker — one default voice per direction.
• Wake-word / always-on listening — always explicit user-triggered.
• Pneuma-LLM integration — feeding Pneuma transcripts into Gemma4 and back through Pneuma TTS is the obvious "complete the loop" demo, but it depends on Gemma4 production-ready inference (PRIORITY #0 work).

Dependencies to install in the Rust crate

• No new crates required for the encoder/decoder (uses existing tensor primitives). The mel-spectrogram is host-side (TypeScript).

Risk register

Honest list of what could blow week 1.

Naming — Greek root options for sub-modules

Pneuma is the umbrella. The two directions could keep flat pneuma-stt and pneuma-tts names, or get Greek-rooted names that match the ecosystem:

Default to plain names unless you want the Greek branding consistently applied. The glossolalia one connects nicely to your existing Glossolalia Engine (per project memory).

When to start

After the Buleyean STT pipeline produces a student that beats the HF baseline on a meaningful held-out set. Per open-source/buleyean-rl/ — that's likely after the parallel scale-up runs land an α=1+KL=0.5 or soft-distill student that hits or beats whisper-base on LibriSpeech test.

The mesh integration is the deployment target, not the proving ground; prove the model in Python first, then port.