forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Windows GPU Embeddings Mesh — Setup / Start / Recovery

distributed-inference/WINDOWS_GPU_EMBED_RUNBOOK.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Windows GPU Embeddings Mesh — Setup / Start / Recovery

Run qwen2.5-0.5b embeddings on the GTX 1080 Ti (gaming PC 10.0.0.52) via the CUDA fat-station (WSL2) behind the gnode batched OpenAI proxy (Windows).

  • Endpoint: http://127.0.0.1:8080/v1/embeddings — use 127.0.0.1, NOT localhost. A stopped Docker container's port-forwarder squats IPv6 ::8080/::8090, so localhost (→ ::1) hits a dead listener; IPv4 127.0.0.1 is correct. (Only a Docker Desktop restart clears the squat; it needs UAC, so we just use IPv4.)
  • One-shot start: powershell -ExecutionPolicy Bypass -File C:\models\start-gpu-embed-mesh.ps1 (-Reset to wsl --shutdown first, -Build to rebuild the binary).

Why GPU + these params

  • Default cuBLAS path = Q4K→f32 host dequant (cached, resident) + f32 SGEMM, so Pascal's crippled fp16 is irrelevant — do NOT set FLUX_GPU_WEIGHT_DTYPE.
  • GNOSIS_RESONANCE_THRESHOLD=1.0 is REQUIRED: resonance<1.0 forces the per-token fallback (no batching). 1.0 = exact FFN + the weight-amortizing segmented batch.
  • FLUX_GPU_MIN_TOKENS=1: embeds do per-token forwards; default min=2 keeps 1-token GEMVs on CPU. 1 routes them to the GPU.
  • Build only needs CUDA 12.x (cudarc pinned cuda-12040; CUDA 13 drops Pascal).

Prereqs (one time, in WSL2 Ubuntu)

# as root (wsl -u root): CUDA 12.6 libs + build tools (NVIDIA wsl-ubuntu repo)
apt-get install -y build-essential cuda-libraries-12-6 cuda-libraries-dev-12-6
# Rust (rustup) as the default user if missing. Models in C:\models:
#   qwen2.5-0.5b-instruct.knot, qwen-tokenizer.gguf

Build the CUDA fat-station (WSL2)

export PATH="$HOME/.cargo/bin:$PATH" CUDA_PATH=/usr/local/cuda-12.6
export CARGO_TARGET_DIR="$HOME/ft-target"             # native ext4, NOT the /mnt 9p mount
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:/usr/lib/wsl/lib
export RUSTFLAGS="-L /usr/local/cuda-12.6/lib64"
cd /mnt/e/monorepo/open-source/gnosis/distributed-inference
cargo build --release --locked -p distributed-inference --bin fat-station --features cuda

Start — fat-station (WSL2, :8090)

export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:/usr/lib/wsl/lib
export TOKENIZER_GGUF_PATH=/mnt/c/models/qwen-tokenizer.gguf
export RAYON_NUM_THREADS=4
export FLUX_GPU_MIN_TOKENS=1
export GNOSIS_RESONANCE_THRESHOLD=1.0          # <-- enables the batched path
nohup ~/ft-target/release/fat-station \
  --knot /mnt/c/models/qwen2.5-0.5b-instruct.knot --role both --layers 0..24 \
  --port 8090 --gossip-port 9090 --advertise-host 127.0.0.1 \
  --node-id win-gpu-embed --model-id qwen05-dense.knot > /mnt/c/models/ft-gpu.log 2>&1 &

Confirm: [cuda] GPU matmul backend ACTIVE on device 0 in C:\models\ft-gpu.log.

Start — gnode batched proxy (Windows, :8080)

$env:PROXY_BATCH_WINDOW_MS='10'; $env:PROXY_MAX_QUEUE='64'
$env:PROXY_WAIT_DEADLINE_MS='120000'; $env:PROXY_FORWARD_MS_HINT='2000'
$env:FAT_STATION_URL='http://127.0.0.1:8090'; $env:PORT='8080'; $env:MODEL_NAME='qwen05-dense'
# NO PROXY_MAX_BATCH — the startup probe auto-sets 32 when /hidden-embed-batch answers.
cd E:\monorepo
node open-source/gnosis/bin/gnode.js run open-source/gnosis/distributed-inference/openai-proxy.mjs

Confirm boot log: [openai-proxy] /hidden-embed-batch present -> MAX_BATCH=32 (auto).

Proxy gate params — why relaxed

The default admission shed (ewma=8000ms, deadline=20000ms, queue=16) sheds a 32-input burst with 429 (each input tokenizes separately → enqueues in waves → ~5 batches → projected wait trips the deadline). The values above (deeper queue, real ~2s ewma seed, long deadline) let legitimate batch bursts through.

Verify

curl -s -m10 -o NUL -w "8090=%{http_code} 8080=" http://127.0.0.1:8090/health; \
curl -s -m10 -o NUL -w "%{http_code}\n" http://127.0.0.1:8080/health
# 32-input batch (cold first call ~60-90s GPU upload; warm single ~0.5s):
node -e "fetch('http://127.0.0.1:8080/v1/embeddings',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({input:Array.from({length:32},(_,i)=>'message '+i)})}).then(r=>r.json()).then(j=>console.log('rows',j.data?.length))"

Recovery — CUDA_ERROR_UNKNOWN after sleep/resume

Symptom: ft-gpu.log shows htod_sync_copy failed: DriverError(CUDA_ERROR_UNKNOWN) then a cudarc panic; the fat-station dies. The box slept and WSL GPU-PV got poisoned. Fix: wsl --shutdown, then restart the fat-station + proxy (or run start-gpu-embed-mesh.ps1 -Reset). nvidia-smi working in WSL ≠ a clean CUDA context; the shutdown is what clears it.

Autostart on boot/logon

A scheduled task GPU-Embed-Mesh-Autostart runs start-gpu-embed-mesh.ps1 at logon (1-min delay, user context so it has WSL GPU access; the script is idempotent so it won't disturb an already-running mesh). Register it once:

$u=whoami
$a=New-ScheduledTaskAction -Execute powershell.exe -Argument '-NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File C:\models\start-gpu-embed-mesh.ps1'
$t=New-ScheduledTaskTrigger -AtLogOn -User $u; $t.Delay='PT1M'
$p=New-ScheduledTaskPrincipal -UserId $u -LogonType Interactive -RunLevel Limited
$s=New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 15)
Register-ScheduledTask -TaskName 'GPU-Embed-Mesh-Autostart' -Action $a -Trigger $t -Principal $p -Settings $s -Force

Test without rebooting: Start-ScheduledTask -TaskName GPU-Embed-Mesh-Autostart then Get-ScheduledTaskInfo -TaskName GPU-Embed-Mesh-Autostart (LastTaskResult 0 = ok). Logs: C:\models\start-gpu-embed-mesh.log (orchestration), proxy.out.log/proxy.err.log. NOTE: the script MUST be pure ASCII — Windows PowerShell 5.1 reads -File as ANSI, so an em-dash/smart-quote becomes mojibake and the script fails to parse. NOTE: a logon trigger does NOT fire on sleep/resume (only on logon). After a resume, GPU-PV may be poisoned (CUDA_ERROR_UNKNOWN) — run start-gpu-embed-mesh.ps1 -Reset, or add a second task triggered on the resume event (Power-Troubleshooter id 1) that calls the script with -Reset.

Stop

# proxy:
Get-CimInstance Win32_Process -Filter "Name='node.exe'" | ? { $_.CommandLine -match 'openai-proxy|gnode' } | % { Stop-Process -Id $_.ProcessId -Force }
# fat-station:
wsl -d Ubuntu -- bash -lc "pkill -f 'fat-station.*--port 8090'"

Performance (qwen2.5-0.5b on 1080 Ti)

Warm single ~0.5s (vs ~2s old CPU node). Batch ~1.5–1.7x — 0.5b is host-overhead bound, so batching is modest here; it compounds on bigger models (1.5b = 2.1x). For a real batched win, run a larger embedding model on a 24GB GPU.