forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

GKQ Inference: Hot Path (Memory-Resident) vs Fallback

distributed-inference/HOT_PATH_DEPLOYMENT.md
forkjoin-ai/gnosis
Repo overview Open in workbench

GKQ Inference: Hot Path (Memory-Resident) vs Fallback

Overview

The Gnosis Knot Quantization (GKQ) inference server supports two deployment modes:

  1. Hot Path (Default): Memory-resident quantized weights with GPU reconstruction cache
  2. Fallback: CDN streaming for resource-constrained environments

The hot path is default because it eliminates cold-start latency and provides sub-millisecond weight access.

Hot Path: Memory-Resident with GPU Cache (Default)

How It Works

Pod startup:
  1. Init container stages quantized weights (20 MB) to /weights/quantized
  2. Server starts → load_quantized_weights() loads 20 MB into RAM
  3. First inference request → get_weights(layer_id) reconstructs on GPU
  4. GPU reconstruction cached with LRU eviction (8 GB default)
  5. Subsequent requests hit GPU cache or reconstruct if evicted

Configuration

env:
  MEMORY_RESIDENT_MODE: "true"      # Enable hot path (default)
  WEIGHTS_PATH: "/weights/quantized/llama-70b-gkq.knot"
  GPU_CACHE_SIZE_MB: "8000"          # 8 GB GPU reconstruction cache
  MAX_AGE_SECONDS: "3600"            # Evict weights older than 1 hour
  USE_CDN_FALLBACK: "false"          # Disable fallback (default)

Performance

  • Startup: ~30 seconds (includes weight load + pod readiness probes)
  • First inference: <100 ms (GPU reconstruction) + inference latency
  • Subsequent inferences: <1 ms weight access (cache hit) + inference latency
  • Memory usage: 16–24 GB per pod (20 MB quantized + 8 GB cache + buffers)
  • GPU memory: Peak ~9 GB (working set during reconstruction)

Advantages

✅ Sub-millisecond weight access for subsequent requests
✅ Zero cold-start latency after first reconstruction
✅ Optimal for high-throughput inference (many requests)
✅ Bandwidth-efficient (no repeat downloads)

When to Use

  • Distributed inference across multiple regions
  • High-throughput serving (100+ requests/sec per pod)
  • GPU availability in deployment (A100, H100)
  • Cost-sensitive deployments (elimination of repeated CDN egress)

Fallback: CDN Streaming

How It Works

Pod startup (CDN fallback mode):
  1. Init container downloads weights from CDN on-demand
  2. Server starts → stream weights from CDN as needed
  3. No local caching (weights fetched per-request)
  4. Falls back to lazy loading if memory constraints hit

Configuration

env:
  MEMORY_RESIDENT_MODE: "false"     # Disable hot path
  USE_CDN_FALLBACK: "true"          # Enable CDN fallback
  CDN_WEIGHTS_URL: "https://cdn.forkjoin.ai/gnosis/weights/llama-70b-gkq.knot"
  GPU_CACHE_SIZE_MB: "1000"         # Smaller cache for resource-constrained envs

Performance

  • Startup: ~5 seconds (no weight download, minimal preparation)
  • First inference: 50–100 ms (CDN fetch) + <100 ms (reconstruction) + inference
  • Subsequent inferences: 50–100 ms (CDN fetch if not cached) + <1 ms (cache hit)
  • Memory usage: 4–8 GB per pod (no resident weights)
  • Bandwidth: ~20 MB per pod startup (from CDN)

Disadvantages

❌ Repeated CDN downloads if weights aren't cached
❌ Higher latency (network + reconstruction)
❌ Bandwidth cost ($0.02–0.05 per GB egress)
❌ Not suitable for real-time inference

When to Use

  • Resource-constrained edge deployments (Cloudflare Workers, Lambda)
  • Batch inference with long think time between requests
  • One-off inference tasks (fine to wait for download)
  • Testing/development (avoid large persistent caches)

Switching Deployment Modes

Enable Hot Path (Default)

kubectl set env deployment/quantized-inference \
  MEMORY_RESIDENT_MODE=true \
  USE_CDN_FALLBACK=false \
  -n gnosis-distributed

Enable CDN Fallback

kubectl set env deployment/quantized-inference \
  MEMORY_RESIDENT_MODE=false \
  USE_CDN_FALLBACK=true \
  GPU_CACHE_SIZE_MB=1000 \
  -n gnosis-distributed

Adjust GPU Cache Size

# Increase for high-concurrency workloads
kubectl set env deployment/quantized-inference \
  GPU_CACHE_SIZE_MB=16000 \
  -n gnosis-distributed

# Decrease for memory-constrained pods
kubectl set env deployment/quantized-inference \
  GPU_CACHE_SIZE_MB=2000 \
  -n gnosis-distributed

Monitoring Hot Path Health

Key Metrics to Watch

# Cache hit rate (should be >95% after warm-up)
curl -s http://service:8000/metrics | grep cache_hit_rate

# GPU cache usage (should stabilize under GPU_CACHE_SIZE_MB)
curl -s http://service:8000/stats | jq '.gpu_cache_usage_percent'

# Reconstruction count (should plateau after initial requests)
curl -s http://service:8000/stats | jq '.reconstructions'

# Eviction count (should be low with proper cache sizing)
curl -s http://service:8000/stats | jq '.evictions'

Prometheus Queries

# Cache hit rate over time
rate(gkq_cache_hits[5m]) / (rate(gkq_cache_hits[5m]) + rate(gkq_cache_misses[5m]))

# GPU cache memory pressure
gkq_gpu_cache_used_mb / gkq_gpu_cache_max_mb

# Reconstruction latency
histogram_quantile(0.95, rate(gkq_reconstruction_time_ms_bucket[5m]))

WASM-SIMD Edge Variant (Future)

For Cloudflare Workers and other WASM environments without GPU:

// Weight reconstruction in WASM-SIMD (browser/edge)
function reconstructWeights(U_q8, sigma_q8, Vt_q8, U_max, sigma_max, V_max) {
  // Dequantize: vectorized in WASM-SIMD
  const U = dequantizeVector(U_q8, U_max);
  const sigma = dequantizeVector(sigma_q8, sigma_max);
  const Vt = dequantizeVector(Vt_q8, V_max);
  
  // Reconstruct: W = U @ diag(Σ) @ V^T
  return matmul(U, diagmul(sigma, Vt));
}

Configuration for WASM edge:

env:
  MEMORY_RESIDENT_MODE: "false"
  USE_WASM_SIMD: "true"
  WASM_CACHE_KB: "512"              # Smaller cache for edge
  RECONSTRUCT_ON_EDGE: "true"       # Offload dequantization to client

Decision Tree

Does your deployment have GPU?
├─ YES → Use hot path (default)
│   └─ Cache hit rate should be >95%
│   └─ GPU cache usage should stabilize
├─ NO → Use CDN fallback
│   └─ Edge deployments use WASM-SIMD (future)
└─ Unsure? → Start with hot path, monitor metrics
   └─ If memory pressure > 80%, reduce GPU_CACHE_SIZE_MB
   └─ If cache hit rate < 80%, increase GPU_CACHE_SIZE_MB

References

  • Implementation: inference-server-gkq.py (MemoryResidentWeightCache class)
  • Configuration: K8s deployment spec environment variables
  • Metrics: /metrics and /stats endpoints on inference server
  • Deployment: DEPLOY_NOW.sh (automated 5-phase pipeline)