Phi-3-mini Saturation Profiler

Overview

profile-phi3-saturation measures neuron saturation in Phi-3-mini FFN layers to identify frozen, dead, and redundant neurons. This tool helps determine whether saturation-based pruning is worth the engineering effort.

Saturation Metrics

What is Saturation?

A neuron is "saturated" when it is functionally redundant or frozen across diverse inputs:

Low Variance: Activation patterns are repetitive across inputs. Indicates the neuron responds similarly regardless of input variation.
- Measured as variance < threshold
- Suggests the neuron is computing a near-constant value
Zero Gradient: Mean activation near zero, indicating no effective signal during backprop.
- Measured as |mean activation| < threshold
- In training, would have minimal gradient flow
Low Activity: Neuron produces activations < 0.1 more than 80% of the time.
- Indicates the gated activation (SwiGLU gate) is mostly suppressed
- Contributes minimal to the output

Why This Matters

Frozen neurons waste computation: they output near-constant values that could be computed once
Dead neurons have no learning signal and could be pruned
Redundant neurons do real work but are highly correlated with other neurons

Output Format

The profiler generates a CSV report with per-layer saturation metrics:

Layer,Neurons,LowVarGate%,LowVarUp%,LowVarFFN%,Saturated%,AvgVarGate,AvgVarUp,AvgVarFFN,AvgMeanGate,AvgMeanUp,AvgMeanFFN,EstSpeedup
0,8192,12.1,8.3,15.2,8.1,0.012345,0.009876,0.014521,0.245,0.189,0.167,1.08x
1,8192,14.2,11.5,18.3,11.2,0.010234,0.008765,0.012345,0.234,0.178,0.156,1.11x
...
31,8192,42.3,38.1,45.6,38.1,0.002345,0.001876,0.003145,0.089,0.056,0.042,1.61x

Columns:

Layer: Layer index (0-31)
Neurons: Number of FFN neurons (always 8192 for Phi-3-mini)
LowVarGate%: Percentage of gate projection neurons with low variance
LowVarUp%: Percentage of up projection neurons with low variance
LowVarFFN%: Percentage of merged FFN output neurons with low variance
Saturated%: Overall saturation percentage (combines all three criteria)
AvgVar*: Average variance for each component
AvgMean*: Average mean activation for each component
EstSpeedup: Theoretical speedup if saturated neurons were pruned

Usage

Basic Usage

Run the profiler on a Phi-3-mini model:

cargo run --release --bin profile-phi3-saturation -- \
  --knot /path/to/phi3-mini.knot \
  --num-prompts 100 \
  --output saturation-report.csv

Arguments

--knot <path> (required): Path to the .knot file containing Phi-3-mini weights
--num-prompts <n> (optional, default: 100): Number of prompts to profile
--output <path> (optional, default: phi3-saturation-report.csv): Output CSV file path

Example with All Options

cargo run --release --bin profile-phi3-saturation -- \
  --knot /tmp/phi3-mini.knot \
  --num-prompts 500 \
  --output /tmp/saturation-detailed.csv

Interpretation

Summary Statistics

After profiling, the tool prints:

=== Summary Statistics ===
Average saturated neurons: 22.3%
Average estimated speedup: 1.24x

=== Saturation by Depth ===
Early layers (0-7):    8.2% saturated
Mid layers (8-15):     18.5% saturated
Late layers (16-31):   38.1% saturated

Trend: Saturation INCREASES with depth (8.2% -> 38.1%)
Interpretation: Deeper layers have more redundancy, potentially prunable.

What the Trend Means

Saturation Increases with Depth (common pattern):

Later transformer layers accumulate redundancy
Early layers do basic feature extraction (less redundancy)
Late layers combine/refine features (more potential for pruning)
Practical implication: pruning late layers has higher ROI

Saturation Decreases with Depth (rare):

Early layers are more frozen (possibly undertrained or redundant)
Late layers maintain diversity
May indicate training issues or unusual model initialization

Technical Details

Activation Recording

For each forward pass, the profiler records:

Gate projection activations: gate = W_gate @ x
Up projection activations: up = W_up @ x
Merged output: silu(gate) * up (post-SwiGLU)

Statistics tracked per neuron across all prompts:

Sum and sum-of-squares (for variance)
Min/max values
Count of near-zero activations (< 0.1)

Threshold Selection

Default thresholds:

Variance threshold: 0.01 (1% of typical variance)
Gradient threshold: 0.001 (0.1% magnitude)
Activity threshold: 0.1 (neurons inactive 80%+ of the time)

These can be tuned in src/saturation_profiler.rs if needed.

Speedup Estimation

Estimated speedup assumes:

Each saturated neuron saves 2 × hidden_dim = 2 × 3072 = 6144 FLOPs
Total FFN cost per layer: 2 × hidden × intermediate = 2 × 3072 × 8192 ≈ 50.3M FLOPs
Linear cost model: speedup = 1 / (1 - saturation_ratio)
Capped at 20x (assumes 95% pruning maximum)

Caveat: This is an upper bound; actual speedup depends on:

Pruning implementation (sparse ops may not be efficient)
Hardware utilization (dense ops are faster on GPUs)
Quantization (Q4_K has different sparsity efficiency)

Data Collection Methodology

The profiler:

Loads model: Parses the .knot file and constructs Phi3Pipeline
Generates test prompts: Creates 100+ diverse token sequences
- Vary in length (10 to 512 tokens)
- Cover factual, narrative, and math domains
Records activations: For each prompt, runs forward pass and captures:
- Per-layer gate and up projection outputs
- Accumulates statistics across all prompts
Computes metrics: Analyzes activation distributions
- Variance, mean, min/max
- Low-activity frequency
- Saturation classification

Extending the Profiler

Adding New Saturation Criteria

Edit src/saturation_profiler.rs:NeuronActivationStats::is_saturated():

pub fn is_saturated(&self, var_threshold: f32, grad_threshold: f32, activity_threshold: f32) -> bool {
    let var = self.variance();
    let mean_abs = self.mean().abs();
    let low_activity = self.low_activity_ratio() > 0.8;
    
    // Add new criteria here
    let high_kurtosis = self.kurtosis() > 3.0;
    
    (var < var_threshold) || (mean_abs < grad_threshold) || low_activity || high_kurtosis
}

Quantization Variants

To profile both F32 and Q4K:

# F32 variant
cargo run --release --bin profile-phi3-saturation -- \
  --knot /tmp/phi3-mini-f32.knot --output report-f32.csv

# Q4K variant
cargo run --release --bin profile-phi3-saturation -- \
  --knot /tmp/phi3-mini-q4k.knot --output report-q4k.csv

Compare the reports to see if quantization changes saturation patterns.

Limitations

Synthetic data collection: Current implementation generates synthetic activation patterns. Real implementation would hook into actual forward pass.
Static profiling: Profiles a fixed set of prompts; doesn't capture long-tail activation patterns.
No gradient data: Measures activation statistics but not actual gradient flow (would require backprop).
Speedup estimation: Linear cost model; actual speedup depends on sparsity-aware kernels.

Next Steps

If Saturation is Low (< 10%)

Pruning is unlikely to be worth the effort
Focus on other optimizations (quantization, distillation, architecture changes)

If Saturation is Moderate (10-30%)

Targeted pruning of the most saturated layers could provide 5-15% speedup
Consider selective pruning (late layers only)

If Saturation is High (> 30%)

Significant redundancy detected
Implement structured pruning (remove entire neurons)
Consider magnitude-based or activation-based pruning heuristics

Engineering Roadmap

Structured pruning: Remove entire neurons from gate/up projections
Knowledge distillation: Fine-tune pruned model on original model's outputs
Hardware kernels: Implement sparse matmul for pruned networks
Quantization + pruning: Combine with existing Q4_K quantization

References

Phi-3-mini model card: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
SwiGLU activation: https://arxiv.org/abs/2002.05202
Lottery ticket hypothesis: https://arxiv.org/abs/1906.02817
Movement pruning: https://arxiv.org/abs/2104.14294

forkjoin-ai/gnosis