We didn't go looking for a routing mechanism in dense transformer models — there isn't one in the architecture. But every token entering Mistral-7B, Mistral-Small-24B, and Qwen3-4B gets routed into one of two processing paths at layers 3-4, and the routing is consistent enough that it's clearly a real circuit, not a statistical artifact.
The core finding
Every token entering the model gets routed into one of two paths:
- Mode A (93–95% of tokens): shallow processing, minimal transformation
- Mode B (5–7%): deep processing, massive representational shift
There's no mixture-of-experts routing. There's no explicit gating layer. The model develops this routing during training, on its own.
Cross-architecture confirmation
| Model | Family | Parameters | Gate Layer | Mode B share |
|---|---|---|---|---|
| Mistral-7B | Mistral | 7B | L3-4 | ~50% of outlier tokens |
| Mistral-Small-24B | Mistral | 24B | L3-4 | 47.6% of responses |
| Qwen3-4B | Qwen | 4B | L3-4 | 7.27% of tokens |
Statistical evidence
| Model | Evidence |
|---|---|
| Mistral-7B | L2-L3 avg distance +2,170%, std dev +17,750%, avg/median ratio 17.9× |
| Mistral-Small-24B | Cohen's d = 4.8, Silhouette = 0.83, AUC-ROC = 0.97, p < 0.001 |
| Qwen3-4B | Feature 374 correlation 0.96 across L3–L5, bimodal ratio 1.09 |
Causal proof
Ablating the gate at Layer 3 in Qwen3-4B:
| Metric | Baseline | After ablation | Change |
|---|---|---|---|
| L6 Mode B mean | 308.6 | 20.5 | -93.4% |
| L6 extreme tokens (>p99.9) | 20 | 0 | -100% |
| L6 max | 11,475.4 | 147.3 | -98.7% |
Removing the gate removes the downstream behavior. That's the test we wanted — a functional circuit, not a correlation.
Three-stage pipeline (Qwen3-4B, 200K tokens)
| Stage | Layers | Behavior | Mode B % |
|---|---|---|---|
| Shallow triage | L0–L5 | Bimodal gate, selective routing | 5–17% |
| Deep explosion | L6–L15 | Std dev explodes 4,289%, tokens leave vocabulary space | 0.1% |
| Final routing | L16–L35 | Steady divergence, second bimodal gate at L35 | 0.2–10.9% |
The L5 compression dip
Every L6-extreme token follows the same trajectory: scores build through L0–L4, drop at L5, then explode 300×+ at L6.
Token ' unwanted': L3=43.4, L4=48.3, L5=31.2, L6=11,409.6
Token ' tops': L3=38.5, L4=45.1, L5=25.7, L6=10,678.6
It's a two-stage funnel: 100% of L6-extreme tokens were L3 Mode B, but only 1.6% of L3 Mode B tokens become L6-extreme. The first gate qualifies; the second commits.
Where standard SAEs break
Sparse autoencoders, the dominant interpretability tool, completely fail on deep computation layers in these models. A pipeline of SipIt invertibility + a pre-trained SAE + GLP diffusion prior recovers what they miss.
| Layer | Role | SAE explained variance | Pipeline explained variance |
|---|---|---|---|
| L3 | Gate | 92.2% | 99.99% |
| L5 | Compression | 85.2% | 99.99% |
| L6 | Explosion | -905% | 100.0% |
| L8 | Post-explosion | -1,111% | 100.0% |
| L16 | Deep compute | -1,058% | 100.0% |
| L24 | Deepest | -3,059% | 100.0% |
| L35 | Final gate | 86.9% | 99.97% |
The takeaway we drew: deep computation layers use dense / distributed representations, not the sparse features SAEs are built to extract. That's a blind spot in the dominant interpretability approach, not a defect in our SAEs.
Thinking-mode experiment
Qwen3's
| Layer | Baseline Mode B % | Think Mode B % | Baseline max | Think max |
|---|---|---|---|---|
| L6 | 0.2% | 0.2% | 11,679 | 17,013 (+45.6%) |
| L35 | 8.9% | 8.1% | 3,896 | 7,272 (+86.7%) |
Emotion probe results
| Category | L6 mean | L6 max | vs. neutral |
|---|---|---|---|
| Emotion | 7,218 | 10,405 | +48% |
| High valence | 6,383 | 7,639 | +31% |
| Neutral | 4,874 | 5,438 | baseline |
The gate fires on the first token of the sentence. The model decides at onset whether deep processing is needed.