TL;DR
By adapting the "logit lens" interpretability technique to Post-LN transformers, the authors inspect the internal reasoning of Leela Chess Zero's policy network layer by layer. Rather than smoothly refining move choices, Leela undergoes a three-phase computation where intermediate layers often find brilliant tactical sacrifices, only for the final layers to override them with conservative, "safe" moves. This reveals a fundamental architectural tension: algorithmic look-ahead can be actively suppressed by learned heuristic safety priors.
What Problem Does This Paper Address?
Do deep neural networks build solutions through gradual, continuous refinement, or do they execute discrete, algorithmic recomputation at each layer? While the logit lens has helped answer this for autoregressive language models, adapting it to non-language domains and modern Post-LN architectures with residual scaling (like DeepNorm) has remained challenging.
Furthermore, superhuman chess engines like Leela Chess Zero (Lc0) can find complex tactical lines without Monte Carlo Tree Search (MCTS), but how their feed-forward policy networks internally balance deep combinatorial look-ahead against broad heuristic evaluation is poorly understood.
Methods
- Post-LN Logit Lens: Standard logit lenses zero-out future layers in Pre-LN transformers, where the residual stream remains unnormalized. For Post-LN transformers with DeepNorm, normalizations and scaling constants ($\alpha$ and $\gamma$) directly transform the residual stream at every layer. The authors zero-ablate future multi-head attention and MLP sublayer outputs while explicitly preserving downstream normalization steps and scaling factors (and zeroing downstream layer-norm biases $\beta$) to faithfully decode intermediate representations through the 64-token policy head.
- Concept Preference Analysis: Rather than using linear probes to see if intermediate layers merely encode concepts, the authors measure what concepts intermediate layers act upon. For each layer's move distribution $\pi_\ell$, they compute the expected change $\Delta c_\ell = \mathbb{E}_{\pi_\ell}[\Delta c_m]$ across 93 human-interpretable evaluation terms extracted from Stockfish 8 (e.g., material, king safety, piece mobility, threats).
Datasets
- CCRL Dataset: A standard benchmark of 2.5 million engine games, from which 10,000 positions were sampled for concept preference analysis and 1,000 positions for policy distribution dynamics.
- Lichess Puzzles: 10,000 tactical puzzles with unique winning lines, stratified across rating bands (Elo 200 to 3000).
- Encyclopedia of Chess Openings (ECO): 200 opening positions used in round-robin internal tournaments (with deterministic $\tau = 0$ and stochastic $\tau = 1.0$ move selection).
- Live Lichess Play: Real-world validation where intermediate layer representations were deployed as online bots across Bullet, Blitz, and Rapid time controls.
- The Pile: Sampled text used to benchmark comparative ranking dynamics in GPT-2 (Small, Large, XL).
Key Insights and Ideas
- Three Phased Computational Stages:
- Early Phase (Layers 0–5): Rapid Elo growth and volatile concept shifts.
- Middle Phase (Layers 5–10): A performance plateau resembling "feature engineering," where move distributions and concept preferences remain largely static.
- Late Phase (Layers 11–Final): Sudden capability acceleration driven by known look-ahead attention heads (layers 11–13) integrating candidate moves.
- Algorithmic Recomputation, Not Smoothing: Kendall's $\tau$ rank correlation between intermediate move rankings and final outputs hovers near zero or remains negative until layer 11, where it suddenly spikes. Meanwhile, policy entropy remains constant. The model is actively reshuffling its move preferences rather than gradually narrowing confidence.
- The "Forgotten Puzzle" Phenomenon: Early and middle layers frequently find correct, forcing tactical combinations (like piece sacrifices delivering forced checkmate). However, final layers often abandon these winning lines in favor of passive, "safe" alternatives.
- Heuristic Priors Overriding Tactics: Concept tracking shows intermediate layers embrace aggressive tactical play (high threat creation and opponent king vulnerability), while final layers pivot sharply toward king safety and threat reduction. Remarkably, the model's value head correctly recognizes that the tactical sacrifice wins (assigning $>98\%$ win probability to the resulting board), but the policy head's final output still chooses the safe move.
[ Layers 0–5 ] --> [ Layers 5–10 ] --> [ Layers 11–15 ]
Rapid Initial Gains Plateau & Exploration Integration & Safety
• Captures / basic threats • "Feature engineering" • Look-ahead heads activate
• High policy volatility • Finds complex tactical wins • Final layers enforce safety priors;
can override tactical wins!
Key Results
- Playing Strength: In internal tournaments ($\tau = 0$), playing strength rises from 443 Elo (input representation) through a plateau of ~1007 Elo (layer 5) to 1057 Elo (layer 10), before accelerating to 1681 Elo (layer 13) and 2263 Elo (full model).
- Puzzle Acceleration: For difficult puzzles (Elo 2200–3000), the improvement rate in the final phase accelerates to over 60 times the rate seen in the middle plateau.
- Cumulative vs. Final Discoveries: Cumulative puzzle solve rates across depth consistently exceed the final layer’s solve rate, formally demonstrating that correct solutions found by intermediate layers are discarded later in the forward pass.
Limitations and Future Work
- Single Architecture: The study analyzes only one Post-LN chess model (Lc0
T82-768x15x24h); comparing these findings to Pre-LN chess models or different transformer architectures would help decouple domain dynamics from architectural artifacts. - Purely Observational: While the work identifies when moves and concepts shift, it does not isolate the exact circuits (individual MLPs or attention heads) causing safety priors to override tactical calculations via causal interventions (such as activation patching).
- One-Step Concept Scope: Concepts are measured strictly on immediate 1-ply resulting positions via Stockfish 8, omitting multi-move strategic depth and long-term planning.
Summary updated