TL;DR
This paper introduces NLS depth, a formal metric to quantify how much "hidden" serial computation an AI model performs between human-readable steps. By defining what counts as an interpretable bottleneck, the authors provide a way to predict a model's "monitorability" based on its architecture and training recipe before a single weight is even updated.
What Problem Does This Paper Address?
As AI models become more capable, we increasingly rely on Chain-of-Thought (CoT)—the step-by-step text reasoning a model produces—to oversee and audit their behavior. However, there is a looming "monitorability" crisis.
If future AI architectures move away from standard transformers toward "latent reasoning" models (which pass complex hidden states between steps instead of just text), the model could perform massive amounts of "thinking" that we can't see. This "opaque" cognition makes it easier for models to hide misaligned reasoning or develop dangerous capabilities undetected. The authors tackle the challenge of how to formally measure this hidden serial depth in a way that is objective, quantifiable, and applicable to both current and proposed architectures.
Methods
The authors operationalize a concept called Natural-Language-rooted node-Separated depth (NLS depth).
The core idea is to treat a model's computation as a mathematical circuit and measure the longest path (depth) that does not pass through an "interpretable bottleneck." Their main contribution is a rigorous standard for what qualifies as a Natural-Language-rooted (NL-rooted) node:
- Natural-language initialization: The node must be initialized from a pre-training prior to produce human-understandable media (text, code, images).
- No expansion of output space: No "secret" tokens or continuous latent states allowed in the bottleneck.
- No re-interpretation: The model cannot be modified to treat these tokens as a different type of data (e.g., using text as a proxy for hidden activations).
- No backpropagation through tokens: Gradients cannot flow through the sampling step, ensuring the "bottleneck" is a discrete, non-differentiable break in the computation.
By measuring the circuit depth between these bottlenecks, they can assign a number to a model's "opaque serial depth."
Datasets
The paper does not introduce a new training dataset. Instead, it performs an extensive architectural analysis of dozens of open-weight models, including:
- Llama series (Llama-2, 3, 3.1)
- Qwen series (Qwen-2, 2.5, 3)
- Mistral/Mixtral
- Gemma (including a detailed breakdown of Gemma3-1B)
- DeepSeek (V2, V3, V4-Pro)
- Specialized architectures: Mamba, DeltaNets (Kimi-K3), and hypothetical "latent reasoning" models.
Key Insights and Ideas
- Layers $\neq$ Depth: While NLS depth correlates strongly with layer count in standard transformers, it’s not a 1:1 mapping. Operations like matrix multiplications contribute to depth based on the log of the dimensions.
- The "Latent Reasoning" Cliff: Standard transformers have NLS depths in the low thousands (e.g., Llama-3.1-405B is ~11,500). In contrast, hypothetical latent reasoning models (like Meta’s COCONUT) could have NLS depths in the millions, representing a massive jump in unobservable cognition.
- Slow Scaling: In the current transformer paradigm, NLS depth grows very slowly—roughly proportional to the 4th root of the number of active parameters ($N^{1/4}$). This suggests that simply making transformers bigger doesn't drastically reduce our ability to monitor them via CoT.
- Adjudication Before Training: Because NLS depth is based on architecture and training rules, developers can (and should) report it before they even start training, serving as a "safety spec" for the model.
Key Results
- Benchmark Correlation: The authors found a strong correlation between NLS depth and the number of layers across open-source models (see Figure 1 in the paper).
- Current Limits: No current open-source CoT model has an NLS depth exceeding 17,000.
- Outliers: Models like Kimi-K3 (using Gated DeltaNets) and DeepSeek-V4-Pro show significantly higher depth per layer than standard transformers, signaling architectural shifts that might be more concerning for monitorability.
- Gemma3-1B Case Study: They provide a worked example calculating the NLS depth of Gemma3-1B to be exactly 1,954.
Limitations and Future Work
- The "DeepSeek Flaw": The authors admit NLS depth can be "tricked" by low-FLOP but serially intense operations. DeepSeek-V4-Pro has a high NLS depth primarily due to the Sinkhorn-Knopp algorithm, which is serially deep but likely doesn't contribute to "reasoning."
- Steganography: A low NLS depth doesn't prevent a model from using steganography (hiding secret messages in plain text). It only ensures that the capacity for hidden serial thinking is limited.
- Human Judgment: While more objective than previous measures, deciding if a training method "modifies the model to interpret tokens differently" still requires some human adjudication.
- Future Directions: The authors suggest developing "FLOP-weighted" serial depth measures to better distinguish between meaningful cognition and trivial serial calculations.
Summary updated