Wide Neural Networks as a Baseline for the Computational No-Coincidence Conjecture

John Dunbar, Scott AaronsonarXiv 2025 · DOI · arXiv

Abstract

This paper, authored by John Dunbar and Scott Aaronson from UT Austin, explores the statistical properties of wide, randomly initialized neural networks to bridge the gap between deep learning theory and AI safety. The authors demonstrate that these networks behave like truly random functions—with independent outputs—if and only if their activation function has a zero mean under a Gaussian distribution. This finding allows for the proposal of neural networks as a realistic baseline for the "Computational No-Coincidence Conjecture" (CNCC), a benchmark for testing the limits of AI interpretability.

Introduction: The Interpretability Challenge

A major hurdle in AI safety is interpretability: determining whether "white-box" methods are identifying meaningful structures or merely seeing patterns in noise. To measure the limits of these tools, researchers at the Alignment Research Center (ARC) proposed the Computational No-Coincidence Conjecture (CNCC). The CNCC suggests that if a random system has a rare property (a "coincidence"), there should be a short, verifiable proof of why that property exists. While the CNCC originally used abstract "random reversible circuits," this paper moves the conjecture toward neural networks to provide a more practical theoretical foundation.

Theoretical Methodology

The authors employ tools from the principles of deep learning theory to analyze neural networks at large but finite widths ($n$):

  • Perturbative Expansion: Instead of assuming infinite width, they use a first-order expansion to account for $1/n$ effects.
  • Hermite Decomposition: Activation functions ($\sigma$) are decomposed into Hermite polynomials, which are orthogonal under Gaussian measures. This allows the "covariance map"—how input correlations change across layers—to be treated as a simple polynomial.
  • Fixed-Point Analysis: They analyze the attractor states of these maps to see if correlations between different inputs decay to zero as the network gets deeper.

The Zero-Mean Requirement

The most striking finding is that for a network's outputs to become independent (uncorrelated) as it gets deeper, the activation function must satisfy $\mathbb{E}_{z \sim \mathcal{N}(0,1)}[\sigma(z)] = 0$.

Evolution of the covariance map $K^{(\ell)}$ for different activation functions. For outputs to become independent, the map must have a stable fixed point at the origin. Standard ReLU (left) fails this, while tanh (center) and shifted ReLU (right) succeed.

As shown in the figure above, standard ReLU or GeLU networks cause inputs to become increasingly correlated, effectively "clumping" the output space. This explains why tanh networks naturally behave more like random functions. Interestingly, one can make ReLU behave like a random function by adding a constant shift: $\sigma(z) = \text{ReLU}(z) - 1/\sqrt{2\pi}$.

Key Results

  • Theorem 4.1: Establishes that under "critical tuning," the covariance between different inputs decays exponentially with depth if and only if the activation function is nonlinear and zero-mean.
  • Corollary 4.1: Proves that the total probability distribution of the outputs approaches a standard Gaussian distribution.
  • Conjecture 5.1: The authors formally propose a "Neural Network Computational No-Coincidence Conjecture," using tanh-based networks as the new gold standard for testing interpretability verifiers.

Limitations and Future Work

While these networks provide a robust baseline, they exhibit a polynomial ($1/n$) rather than exponential dependence between outputs due to width effects. Furthermore, this analysis applies to randomly initialized networks; real-world trained networks introduce non-random structures that this baseline is intended to help distinguish from noise. Future work is needed to see if current interpretability techniques, like sparse autoencoders, can solve the challenges posed by this new conjecture.

Summary updated

BibTeX
@misc{dunbar2025wide,
  author = {John Dunbar and Scott Aaronson},
  title = {{Wide Neural Networks as a Baseline for the Computational No-Coincidence Conjecture}},
  year = {2025},
  eprint = {2510.06527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06527},
}
Figure