TL;DR
ConwayNet introduces a fixed, Game-of-Life-inspired "lateral inhibition" mechanism into neural network layers. By forcing local inputs to regulate outputs through differentiable "Birth" and "Survival" gates, the authors create networks with spatially coherent representations that are significantly more interpretable, reproducible, and resilient to noise.
What Problem Does This Paper Address?
Deep neural networks are powerful "universal function approximators," but they are often criticized as "black boxes." Their internal activations frequently look like high-frequency "salt-and-pepper" noise, making them difficult to interpret mechanistically. Furthermore, they are notoriously sensitive to pixel-level noise and can exhibit high "churn"—where two networks trained from nearly identical starting points disagree on a significant number of test images.
In contrast, Cellular Automata (CA) like Conway’s Game of Life (GoL) use simple, local rules to generate complex, programmable, and highly interpretable dynamics. This paper attempts to bridge these two worlds by using GoL-style rules to regulate the flow of information within standard neural network architectures.
Methods
The authors propose a mechanism called lateral inhibition. Unlike standard layers where every parameter is learned, this mechanism uses a fixed, non-learned rule based on neighborhood occupancy.
- The Standard Path: An input $x$ undergoes a standard affine transformation and activation (e.g.,
tanh) to produce a candidate activation $z$. - The CA Path: In parallel, the input $x$ is used to calculate the "occupancy" $m$ of a unit’s local neighborhood (a $3 \times 3$ Moore neighborhood).
- Birth and Survival Gates: Using smooth, differentiable versions of GoL rules (specifically the B5/S3 rule), the model calculates "Birth" and "Survival" gates based on $m$.
- The Blend: The final output $y$ is a state-conditioned blend of these gates. If the unit’s internal state $z$ is "off," the Birth gate dominates; if it is "on," the Survival gate dominates.
This mechanism is architecture-agnostic and was tested on Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), Wide Residual Networks (WRNs), and Graph Convolutional Networks (GCNs).
Datasets
The authors evaluated ConwayNet across several standard computer vision benchmarks and their "color" variants:
- MNIST-family: MNIST, EMNIST, FashionMNIST, and KMNIST.
- CIFAR-10: Used for testing on more complex natural images with Wide Residual Networks.
- Corrupted Data: The models were tested against Gaussian, shot, and impulse noise using a protocol similar to the ImageNet-C benchmark to evaluate out-of-distribution robustness.
Key Insights and Ideas
- Fixed Rule, Learned Steering: Even though the GoL rule is fixed and receives no gradients, it shapes training. The network learns to "steer" its activations into regions where the fixed rule allows them to persist.
- Spatial Coherence: In a standard MLP, hidden activations look like random noise. In ConwayNet, activations are smooth, contiguous, and clearly resemble the input object (e.g., the digit "2").
- Scale Separation: The authors found that coarse structures persist through the depth of the network, while fine details are "renewed" at every layer. This mimics the separation of scales seen in physical systems like fluid dynamics.
- The Co-activation Graph: By analyzing which units fire together, the authors found that ConwayNet naturally organizes itself into a spatial graph that recovers the geometry of the input image, unlike "plain" networks which have no such geometric structure.
Key Results
- Noise Resilience: ConwayNet shows dramatic improvements in noise robustness. On EMNIST, the lateral inhibition CNN improved accuracy under noise by over 30 percentage points compared to a standard CNN.
- Interpretability: Saliency maps (which show what the network is "looking at") are much cleaner. In ConwayNet, the gradient saliency rests almost entirely on the object itself rather than the background.
- Reproducibility: Under label noise, ConwayNet reduced "churn" (disagreement between nearly identical models) by a factor of thirty.
- Performance Trade-off: These benefits come at a small cost to "clean" accuracy (usually 1-2 points), though the Wide Residual Network actually saw a slight gain (+0.8 points) on CIFAR-10.
Limitations and Future Work
- Training Difficulty: ConwayNets are harder to train; they converge more slowly and often require longer training schedules.
- Sensitivity to Normalization: The mechanism is sensitive to how inputs are normalized, as the "occupancy" bands are fixed.
- Manual "Wiring": Integrating the mechanism into new architectures currently requires manual effort to determine where to read inputs and how to reduce channels.
- Rule Search: The authors primarily used the B5/S3 rule. There are over 262,000 possible "Life-like" rules; exploring this space with evolutionary algorithms or foundation models is a suggested path for future research.
Summary updated