TL;DR
Researchers have developed a new method to automatically discover symbolic local learning rules that can replace backpropagation. By using a physics-inspired notation to build a dictionary of mathematical terms, the system learns simple, interpretable equations for each layer that allow a neural network to train effectively without needing global gradient information.
What Problem Does This Paper Address?
Backpropagation (BP) is the engine of modern deep learning, but it has two major drawbacks. First, it is biologically implausible: the brain likely doesn't share precise gradient information across all neurons simultaneously. Second, it creates a communication bottleneck in hardware; because every layer depends on the ones after it to update, updates are sequential and energy-intensive.
While "local learning rules" (where each layer updates based only on its immediate neighbors) offer a solution, they are notoriously hard to design. Previous attempts to "meta-learn" these rules often used black-box neural networks (MLPs) to predict updates, leaving us with no idea how the discovered rules actually work. This paper aims to discover rules that are both effective and human-readable.
Methods
The authors introduce a systematic way to build and learn symbolic local losses:
- Trace Monomial Dictionary: Inspired by Penrose diagram notation (a graphical way to represent tensor operations, see Figure 1), the method generates all possible scalar functions (monomials) that can be formed by contracting the indices of local variables like inputs ($x$), weights ($W$), and outputs ($y$). Examples of tensor variables and trace monomials in Penrose notation
- Layerwise Polynomial Losses: For each layer, a local loss function is defined as a weighted sum of these monomials. The "learning" happens in an outer loop, where the system optimizes these weights ($\alpha$ coefficients) to ensure that the resulting local updates minimize a global target objective.
- Tunable Locality: The authors define a "neighborhood degree." A degree of 1 means a layer only sees its own variables (purely local), while higher degrees allow it to see information from a few layers downstream.
- Meta-Optimization: To keep training stable, they employ several tricks: normalizing the monomials, gradually increasing the training time horizon, and using a sparse prior to ensure the final equations are simple.
Datasets
The method was evaluated on minimal datasets to keep the high computational cost of meta-learning manageable:
- Sliced MNIST: A low-dimensional version of MNIST using only the middle row of pixels, with random sign flips to ensure the problem is non-linear.
- Sliced Fashion-MNIST: Used to test how well rules discovered on one dataset transfer to another.
- Full MNIST: Used specifically for the self-supervised learning (SSL) tasks.
Key Insights and Ideas
- Interpretability is Key: Unlike MLP-based meta-learners, this method produces actual equations. For example, in classification, the discovered rules for the final layers naturally "rediscover" logit alignment and squared cross-entropy terms.
- Surrogate Backward Channels: In supervised tasks, the rules for early layers (which don't have access to the label) develop "forward-looking" terms. These act as a surrogate for backpropagation's backward channel, allowing the first layer to adapt to features being learned downstream.
- The Locality Trade-off: The researchers found that for classification, a neighborhood degree of 2 (seeing one layer ahead) significantly outperforms a degree of 1. This corroborates theoretical ideas that some form of feedback is necessary for complex supervised learning.
- Symmetry Discovery: In self-supervised learning (using the Barlow Twins objective), the method discovered "swap-invariant" coefficients, correctly identifying that the two augmented versions of an image should be treated symmetrically.
Key Results
The discovered symbolic rules were tested across classification, reconstruction, and self-supervised learning:
- Classification: The learned rules achieved 67% accuracy on Sliced MNIST. While lower than standard backpropagation (71%), it outperformed purely local baselines (63%) and Independent Component Analysis (65%).
- Reconstruction: The rules induced non-trivial cooperation across layers, significantly beating "frozen backprop" baselines where only the final layers are allowed to learn.
- Transferability: Rules learned on Sliced MNIST transferred successfully to Sliced Fashion-MNIST, showing that the discovered symbolic logic is somewhat universal to the task rather than just the specific data.
- Sparsity: The evolution of the $\alpha$ coefficients (shown in Figure 3A) demonstrates that the method settles on a small number of dominant terms, leading to sparse, readable equations. Evolution of coefficients and loss during training
Limitations and Future Work
The primary limitation is scalability. Meta-optimizing these rules is computationally expensive and numerically delicate; the authors describe the optimization landscape as "fractal." Currently, the method is limited to small networks and low-dimensional data.
Future work will focus on:
- Scaling to more realistic architectures (like Transformers or Convnets).
- Improving the stability of the meta-optimization process.
- Investigating shape invariance—the ability to take a rule learned on a small network and apply it to a much larger one without changing the symbolic coefficients.
Summary updated