A method to automatically discover symbolic local learning rules

Andrea Perin, Fabio Anselmi, Stephane DenyarXiv 2026 · arXiv

TL;DR

Researchers have developed a new method to automatically discover symbolic local learning rules that can replace backpropagation. By using a physics-inspired notation to build a dictionary of mathematical terms, the system learns simple, interpretable equations for each layer that allow a neural network to train effectively without needing global gradient information.

What Problem Does This Paper Address?

Backpropagation (BP) is the engine of modern deep learning, but it has two major drawbacks. First, it is biologically implausible: the brain likely doesn't share precise gradient information across all neurons simultaneously. Second, it creates a communication bottleneck in hardware; because every layer depends on the ones after it to update, updates are sequential and energy-intensive.

While "local learning rules" (where each layer updates based only on its immediate neighbors) offer a solution, they are notoriously hard to design. Previous attempts to "meta-learn" these rules often used black-box neural networks (MLPs) to predict updates, leaving us with no idea how the discovered rules actually work. This paper aims to discover rules that are both effective and human-readable.

Methods

The authors introduce a systematic way to build and learn symbolic local losses:

  1. Trace Monomial Dictionary: Inspired by Penrose diagram notation (a graphical way to represent tensor operations, see Figure 1), the method generates all possible scalar functions (monomials) that can be formed by contracting the indices of local variables like inputs ($x$), weights ($W$), and outputs ($y$). Examples of tensor variables and trace monomials in Penrose notation
  2. Layerwise Polynomial Losses: For each layer, a local loss function is defined as a weighted sum of these monomials. The "learning" happens in an outer loop, where the system optimizes these weights ($\alpha$ coefficients) to ensure that the resulting local updates minimize a global target objective.
  3. Tunable Locality: The authors define a "neighborhood degree." A degree of 1 means a layer only sees its own variables (purely local), while higher degrees allow it to see information from a few layers downstream.
  4. Meta-Optimization: To keep training stable, they employ several tricks: normalizing the monomials, gradually increasing the training time horizon, and using a sparse prior to ensure the final equations are simple.
Figure 1: Examples illustrating the Penrose diagram notation.

Datasets

The method was evaluated on minimal datasets to keep the high computational cost of meta-learning manageable:

  • Sliced MNIST: A low-dimensional version of MNIST using only the middle row of pixels, with random sign flips to ensure the problem is non-linear.
  • Sliced Fashion-MNIST: Used to test how well rules discovered on one dataset transfer to another.
  • Full MNIST: Used specifically for the self-supervised learning (SSL) tasks.

Key Insights and Ideas

  • Interpretability is Key: Unlike MLP-based meta-learners, this method produces actual equations. For example, in classification, the discovered rules for the final layers naturally "rediscover" logit alignment and squared cross-entropy terms.
  • Surrogate Backward Channels: In supervised tasks, the rules for early layers (which don't have access to the label) develop "forward-looking" terms. These act as a surrogate for backpropagation's backward channel, allowing the first layer to adapt to features being learned downstream.
  • The Locality Trade-off: The researchers found that for classification, a neighborhood degree of 2 (seeing one layer ahead) significantly outperforms a degree of 1. This corroborates theoretical ideas that some form of feedback is necessary for complex supervised learning.
  • Symmetry Discovery: In self-supervised learning (using the Barlow Twins objective), the method discovered "swap-invariant" coefficients, correctly identifying that the two augmented versions of an image should be treated symmetrically.

Key Results

The discovered symbolic rules were tested across classification, reconstruction, and self-supervised learning:

  • Classification: The learned rules achieved 67% accuracy on Sliced MNIST. While lower than standard backpropagation (71%), it outperformed purely local baselines (63%) and Independent Component Analysis (65%).
  • Reconstruction: The rules induced non-trivial cooperation across layers, significantly beating "frozen backprop" baselines where only the final layers are allowed to learn.
  • Transferability: Rules learned on Sliced MNIST transferred successfully to Sliced Fashion-MNIST, showing that the discovered symbolic logic is somewhat universal to the task rather than just the specific data.
  • Sparsity: The evolution of the $\alpha$ coefficients (shown in Figure 3A) demonstrates that the method settles on a small number of dominant terms, leading to sparse, readable equations. Evolution of coefficients and loss during training
Figure 3: Classification task on the Sliced MNIST held-out split. (A) Evolution of α coefficients throughout training for a representative run. Each tick represents the α coefficient for one trace monomial of our dictionary. (B) Loss evolution throughout application of the learned rules (red), compared to backpropagation (grey), backpropagation with first layer frozen (dashed light grey), backpropagation with first layer frozen to the dataset’s PCA (blue) and ICA (green). Results are aggregated over Neval = 128 network initializations: the median loss value is reported as a solid line, and the shaded region corresponds to the 10th–90th percentile range. (C) Weights of the network after training with our learned rules (top) vs. backpropagation (bottom). Weights for PCA and ICA are shown in App. Fig. 11.

Limitations and Future Work

The primary limitation is scalability. Meta-optimizing these rules is computationally expensive and numerically delicate; the authors describe the optimization landscape as "fractal." Currently, the method is limited to small networks and low-dimensional data.

Future work will focus on:

  • Scaling to more realistic architectures (like Transformers or Convnets).
  • Improving the stability of the meta-optimization process.
  • Investigating shape invariance—the ability to take a rule learned on a small network and apply it to a much larger one without changing the symbolic coefficients.

Summary updated

Abstract

We present a general method for discovering symbolic local learning rules that can replace backpropagation, applicable, in principle, to any deep learning architecture and task. The method automatically discovers local loss functions specific to each layer by (1) constructing a dictionary of monomials of local tensor variables, inspired by Penrose diagram notation, (2) regressing over coefficients weighting these monomials to build layerwise polynomial losses whose gradients together minimize a target global loss. We test the method on classification, reconstruction, and self-supervised learning tasks with small multilayer networks and minimal datasets. Across all tasks, the discovered rules induce non-trivial interactions between layers, which lead to significant reductions in the corresponding global losses. Furthermore, the method selects sparse combinations of monomials, finding local rules that are more interpretable than those generated by MLP-based alternatives. The potential implications of our method range from energy-efficient deep learning to the discovery of biologically plausible learning mechanisms.
BibTeX
@misc{perin2026method,
  author = {Andrea Perin and Fabio Anselmi and Stephane Deny},
  title = {{A method to automatically discover symbolic local learning rules}},
  year = {2026},
  eprint = {2611.12248},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12248},
}
Figure