The Ebb and Flow of Deep Learning: a Theory of Local Learning

Pierre Baldi, Peter J. Sadowski

TL;DR

This paper introduces a formal framework for understanding local learning rules in neural networks, moving beyond the vague concept of "Hebbian learning." It demonstrates that while local learning is powerful for representation learning, it's insufficient for complex tasks without a backward learning channel, and backpropagation emerges as optimal in terms of information transmission and computational efficiency.

What Problem Does This Paper Address?

The paper tackles the fundamental challenge of how neural networks learn, particularly in deep architectures. It highlights the long-standing vagueness surrounding "Hebbian learning" and the limited number of learning rules discovered. The authors aim to provide a systematic framework to organize, analyze, and discover new learning rules, and to understand the limitations of purely local learning, especially in deep networks. They question why the space of learning rules appears sparse and whether more efficient algorithms exist than backpropagation.

Methods

The authors propose a framework for local learning rules based on two key aspects:

  1. Locality: Learning rules must depend only on locally available variables (e.g., pre- and post-synaptic activities).
  2. Functional Form: They analyze polynomial learning rules, stratifying them by degree (n) and effective degree (d).

They systematically study these rules in both linear and non-linear units and networks. A crucial concept introduced is the "learning channel," which describes how information about targets is communicated to deep layers. The paper analyzes the capacity of this channel for various algorithms, defining it as the ratio of transmitted information (bits) to computational cost (operations) per weight.

Datasets

Not explicitly discussed as distinct datasets for training and evaluation. The paper uses general concepts of input vectors and targets, and mentions examples like MNIST for simulations.

Key Insights and Ideas

  • "Hebbian Learning" is too vague: The paper argues for replacing the nebulous concept of Hebbian learning with a precise definition of "local learning rules," specifying which variables are local and the functional form of the update rule.
  • Deep Local Learning is Limited: Stacking local learning rules in deep networks (deep local learning) can learn representations but fails to learn complex input-output functions, even with top-layer targets.
  • The Learning Channel is Crucial: Learning complex functions requires a backward learning channel to communicate target information to deep layers. This channel can be the forward connections used in reverse or a separate channel.
  • Backpropagation is Optimal: Backpropagation is shown to be optimal in terms of learning channel capacity, maximizing the information rate (bits per weight) while minimizing computational cost. This optimality holds for both feedforward and recurrent networks.
  • Sparsity of Learning Rules: The framework helps explain why the space of discovered learning rules appears sparse, suggesting that only a few rules possess the necessary properties for efficient learning.

Key Results

  • A formal framework for analyzing local learning rules is established, stratifying them by polynomial degree.
  • It's demonstrated that deep local learning alone cannot learn complex functions.
  • The concept of a "learning channel" is introduced, and its capacity is analyzed for various algorithms.
  • Backpropagation is proven to be optimal among the considered algorithms in terms of learning channel capacity, achieving the maximal rate and improvement.
  • The theory clarifies the limitations of Hebbian learning and provides a basis for understanding the optimality of backpropagation.

Limitations and Future Work

  • Approximations: The analysis often relies on approximations, particularly for non-linear units and in the context of the learning channel's information rate.
  • Focus on Polynomial Rules: The study primarily focuses on polynomial learning rules, acknowledging that other functional forms might exist.
  • Biological Relevance: While drawing inspiration from biological systems, the paper primarily uses artificial neural network models. The direct biological relevance of these findings is presented as an open question.
  • Future Work: The authors suggest extending the theory to other learning paradigms (e.g., reinforcement learning), exploring different network architectures, and investigating the role of random versus symmetric weights in learning channels. The paper also hints at further exploration of the nature and semantics of the backward learning channel.

Summary updated

Abstract

In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. We estimate the learning channel capacity associated with several algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost, even in recurrent networks. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.
BibTeX
@article{baldi2016ebb,
  author = {Pierre Baldi and Peter J. Sadowski},
  title = {{The Ebb and Flow of Deep Learning: a Theory of Local Learning}},
  journal = {Neural Networks},
  year = {2016},
  doi = {10.1016/j.neunet.2016.07.006},
}
Figure