Feedback alignment in deep convolutional networks

Theodore H. Moskovitz, Ashok Litwin-Kumar, L. F. AbbottarXiv 2018 · arXiv

Abstract

This study investigates the scalability of Feedback Alignment (FA), a biologically plausible alternative to backpropagation, in deep convolutional neural networks. By introducing Uniform Sign-Concordant Feedback (uSF) and weight normalization techniques, the authors demonstrate that FA can achieve performance competitive with backpropagation on the ImageNet dataset. The research addresses the long-standing "weight transport problem," finding that matching the signs of forward and backward weights is more critical than matching their precise magnitudes for successful learning in deep architectures.

Introduction

A major critique of standard deep learning from a neuroscience perspective is the "weight transport problem." To calculate error gradients, backpropagation (BP) requires the backward pass to use the exact transpose of the forward weight matrix ($B = W^T$). In biological brains, signals travel through distinct physical synapses in the forward and feedback directions; there is no known mechanism for a feedback synapse to perfectly mirror the strength of a forward synapse.

Feedback Alignment (FA) was proposed to solve this by using fixed, random weights for the backward pass. While FA works well on simple tasks like MNIST, it historically fails to scale to deep convolutional neural networks (CNNs) and complex datasets like ImageNet. This paper investigates why FA fails at depth and provides the modifications necessary to make it a viable learning rule for large-scale visual recognition.

Methods and Biological Constraints

The authors identify two primary obstacles for FA in deep networks: vanishing/exploding gradients and poor alignment in the early layers. To solve these, they introduce several modifications:

  1. Uniform Sign-Concordant Feedback (uSF): Instead of purely random feedback weights, the feedback matrix $B$ is set to match the sign of the forward weights $W$ (i.e., $B = \text{sign}(W)$). This is a "weaker" form of symmetry than BP but ensures the feedback signal remains in the correct direction.
  2. Normalization Techniques: Because forward weights change while feedback weights often remain fixed, the relative scales can drift, leading to gradient instability. The authors propose scaling feedback weights based on initial forward magnitudes (Init) or dynamically scaling the feedback matrix to match the $L_2$-norm of the forward weights (Strict Normalization, SN).
  3. Fixed E/I Connectivity: The researchers experimented with enforcing "Dale’s Law" by freezing the signs of weights (excitatory vs. inhibitory) to see if biological constraints hindered or helped learning.
Ratios of feedback to forward weight magnitudes (Naive and Glorot) across layers, highlighting the drift that occurs without normalization.

Key Results

The most critical finding is that for deep networks, the direction (sign) of the feedback signal is far more important than its precise magnitude. Matching signs allows the network to align its forward and backward passes effectively even in very deep architectures. In shallow networks, random feedback weights eventually "align" with forward weights, but in deep networks, this alignment rarely reaches the earliest layers without sign-concordance.

Training error curves for Top-1 (a) and Top-5 (b) accuracy across different learning rules on ImageNet.

On a 9-layer CNN, standard BP achieved a 79.5% Top-1 error. Their best modified FA (FA-uSF SN) achieved 78.9%—actually slightly outperforming the baseline BP in that specific configuration. When the $L_2$-norms of weights were fixed to stabilize training, FA-uSF achieved 51.2% Top-1 error on ImageNet, compared to 46.9% for BP. While BP remains superior in many configurations, this represents a massive improvement over "vanilla" FA, which essentially fails on ImageNet (95%+ error).

Discussion and Limitations

While sign-concordance is more biological than weight-transpose symmetry, it still requires the feedback path to know the signs of the forward path. The authors suggest this might be achieved through cell-type-specific wiring, but a purely local biological mechanism for this is still a subject of debate. Furthermore, enforcing fixed Excitatory/Inhibitory (E/I) signs significantly hurt performance, as weights often need to flip signs late into training to navigate the optimization landscape.

Future work is needed to see if these FA improvements hold up in "locally connected" networks without weight sharing, as convolutional layers themselves are biologically questionable. Additionally, exploring how temporal delays in homeostatic normalization affect learning would be a logical next step in bridging the gap between artificial and biological neural networks.

Summary updated

BibTeX
@misc{moskovitz2018feedback,
  author = {Theodore H. Moskovitz and Ashok Litwin-Kumar and L. F. Abbott},
  title = {{Feedback alignment in deep convolutional networks}},
  year = {2018},
  eprint = {1812.06488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.06488},
}
Figure