The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber

TL;DR

This paper reveals that every linear layer in a neural network trained by gradient descent is mathematically equivalent to a key-value memory system. By leveraging this "dual form," researchers can pinpoint exactly which training examples a model "attends to" when making a specific test-time prediction. This perspective turns the traditional "black box" of neural weights into a searchable database of training experiences.

What Problem Does This Paper Address?

Despite their massive success, deep neural networks (NNs) remain notoriously difficult to interpret. We can easily verify that a model works, but we rarely know why it made a specific decision or which parts of its training data were most influential.

Traditionally, we think of learning as "compressing" data into a fixed weight matrix. However, this compression is lossy and obscures the relationship between individual training samples and test-time outputs. The authors tackle this lack of transparency by revisiting a mathematical concept from the 1960s—the dual form of the perceptron—and applying it to modern deep learning. Their goal is to create a direct, observable link between a model's output and the specific training patterns it encountered during its "life."

Methods: The Dual Form of a Linear Layer

The core of the paper relies on translating the standard view of a neural network layer into its dual form.

In the standard (primal) view, a layer uses a weight matrix $W$ to transform an input $x$ into an output $y$. In the dual view, this exact same operation is expressed as an unnormalized attention mechanism over the model's entire training history.

In the dual form, the layer acts as a memory system storing every training input as keys and corresponding error signals as values.

Key components of this dual system include:

  • Keys ($X$): Every input vector $x_t$ the layer saw during training.
  • Values ($E$): The error signals (gradients) $e_t$ received by the layer at each training step.
  • Query ($x$): The current test input.

Mechanism: The output is calculated by comparing the test query to all training keys using a dot product (which generates attention weights) and then taking a weighted sum of the corresponding error values. Essentially, the model doesn't just "know" the answer; it retrieves relevant pieces of its past training experience to construct a response to new inputs.

Datasets

Because the dual form requires storing every single training input and gradient—a massive memory requirement—the authors conducted experiments on smaller-scale datasets to validate their theory:

  • Image Classification: MNIST (handwritten digits) and Fashion-MNIST (clothing items).
  • Language Modeling: A word-level version of WikiText-2 and a character-level model trained on Aesop’s Fables.

Key Insights and Ideas

  • Nothing is "Forgotten": In the dual form, every training step is recorded. The authors argue that "forgetting" in neural networks isn't necessarily the deletion of data, but rather a failure of the retrieval process. If the attention mechanism can't distinguish between relevant and irrelevant memories, performance drops.
  • Spotlights of Attention: The attention weights act as "spotlights," showing which training samples are activated by a test input. For example, a test image of a "6" strongly attends to training images of "6"s.
  • Interference Visualized: In continual learning (learning Task A, then Task B), models often catastrophically forget Task A. The dual form shows why: when shown a Task A input, the model's "spotlight" accidentally hits Task B memories, causing retrieval interference.
  • Layer-wise Abstraction: Early layers might attend to raw pixel overlaps, but deeper layers attend to training samples that share high-level semantic meaning with the test input.

Key Results

The authors validated their theory through several experiments:

  1. Correlation with Output: They found that the sum of attention weights for a specific class correlates strongly with the model's final prediction, especially in the later layers.
  2. Multi-Task Insights: When trained on both MNIST and Fashion-MNIST, the model showed "cross-task" attention. A test image of a boot (Fashion-MNIST) might attend to training images of the digit "9" (MNIST) in deeper layers if their internal representations share similarities.
Attention weights across layers. Early layers (layer-0) exhibit localized focus, while later layers (layer-2) show broader, distributed attention across tasks.
  1. Contextual NLP Retrieval: In language models, the dual form successfully retrieved training passages with similar sentence structures and themes. For instance, a query about a "warship" caused the model to attend to training segments about other warships, even if the specific vocabulary differed.

Limitations and Future Work

The primary limitation of this framework is scalability. Storing every training input and gradient for a massive model like GPT-3 would require roughly 14 Petabytes of storage. While technically feasible for a large organization, it is currently impractical for standard research environments.

Furthermore, the analysis is currently "post-hoc"—it explains what a trained model is doing but doesn't yet provide a direct mechanism to improve the training process itself. Future work could involve using more powerful kernels (like softmax) within this dual framework to create more robust, inherently memory-based models.

Summary updated

Abstract

Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs u...
BibTeX
@inproceedings{irie2022dual,
  author = {Kazuki Irie and Róbert Csordás and Jürgen Schmidhuber},
  title = {{The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=ckUWWZQF3Ni},
}
Figure