Abstract
Deep neural networks are often criticized as "black boxes" due to the difficulty in tracing specific predictions back to training data. This paper revisits the mathematical dual form of linear layers, demonstrating that any linear layer trained via gradient descent is equivalent to an unnormalized dot-product attention mechanism. In this dual representation, training inputs serve as keys and error signals serve as values. This perspective allows for precise identification of the training examples that influence a model's output at test time, providing a novel framework for interpretability and understanding phenomena like catastrophic forgetting.
Introduction
While deep learning has achieved remarkable success, understanding why a model makes a specific prediction remains a challenge. Existing interpretability methods often focus on identifying salient features in the input (the "what"). However, they rarely explain the origin of the model's knowledge (the "where" in the training history). This research bridges that gap by reviving the dual form of the perceptron, extending it to modern neural architectures.
The Dual Form as Attention
The central contribution of this work is the mathematical proof that the standard "primal" forward pass of a linear layer ($y = Wx$) is identical to a "dual" attention-based retrieval system.
In the dual form, the weight matrix $W$ is replaced by a memory of the entire training history. When a test input $x$ (the query) is presented:
- It is compared against all training inputs $x_t$ (the keys).
- The resulting attention weights determine the contribution of the corresponding error signals $e_t$ (the values).
- The final output is a weighted sum of these error signals.
This equivalence implies that neural networks do not just store patterns; they store the "corrections" they made during training. Every prediction is essentially a retrieval of relevant lessons learned from past mistakes.
Experimental Results and Interpretability
The authors validated this framework on datasets including MNIST, Fashion-MNIST, and WikiText-2. By analyzing the "spotlights of attention"—the specific training samples receiving high weights during a test pass—they uncovered several key insights.
Class Correlation
A significant finding is that the total attention weight assigned to training samples of a particular class correlates strongly with the model's final classification.
As shown in the figure above, as information flows through the layers, the attention becomes increasingly concentrated on the "correct" class. In deeper layers, the model's prediction is driven by a specific subset of training examples that share semantic similarities with the query.
Visualizing Forgetting and Interference
The dual form provides a unique lens into catastrophic forgetting in continual learning. By visualizing attention weights, the researchers showed that when a model is trained on a new task (e.g., Fashion-MNIST) after an old one (e.g., MNIST), the retrieval mechanism becomes "polluted." A test image of a digit might erroneously trigger high attention toward clothing items, explaining the drop in performance on the original task.
Discussion and Limitations
The primary hurdle for the dual form is scalability. Storing every training input and gradient is memory-prohibitive for large-scale models like GPT-3, which would require petabytes of storage in this format. However, the theoretical insight remains profound: self-attention in Transformers can be viewed as a hierarchical retrieval process, combining information from the current context with "memories" of the training set.
Future research directions include developing compression techniques, such as kernel sketching, to make the dual form practical for larger models, and exploring whether training directly in the dual space can improve robustness.
Summary updated