Abstract
Running backpropagation end to end on large neural networks is fraught with difficulties like vanishing gradients and degradation. In this paper we present an alternative architecture composed of many small neural networks that interact with one another. Instead of propagating gradients back through the architecture we propagate vector-valued messages computed via forward passes, which are then used to update the parameters. Currently the performance is conjectured as we are yet to implement the architecture. However, we do back it up with some theory. A previous version of this paper was entitled "Fusion encoder networks" and detailed a slightly different architecture.
Introduction
Training very deep neural networks using standard backpropagation is notoriously difficult. As gradients are passed backward through many layers, they often "vanish" or "explode," leading to model degradation. While modern techniques like Residual Connections and Batch Normalization mitigate these issues, they still rely on a single, global gradient signal. We propose a radical departure: decoupling the architecture into many small, modular components that communicate via forward-pass messages rather than global error signals.
Methods: Extraction Propagation (Xprop)
The proposed system, the Extraction Network (exnet), is a Directed Acyclic Graph (DAG) where each vertex and arc is associated with a small, independent neural network. The system operates via the Extraction Propagation (Xprop) algorithm, consisting of three phases:
- The Up-Pass: Input data is "tokenized" at the leaves. Primary propagator networks ($\phi$) compute vector-valued "primary extractions" recursively from the leaves up to the root ($\psi$), where a final prediction $y$ is made.
- The Down-Pass: Starting from the root, "complementary extractions" are computed by complementary propagator networks ($\phi^\dagger$) and passed back down. These messages represent the context or missing information from the rest of the network.
- The Parameter Update: Instead of global backprop, each component is updated using local backpropagation. The target for these updates is derived from combining primary and complementary extractions, allowing nodes to learn how to contribute to the global goal using only local information.
Key Insights
- Local Learning, Global Goals: By having access to both descendant information (primary extraction) and global context (complementary extraction), a node can approximate an optimal local update without a top-down gradient.
- Belief Propagation Inspiration: The scheduling of messages in Xprop is inspired by classic graphical model algorithms.
- Structural Versatility: Exnets can theoretically implement tree structures for sequences/images, multi-layer perceptrons, and even decentralized Attention-based architectures (Transformers).
- Vertex Optimality: Theoretical proofs suggest that if individual components converge locally, the entire system can propagate information such that any internal node could produce the optimal global prediction.
Limitations and Future Work
The primary limitation is the lack of empirical implementation; performance remains conjectured. Additionally, the interaction between many learning components may lead to instability, and the proposed Xprop* variant for handling update "lag" has high space complexity. Future work will focus on empirical validation against standard architectures like ResNets and Transformers.
Summary updated