Abstract
This paper introduces Entangled Residual Mappings, a generalization of the standard identity skip connections used in modern deep learning architectures like ResNets and Transformers. By replacing the identity matrix with specialized "entangled" matrices—such as sparse, orthogonal, or structured tensor mappings—we demonstrate that incorporating specific structural biases into the skip pathway can significantly improve generalization performance. We evaluate our approach across computer vision, sequence modeling, and sentiment analysis tasks, showing consistent gains while maintaining the core benefits of iterative feature refinement.
1. Introduction
Since the introduction of Residual Networks (ResNets), the skip connection ($y = f(x) + x$) has become a cornerstone of deep learning. Its success is primarily attributed to its ability to prevent vanishing gradients and facilitate iterative feature refinement, where each layer learns to adjust the previous layer's representation rather than constructing it from scratch.
Despite its ubiquity, the choice of the identity mapping ($+x$) is often treated as a default necessity. This paper challenges that assumption by asking whether the identity matrix is truly the optimal prior for these connections. We propose "entangling" the skip pathway with specific structural biases—such as sparsity or orthogonality—to help networks learn superior representations while preserving the fundamental mechanism of iterative refinement.
2. Methodology
We propose the Entangled Residual Block, defined by the mapping:
$$R_f(x) := f(x) + \Gamma x$$
where $\Gamma$ is a constant matrix that shares key properties with the identity matrix (e.g., eigenvalues near 1) but introduces new structural inductive biases.
2.1 Entanglement Variations
We explore several forms of entanglement:
- Sparse Entanglement: Controlled by a hyperparameter $\gamma$, this distributes signals across dimensions. At $\gamma=0$, it reduces to a standard identity; at $\gamma=1$, it becomes a fully entangled average.
- Orthogonal Mappings: These utilize orthogonal matrices in the skip path to preserve the signal's norm, which is particularly relevant for maintaining stability in deep architectures.
- Tensor Entanglement: For multidimensional data in CNNs and Transformers, $\Gamma$ is implemented as a $(k-1)$-rank convolution (e.g., $1 \times 1$ or $3 \times 3$ kernels) to entangle spatial or channel dimensions.
Crucially, we provide a mathematical proof (Lemma 1) demonstrating that as long as the entanglement coefficient $\gamma$ remains small, the network preserves its capacity for iterative feature refinement.
3. Experimental Results
The proposed method was evaluated across a diverse set of benchmarks, including ImageNet (ILSVRC-2012), CIFAR-10/100, Sequential MNIST, and the IMDB sentiment dataset.
3.1 Key Findings
- Computer Vision: On ImageNet, an "Entangled+" ResNet50 achieved 76.69% top-1 accuracy, outperforming the standard ResNet50 baseline of 76.12%. Similarly, for Vision Transformers (ViT-S), channel-wise entanglement ($\gamma=0.1$) improved accuracy from 72.75% to 73.14%.
- Sequence Modeling: In Recurrent Neural Networks (RNNs), orthogonal skip connections provided a powerful inductive bias for "time-variant" sequences. On the Permuted Sequential MNIST task, the Orthogonal LSTM achieved 95.74% accuracy, significantly outperforming the standard LSTM's 93.31%.
- Sparsity and Spatial Bias: For CNNs and ViTs, small amounts of spatial entanglement—allowing the skip connection to incorporate information from neighboring pixels—consistently improved generalization.
4. Discussion and Conclusion
Our results suggest that the identity mapping should be viewed as a specific prior that assumes features should remain largely invariant across layers. Entanglement allows for the adjustment of this prior to better suit the task at hand. However, we observe that iterative refinement is fragile; excessive entanglement (high $\gamma$) leads to performance degradation as the network loses its ability to refine features incrementally.
While the performance gains are often marginal (typically <1%), they are consistent across architectures. Future work will explore making these entanglement matrices learnable or dynamic, potentially allowing the network to adaptively determine the optimal level of mixing at different depths.
Summary updated