Entangled Residual Mappings

Mathias Lechner, Ramin M. Hasani, Zahra Babaiee, Radu Grosu, Daniela Rus, Thomas A. Henzinger, Sepp HochreiterarXiv 2022 · DOI · arXiv

Abstract

This paper introduces Entangled Residual Mappings, a generalization of the standard identity skip connections used in modern deep learning architectures like ResNets and Transformers. By replacing the identity matrix with specialized "entangled" matrices—such as sparse, orthogonal, or structured tensor mappings—we demonstrate that incorporating specific structural biases into the skip pathway can significantly improve generalization performance. We evaluate our approach across computer vision, sequence modeling, and sentiment analysis tasks, showing consistent gains while maintaining the core benefits of iterative feature refinement.

1. Introduction

Since the introduction of Residual Networks (ResNets), the skip connection ($y = f(x) + x$) has become a cornerstone of deep learning. Its success is primarily attributed to its ability to prevent vanishing gradients and facilitate iterative feature refinement, where each layer learns to adjust the previous layer's representation rather than constructing it from scratch.

Despite its ubiquity, the choice of the identity mapping ($+x$) is often treated as a default necessity. This paper challenges that assumption by asking whether the identity matrix is truly the optimal prior for these connections. We propose "entangling" the skip pathway with specific structural biases—such as sparsity or orthogonality—to help networks learn superior representations while preserving the fundamental mechanism of iterative refinement.

2. Methodology

We propose the Entangled Residual Block, defined by the mapping:

$$R_f(x) := f(x) + \Gamma x$$

where $\Gamma$ is a constant matrix that shares key properties with the identity matrix (e.g., eigenvalues near 1) but introduces new structural inductive biases.

Conceptual overview of Entangled Residual Mappings and their implementation across various neural architectures.

2.1 Entanglement Variations

We explore several forms of entanglement:

  • Sparse Entanglement: Controlled by a hyperparameter $\gamma$, this distributes signals across dimensions. At $\gamma=0$, it reduces to a standard identity; at $\gamma=1$, it becomes a fully entangled average.
  • Orthogonal Mappings: These utilize orthogonal matrices in the skip path to preserve the signal's norm, which is particularly relevant for maintaining stability in deep architectures.
  • Tensor Entanglement: For multidimensional data in CNNs and Transformers, $\Gamma$ is implemented as a $(k-1)$-rank convolution (e.g., $1 \times 1$ or $3 \times 3$ kernels) to entangle spatial or channel dimensions.

Crucially, we provide a mathematical proof (Lemma 1) demonstrating that as long as the entanglement coefficient $\gamma$ remains small, the network preserves its capacity for iterative feature refinement.

3. Experimental Results

The proposed method was evaluated across a diverse set of benchmarks, including ImageNet (ILSVRC-2012), CIFAR-10/100, Sequential MNIST, and the IMDB sentiment dataset.

Top-1 accuracy on ImageNet for ResNet variants, comparing Entangled, Identity, and Orthogonal skip connections.

3.1 Key Findings

  • Computer Vision: On ImageNet, an "Entangled+" ResNet50 achieved 76.69% top-1 accuracy, outperforming the standard ResNet50 baseline of 76.12%. Similarly, for Vision Transformers (ViT-S), channel-wise entanglement ($\gamma=0.1$) improved accuracy from 72.75% to 73.14%.
  • Sequence Modeling: In Recurrent Neural Networks (RNNs), orthogonal skip connections provided a powerful inductive bias for "time-variant" sequences. On the Permuted Sequential MNIST task, the Orthogonal LSTM achieved 95.74% accuracy, significantly outperforming the standard LSTM's 93.31%.
  • Sparsity and Spatial Bias: For CNNs and ViTs, small amounts of spatial entanglement—allowing the skip connection to incorporate information from neighboring pixels—consistently improved generalization.

4. Discussion and Conclusion

Our results suggest that the identity mapping should be viewed as a specific prior that assumes features should remain largely invariant across layers. Entanglement allows for the adjustment of this prior to better suit the task at hand. However, we observe that iterative refinement is fragile; excessive entanglement (high $\gamma$) leads to performance degradation as the network loses its ability to refine features incrementally.

While the performance gains are often marginal (typically <1%), they are consistent across architectures. Future work will explore making these entanglement matrices learnable or dynamic, potentially allowing the network to adaptively determine the optimal level of mixing at different depths.

Summary updated

Abstract

Residual mappings have been shown to perform representation learning in the first layers and iterative feature refinement in higher layers. This interplay, combined with their stabilizing effect on the gradient norms, enables them to train very deep networks. In this paper, we take a step further and introduce entangled residual mappings to generalize the structure of the residual connections and evaluate their role in iterative learning representations. An entangled residual mapping replaces the identity skip connections with specialized entangled mappings such as orthogonal, sparse, and structural correlation matrices that share key attributes (eigenvalues, structure, and Jacobian norm) with identity mappings. We show that while entangled mappings can preserve the iterative refinement of features across various deep models, they influence the representation learning process in convolutional networks differently than attention-based models and recurrent neural networks. In general, we find that for CNNs and Vision Transformers entangled sparse mapping can help generalization while orthogonal mappings hurt performance. For recurrent networks, orthogonal residual mappings form an inductive bias for time-variant sequences, which degrades accuracy on time-invariant tasks.
BibTeX
@misc{lechner2022entangled,
  author = {Mathias Lechner and Ramin M. Hasani and Zahra Babaiee and Radu Grosu and Daniela Rus and Thomas A. Henzinger and Sepp Hochreiter},
  title = {{Entangled Residual Mappings}},
  year = {2022},
  eprint = {2206.01261},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.01261},
}
Figure