Rethinking skip connection model as a learnable Markov chain

Dengsheng Chen, Jie Hu, Wenwen Qiang, Xiaoming Wei, Enhua Wu

Introduction

Residual-style architectures, characterized by skip connections ($x_l = x_{l-1} + f(x_{l-1})$), have become the cornerstone of modern deep learning, from ResNets to Transformers. While their empirical success in mitigating vanishing gradients is well-documented, a deeper theoretical understanding of their optimization dynamics remains an active area of research. This paper, presented at ICLR 2023, proposes a novel perspective: viewing skip connections as nodes in a learnable Markov chain. This framework reveals that standard optimizers often follow inefficient paths through feature space, leading to the development of a simple "penal connection" regularization that significantly improves training efficiency and model performance.

The Problem: Optimizer Laziness

A key observation of the authors is that standard SGD-based optimizers are "lazy." They tend to find paths through the network that are easy to optimize—characterized by smooth loss landscapes—but these paths are often not the most direct or efficient way to transform input data into the target output.

Figure 3: Comparison between an inefficient, easy-to-learn path (black) and an efficient, hard-to-learn path (red) through a loss landscape.

As illustrated in Figure 3, without specific regularization, a model may "zigzag" through feature space. In mathematical terms, the "predicted direction" provided by a residual block often deviates significantly from the "ideal direction" required to reach the target label. This inefficiency leads to slower convergence and can contribute to model degradation in very deep networks.

Methodology: Skip Connections as Markov Chains

The authors formalize the forward pass of a residual network as a state transition process. Each residual block is treated as a node $n_l$ in a Markov chain, where the input $x_{l-1}$ represents the current state and the residual output $z_l$ is the predicted transition to the next state.

Figure 1: (a) Architecture of a Markov chain node. (b) Geometric interpretation of the transition from source domain to target domain, highlighting the discrepancy between the predicted direction and the ideal direction.

To quantify the efficiency of these transitions, the authors introduce an Efficiency Indicator ($\epsilon$) based on the cosine similarity between the predicted direction $z_l$ and an "ideal direction" $d_l$ (defined as the gradient of the loss with respect to the current state). A perfectly efficient chain would have $z_l$ aligned exactly with $d_l$.

The Penal Connection

To force the model toward a more efficient, "$\delta$-convex" path, the authors propose the Penal Connection. This is a regularization term that penalizes the norm of the residual output $z_l$ in a way that encourages alignment with the ideal direction.

The theoretical derivation leads to a remarkably simple implementation. In PyTorch, this can be achieved with a single line of code using a gradient hook: zl.register_hook(lambda gxl: gxl + tau * zl)

Here, zl is the residual output, gxl is the gradient of the loss with respect to the state, and $\tau$ is a small hyperparameter. This hook modifies the gradient during backpropagation, effectively guiding the optimizer toward more direct paths.

Key Results

The effectiveness of the penal connection was validated across multiple domains and architectures:

  • Natural Language Processing: In WMT16 translation tasks, the Markov chain model achieved superior performance in just 200 epochs compared to a baseline Transformer trained for 2000 epochs, representing a 10x improvement in convergence speed.
  • Computer Vision: On ImageNet-1k, the method provided consistent top-1 accuracy gains of +0.2% to +0.3% across ResNet, ViT, and Swin Transformer architectures.
  • Depth Resilience: The approach successfully mitigated model degradation in deep networks. On CIFAR-100, a 152-layer ResNet with penal connections continued to improve, while the standard baseline's performance saturated.

Conclusion

By rethinking skip connections through the lens of learnable Markov chains, this work identifies a fundamental inefficiency in standard deep learning optimization. The proposed penal connection offers a computationally inexpensive yet theoretically grounded method to improve the efficiency of feature transformations. Its universal applicability across CNNs and Transformers makes it a versatile tool for enhancing modern neural architectures.

Summary updated

BibTeX
@inproceedings{chen2023rethinking,
  author = {Dengsheng Chen and Jie Hu and Wenwen Qiang and Xiaoming Wei and Enhua Wu},
  title = {{Rethinking skip connection model as a learnable Markov chain}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=yQdBtFfleh6},
}
Figure