Rethinking skip connection model as a learnable Markov chain

Dengsheng Chen, Jie Hu, Wenwen Qiang, Xiaoming Wei, Enhua Wu

Introduction

Skip connections have become the industry standard for building deep neural networks, serving as the backbone for architectures like ResNet and Transformers. While it is widely accepted that they aid optimization and prevent vanishing gradients, our theoretical understanding of exactly how they navigate the internal feature space remains incomplete.

This paper reinterprets skip-connection models as learnable Markov chains, where each layer attempts to shift data closer to a target domain. By introducing a simple, one-line code modification called a "penal connection," the authors regularize the model to follow a more efficient optimization path. This approach leads to faster convergence and better performance across both computer vision and natural language processing tasks.

The Problem: Inefficient Optimization Paths

Despite the success of skip connections, standard optimizers like SGD and Adam are often "lazy." They tend to settle for inefficient, zig-zagging paths through the network's layers rather than finding the most direct route to the target domain.

This inefficiency contributes to model degradation, a phenomenon where adding more layers eventually stops improving—or even degrades—performance because the model gets trapped in local optima. The central challenge addressed by this paper is how to force a deep network to learn the most direct and efficient transformation from input to output.

Inefficient vs. Efficient Optimization Paths

Methodology: The Markov Chain Perspective

To solve this routing inefficiency, the researchers propose a novel framework that views the network through the lens of probability and state transitions:

  1. The Markov Chain View: The authors treat each residual block as a node in a Markov chain. The output of a given layer ($x_l$) depends only on the previous state ($x_{l-1}$) and a "predicted direction" ($z_l$) provided by the residual block.
  2. Ideal vs. Predicted Direction: They define an "ideal direction" ($d_l$)—the theoretical path that would most effectively reduce the loss. They then introduce an efficiency indicator ($\\epsilon$) based on the cosine similarity between the model's actual prediction ($z_l$) and this ideal path ($d_l$).
Markov Chain Framework and Directional Vectors
  1. Penal Connection: To optimize for efficiency, a penalty term is added to the loss function. This encourages the model to align its predictions with the ideal direction.

Through mathematical derivation, the authors demonstrate that this complex regularization can be implemented as a remarkably simple gradient modification. In PyTorch, it requires just one line of code during the backward pass:

zl.register_hook(lambda gxl: gxl + tau * zl)

This effectively adds a scaled version of the residual feature to its own gradient during backpropagation, acting as a tension wire that "unfolds" the Markov chain so every layer contributes a non-negative push toward the target.

Experimental Validation & Results

The authors validated their approach across several standard benchmarks, demonstrating significant improvements in both efficiency and accuracy:

  • Translation (WMT16 English-German): When testing the "Markov" version of the Transformer architecture, the approach reduced Perplexity (PPL) from 33.3 to 7.9 and boosted accuracy from 36.4% to 70.0% under the "Setting Q" training schedule. Remarkably, the efficient Markov chain reached better results in 200 epochs than the baseline did in 2,000 epochs—a 10x speedup in convergence.
  • Image Recognition (ImageNet-1K): The method provided a consistent boost of +0.2% to +0.3% top-1 accuracy across various architectures, including ResNet50, ViT, DeiT, and Swin Transformers.
  • Depth Scaling (CIFAR-10 & CIFAR-100): Usually, as ResNets get deeper, their accuracy eventually plateaus. By viewing the model as a Markov chain, the "penalized" models continued to gain accuracy even at extreme depths. On CIFAR-100, a 50-layer Markov-ResNet achieved 79.1% accuracy, significantly outperforming the standard ResNet-50 (78.5%).

Limitations and Future Directions

While the results are compelling, the approach does introduce a new hyperparameter, $\\tau$, which dictates the strength of the penal connection. The optimal value of $\\tau$ varies significantly across different tasks (e.g., $10^{-4}$ for NLP versus $10^{-7}$ for Computer Vision).

Furthermore, the "ideal direction" is currently calculated based on the loss function. The authors acknowledge that different choices of distance functions could yield different results, making the search for the absolute "most ideal" direction an open research question. Nonetheless, this paper provides a strong empirical and theoretical foundation for rethinking how deep networks route information, paving the way for more efficient optimization strategies.

Summary updated

Abstract

Over the past few years afterward the birth of ResNet, skip connection has become the defacto standard for the design of modern architectures due to its widespread adoption, easy optimization, and proven performance. Prior work has explained the effectiveness of the skip connection mechanism from different perspectives. In this work, we deep dive into the model's behaviors with skip connections which can be formulated as a learnable Markov chain. An efficient Markov chain is preferred as it always maps the input data to the target domain in a better way. However, while a model is explained as a Markov chain, it is not guaranteed to be optimized following an efficient Markov chain by existing SGD-based optimizers prone to getting trapped in local optimal points. In order to move towards a more efficient Markov chain, we propose a simple routine of penal connection to make any residual-like model become a learnable Markov chain. Aside from that, the penal connection can also be viewed as a particular model regularization and can be easily implemented with one line of code in the most popular deep learning frameworks. The encouraging experimental results in multi-modal translation and image recognition empirically confirm our conjecture of the learnable Markov chain view and demonstrate the superiority of the proposed penal connection.
BibTeX
@inproceedings{chen2023rethinking,
  author = {Dengsheng Chen and Jie Hu and Wenwen Qiang and Xiaoming Wei and Enhua Wu},
  title = {{Rethinking skip connection model as a learnable Markov chain}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=yQdBtFfleh6},
}
Figure