Orthogonal and Idempotent Transformations for Learning Deep Neural Networks

Jingdong Wang, Yajie Xing, Kexin Zhang, Cha ZhangarXiv 2017 · arXiv

Abstract

Deep neural networks often suffer from vanishing gradients, a problem traditionally mitigated by identity skip-connections in Residual Networks (ResNets). This paper investigates a broader class of linear transformations—specifically orthogonal and idempotent matrices—as alternatives to the identity mapping. We demonstrate that these transformations maintain gradient stability and information flow while enabling superior feature mixing in multi-branch architectures.

Introduction

The success of deep learning is largely attributed to architectural innovations that facilitate the training of very deep models. Residual Networks (ResNets) introduced identity skip-connections to create "expressways" for gradient flow, effectively bypassing the vanishing gradient problem. While identity mappings are highly effective, they are a specific case of a more general class of linear transformations. This research explores whether other mathematical structures, such as orthogonal and idempotent matrices, can provide similar stability while offering enhanced representational power, particularly in complex multi-branch networks where identity mappings may limit inter-branch interaction.

Proposed Methods

The authors propose a generalized residual building block defined by the equation: $$y = Px + F(x, W)$$ where $P$ is a transformation matrix. Two primary types of transformations are explored:

  1. Orthogonal Transformations: These utilize matrices $Q$ such that $Q^T Q = I$. Orthogonal matrices preserve the vector norm, ensuring that signals neither explode nor vanish during propagation. A significant property is that the composition of orthogonal transformations remains orthogonal, maintaining stability across many layers.
  2. Idempotent Transformations: These utilize matrices $P$ such that $P^2 = P$. In a deep sequence, multiple idempotent skip-connections mathematically collapse into a single effective transformation, regardless of the network's depth.
Generalized building blocks: (a) general form, (b-c) single-branch variants, (d-e) multi-branch variants.

The study evaluates these transformations using specific configurations like "Merge-and-Run" (Idempotent-MR) and Kronecker product-based orthogonal matrices (Orthogonal-TP).

Theoretical and Empirical Insights

A key theoretical contribution is the "Identity Equivalence" theorem. It states that for single-branch networks, orthogonal skip-connections are mathematically equivalent to standard identity connections, provided the weights in the residual path $F(x, W)$ are appropriately adjusted. This implies that the primary benefit of skip-connections lies in the existence of a stable pathway rather than the specific use of an identity matrix.

However, the advantages of these new transformations become evident in multi-branch networks. Unlike identity mappings, which keep parallel branches isolated, orthogonal and idempotent transforms facilitate information "mixing" across branches, similar to $1 \times 1$ convolutions. This promotes better feature reuse and reduces the "null space" where information might otherwise be lost.

Experimental Results

The proposed methods were benchmarked on CIFAR-10, CIFAR-100, and SVHN datasets.

  • Single-Branch Performance: On CIFAR-100 with a depth of 56, the Idempotent-CMR approach achieved 73.05% accuracy, performing comparably to the Identity baseline of 73.41%.
  • Multi-Branch Superiority: The proposed transforms significantly outperformed identity mappings in multi-branch settings. In a 4-branch configuration on CIFAR-100 (Depth 56), Orthogonal-TP reached 72.71% accuracy, a substantial improvement over the 69.52% achieved by the Identity baseline.
  • Scalability: In extreme multi-branch scenarios where each branch consists of a single channel, orthogonal and idempotent methods consistently surpassed identity mappings across all tested benchmarks.

Conclusion and Future Directions

This work demonstrates that orthogonal and idempotent transformations are viable and often superior alternatives to identity skip-connections, especially in modern multi-branch architectures. While the current study utilizes fixed transformation matrices, future research could explore learnable transformations that adapt to specific data distributions. Additionally, further investigation into the practical computational overhead and memory footprint in large-scale production environments would be beneficial for real-world deployment.

Summary updated

Abstract

Identity transformations, used as skip-connections in residual networks, directly connect convolutional layers close to the input and those close to the output in deep neural networks, improving information flow and thus easing the training. In this paper, we introduce two alternative linear transforms, orthogonal transformation and idempotent transformation. According to the definition and property of orthogonal and idempotent matrices, the product of multiple orthogonal (same idempotent) matrices, used to form linear transformations, is equal to a single orthogonal (idempotent) matrix, resulting in that information flow is improved and the training is eased. One interesting point is that the success essentially stems from feature reuse and gradient reuse in forward and backward propagation for maintaining the information during flow and eliminating the gradient vanishing problem because of the express way through skip-connections. We empirically demonstrate the effectiveness of the proposed two transformations: similar performance in single-branch networks and even superior in multi-branch networks in comparison to identity transformations.
BibTeX
@misc{wang2017orthogonal,
  author = {Jingdong Wang and Yajie Xing and Kexin Zhang and Cha Zhang},
  title = {{Orthogonal and Idempotent Transformations for Learning Deep Neural Networks}},
  year = {2017},
  eprint = {1707.05974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.05974},
}
Figure