Abstract
Deep neural networks often suffer from vanishing gradients, a problem traditionally mitigated by identity skip-connections in Residual Networks (ResNets). This paper investigates a broader class of linear transformations—specifically orthogonal and idempotent matrices—as alternatives to the identity mapping. We demonstrate that these transformations maintain gradient stability and information flow while enabling superior feature mixing in multi-branch architectures.
Introduction
The success of deep learning is largely attributed to architectural innovations that facilitate the training of very deep models. Residual Networks (ResNets) introduced identity skip-connections to create "expressways" for gradient flow, effectively bypassing the vanishing gradient problem. While identity mappings are highly effective, they are a specific case of a more general class of linear transformations. This research explores whether other mathematical structures, such as orthogonal and idempotent matrices, can provide similar stability while offering enhanced representational power, particularly in complex multi-branch networks where identity mappings may limit inter-branch interaction.
Proposed Methods
The authors propose a generalized residual building block defined by the equation: $$y = Px + F(x, W)$$ where $P$ is a transformation matrix. Two primary types of transformations are explored:
- Orthogonal Transformations: These utilize matrices $Q$ such that $Q^T Q = I$. Orthogonal matrices preserve the vector norm, ensuring that signals neither explode nor vanish during propagation. A significant property is that the composition of orthogonal transformations remains orthogonal, maintaining stability across many layers.
- Idempotent Transformations: These utilize matrices $P$ such that $P^2 = P$. In a deep sequence, multiple idempotent skip-connections mathematically collapse into a single effective transformation, regardless of the network's depth.
The study evaluates these transformations using specific configurations like "Merge-and-Run" (Idempotent-MR) and Kronecker product-based orthogonal matrices (Orthogonal-TP).
Theoretical and Empirical Insights
A key theoretical contribution is the "Identity Equivalence" theorem. It states that for single-branch networks, orthogonal skip-connections are mathematically equivalent to standard identity connections, provided the weights in the residual path $F(x, W)$ are appropriately adjusted. This implies that the primary benefit of skip-connections lies in the existence of a stable pathway rather than the specific use of an identity matrix.
However, the advantages of these new transformations become evident in multi-branch networks. Unlike identity mappings, which keep parallel branches isolated, orthogonal and idempotent transforms facilitate information "mixing" across branches, similar to $1 \times 1$ convolutions. This promotes better feature reuse and reduces the "null space" where information might otherwise be lost.
Experimental Results
The proposed methods were benchmarked on CIFAR-10, CIFAR-100, and SVHN datasets.
- Single-Branch Performance: On CIFAR-100 with a depth of 56, the Idempotent-CMR approach achieved 73.05% accuracy, performing comparably to the Identity baseline of 73.41%.
- Multi-Branch Superiority: The proposed transforms significantly outperformed identity mappings in multi-branch settings. In a 4-branch configuration on CIFAR-100 (Depth 56), Orthogonal-TP reached 72.71% accuracy, a substantial improvement over the 69.52% achieved by the Identity baseline.
- Scalability: In extreme multi-branch scenarios where each branch consists of a single channel, orthogonal and idempotent methods consistently surpassed identity mappings across all tested benchmarks.
Conclusion and Future Directions
This work demonstrates that orthogonal and idempotent transformations are viable and often superior alternatives to identity skip-connections, especially in modern multi-branch architectures. While the current study utilizes fixed transformation matrices, future research could explore learnable transformations that adapt to specific data distributions. Additionally, further investigation into the practical computational overhead and memory footprint in large-scale production environments would be beneficial for real-world deployment.
Summary updated