Abstract: Basing on the analysis by revealing the equivalence of modern networks, we find that both ResNet and DenseNet are essentially derived from the same "dense topology", yet they only differ in the form of connection -- addition (dubbed "inner link") vs. concatenation (dubbed "outer link"). However, both two forms of connections have the superiority and insufficiency. To combine their advantages and avoid certain limitations on representation learning, we present a highly efficient and modularized Mixed Link Network (MixNet) which is equipped with flexible inner link and outer link modules. Consequently, ResNet, DenseNet and Dual Path Network (DPN) can be regarded as a special case of MixNet, respectively. Furthermore, we demonstrate that MixNets can achieve superior efficiency in parameter over the state-of-the-art architectures on many competitive datasets like CIFAR-10/100, SVHN and ImageNet.
Introduction
In the evolution of Convolutional Neural Networks (CNNs), two architectures have emerged as standard-bearers: ResNet, which utilizes element-wise addition (skip connections), and DenseNet, which relies on feature concatenation. While both architectures significantly improve gradient flow and performance, they possess inherent limitations. ResNet’s additive connections can eventually saturate the feature space, potentially impeding information flow in very deep networks. Conversely, DenseNet’s concatenation strategy can lead to substantial feature redundancy, as raw features are repeatedly passed through subsequent layers.
The Unified Dense Topology
A core contribution of this work is the mathematical demonstration that ResNet and DenseNet are fundamentally two sides of the same "dense topology" coin. By recursively expanding the identity mapping equation in ResNet, it becomes evident that the input to any given layer is the sum of all preceding layers' outputs. Thus, both architectures connect every layer to all previous layers; they differ only in the connection function: addition ($+$) for ResNet and concatenation ($||$) for DenseNet.
Mixed Link Networks (MixNet)
To leverage the strengths of both approaches while mitigating their weaknesses, the authors propose Mixed Link Networks (MixNet). MixNet is a modularized architecture that strategically blends two types of connectivity:
- Inner Link Modules: These utilize additive feature merging (similar to ResNet) to facilitate gradient flow and enhance feature expressiveness.
- Outer Link Modules: These utilize concatenation (similar to DenseNet) to preserve raw feature diversity and maintain a direct path for earlier features.
A key innovation in MixNet is the use of shifted (unfixed) inner links. Rather than always adding new features to the same fixed position in the feature map, MixNet shifts the addition position as the network grows. This prevents the saturation observed in standard ResNets and reduces the redundancy found in DenseNets. This generalized framework allows ResNet, DenseNet, and Dual Path Networks (DPN) to be viewed as specific, restricted instances of the MixNet architecture.
Experimental Results and Efficiency
The researchers evaluated MixNet across several competitive benchmarks, including CIFAR-10/100, SVHN, and ImageNet. The results consistently highlight MixNet's superior parameter efficiency:
- CIFAR-100: MixNet-250 outperformed ResNeXt-29 while requiring 57% fewer parameters. It also achieved a lower error rate (17.06%) than DenseNet-BC (17.18%) at a similar complexity level.
- ImageNet: MixNet-141 achieved performance comparable to the state-of-the-art DPN-98 but with 50% fewer parameters (41M vs. 61.6M).
- SVHN: MixNet-100 achieved a 1.57% error rate with only 1.5M parameters, rivaling much larger models like DFN-MR which uses 24.8M parameters.
Conclusion and Future Work
MixNet provides a unified perspective on modern network topologies, proving that the choice between addition and concatenation is not binary but complementary. By blending these links, MixNet achieves state-of-the-art accuracy with significantly lower parameter counts. While the current "shifted addition" heuristic is highly effective, future research may explore automated methods for optimizing link positions and further benchmarking inference latency on diverse hardware backends to complement the observed parameter efficiency.
Summary updated