Abstract
DenseNet architectures are renowned for their strong feature extraction capabilities, yet their dense connection patterns often lead to significant parameter redundancy and computational bottlenecks. This paper introduces S-DenseNet, a compact variant that utilizes a novel "Skyline" optimization method to automatically convert standard convolutions into sparse, structured group convolutions (S-Conv). By identifying and retaining only the most critical filter weights, S-DenseNet achieves a dramatic reduction in complexity—using only 40% of the parameters and 20% of the FLOPs of the original DenseNet—while simultaneously improving classification accuracy on standard benchmarks like CIFAR and ImageNet.
The Problem: Redundancy in Dense Connections
DenseNet’s core strength is its "dense connectivity," where every layer receives feature maps from all preceding layers. While this promotes feature reuse and mitigates the vanishing gradient problem, it creates a massive scalability issue: as the network grows deeper, the number of input channels explodes. This leads to high computational costs (FLOPs) and memory overhead, making deployment on resource-constrained devices like smartphones or embedded sensors difficult.
Existing compression methods, such as unstructured pruning, often fail to deliver real-world speedups because standard hardware is not optimized for random weight sparsity. The authors sought a way to achieve structured sparsity—specifically through group convolutions—that could be learned automatically during training rather than being manually designed.
Methods: The Skyline Strategy
The core innovation of S-DenseNet is the S-Conv module, which applies the Skyline method (a multi-objective optimization technique) to filter selection:
- Skyline Filter Selection: During training, the algorithm evaluates the importance of filters using their $l_1$ norm. It treats the selection process as a search for "Skyline points"—filters that provide the highest feature importance for the lowest parameter cost.
- Automatic Grouping: Unlike previous models that use a fixed number of groups, S-DenseNet automatically determines the optimal grouping strategy and the number of filters to retain based on a compression threshold ($\tau$).
- AS-LGC (AllSize-LearnedGroupConv): The authors extended the concept of Learned Group Convolutions. While previous methods were limited to $1 \times 1$ filters, S-DenseNet’s version applies to filters of any size, allowing for comprehensive compression across the entire network architecture.
- Three-Stage Training: The process involves an initial training phase, an optimization phase where the network is gradually sparsified according to the Skyline points, and a final fine-tuning phase to restore and enhance performance.
Key Insights and Ideas
- Redundancy as an Opportunity: The authors demonstrated that even a 10% reduction in DenseNet parameters via simple pruning resulted in zero accuracy loss, confirming that the architecture is inherently over-parameterized.
- Structured vs. Unstructured Sparsity: By converting standard convolutions into group convolutions, the model achieves sparsity that is natively supported by standard deep learning frameworks (like PyTorch), leading to actual inference speedups on standard CPUs and GPUs.
- Dynamic Routing: The relationship between input and output channels is not fixed. By letting the Skyline method dictate which inputs connect to which outputs, the network learns a more efficient internal routing of information than a human engineer might design.
As shown in the heatmap above, the S-Conv module learns a structured connection pattern where only the most informative channels are preserved, effectively "trimming the fat" from the dense connections while maintaining a format that hardware can process efficiently.
Key Results
- Efficiency Gains: On CIFAR datasets, S-DenseNet achieved better accuracy than the baseline DenseNet while using 60% fewer parameters and 80% fewer FLOPs.
- Accuracy Boost: Surprisingly, the compressed model often outperformed the original. For instance, on CIFAR-10, S-DenseNet reduced the error rate by approximately 1% compared to the standard DenseNet-100.
- Inference Speed: In real-world testing on a single GPU, S-DenseNet-86 was 22% faster at inference time than DenseNet-40, despite having a more complex underlying topology.
- ImageNet Performance: S-DenseNet achieved a Top-1 accuracy of 73.88%, outperforming other lightweight models like MobileNet-V1 and CondenseNet while maintaining lower complexity.
Conclusion and Future Work
S-DenseNet demonstrates that "dense" connections do not have to be computationally expensive. By using the Skyline method to automate the discovery of structured sparsity, the authors have created a model that is both more accurate and significantly lighter than its predecessor. Future work may involve applying this Skyline-based grouping strategy to other architectures, such as ResNet or Transformer-based models, to further push the boundaries of efficient deep learning.
Summary updated