S-DenseNet: A DenseNet Compression Model Based on Convolution Grouping Strategy Using Skyline Method

Changyong Yu, Xin He, Haitao Ma, Xin Qi, Jiarui Lu, Yuhai ZhaoIEEE Access 2019 · OpenReview

Abstract

DenseNet architectures are renowned for their strong feature extraction capabilities, yet their dense connection patterns often lead to significant parameter redundancy and computational bottlenecks. This paper introduces S-DenseNet, a compact variant that utilizes a novel "Skyline" optimization method to automatically convert standard convolutions into sparse, structured group convolutions (S-Conv). By identifying and retaining only the most critical filter weights, S-DenseNet achieves a dramatic reduction in complexity—using only 40% of the parameters and 20% of the FLOPs of the original DenseNet—while simultaneously improving classification accuracy on standard benchmarks like CIFAR and ImageNet.

The Problem: Redundancy in Dense Connections

DenseNet’s core strength is its "dense connectivity," where every layer receives feature maps from all preceding layers. While this promotes feature reuse and mitigates the vanishing gradient problem, it creates a massive scalability issue: as the network grows deeper, the number of input channels explodes. This leads to high computational costs (FLOPs) and memory overhead, making deployment on resource-constrained devices like smartphones or embedded sensors difficult.

Existing compression methods, such as unstructured pruning, often fail to deliver real-world speedups because standard hardware is not optimized for random weight sparsity. The authors sought a way to achieve structured sparsity—specifically through group convolutions—that could be learned automatically during training rather than being manually designed.

Methods: The Skyline Strategy

The core innovation of S-DenseNet is the S-Conv module, which applies the Skyline method (a multi-objective optimization technique) to filter selection:

  1. Skyline Filter Selection: During training, the algorithm evaluates the importance of filters using their $l_1$ norm. It treats the selection process as a search for "Skyline points"—filters that provide the highest feature importance for the lowest parameter cost.
  2. Automatic Grouping: Unlike previous models that use a fixed number of groups, S-DenseNet automatically determines the optimal grouping strategy and the number of filters to retain based on a compression threshold ($\tau$).
  3. AS-LGC (AllSize-LearnedGroupConv): The authors extended the concept of Learned Group Convolutions. While previous methods were limited to $1 \times 1$ filters, S-DenseNet’s version applies to filters of any size, allowing for comprehensive compression across the entire network architecture.
  4. Three-Stage Training: The process involves an initial training phase, an optimization phase where the network is gradually sparsified according to the Skyline points, and a final fine-tuning phase to restore and enhance performance.

Key Insights and Ideas

  • Redundancy as an Opportunity: The authors demonstrated that even a 10% reduction in DenseNet parameters via simple pruning resulted in zero accuracy loss, confirming that the architecture is inherently over-parameterized.
  • Structured vs. Unstructured Sparsity: By converting standard convolutions into group convolutions, the model achieves sparsity that is natively supported by standard deep learning frameworks (like PyTorch), leading to actual inference speedups on standard CPUs and GPUs.
  • Dynamic Routing: The relationship between input and output channels is not fixed. By letting the Skyline method dictate which inputs connect to which outputs, the network learns a more efficient internal routing of information than a human engineer might design.
Heatmap of learned sparsity in S-Conv, illustrating the dynamic routing between input and output channels.

As shown in the heatmap above, the S-Conv module learns a structured connection pattern where only the most informative channels are preserved, effectively "trimming the fat" from the dense connections while maintaining a format that hardware can process efficiently.

Key Results

  • Efficiency Gains: On CIFAR datasets, S-DenseNet achieved better accuracy than the baseline DenseNet while using 60% fewer parameters and 80% fewer FLOPs.
  • Accuracy Boost: Surprisingly, the compressed model often outperformed the original. For instance, on CIFAR-10, S-DenseNet reduced the error rate by approximately 1% compared to the standard DenseNet-100.
  • Inference Speed: In real-world testing on a single GPU, S-DenseNet-86 was 22% faster at inference time than DenseNet-40, despite having a more complex underlying topology.
  • ImageNet Performance: S-DenseNet achieved a Top-1 accuracy of 73.88%, outperforming other lightweight models like MobileNet-V1 and CondenseNet while maintaining lower complexity.

Conclusion and Future Work

S-DenseNet demonstrates that "dense" connections do not have to be computationally expensive. By using the Skyline method to automate the discovery of structured sparsity, the authors have created a model that is both more accurate and significantly lighter than its predecessor. Future work may involve applying this Skyline-based grouping strategy to other architectures, such as ResNet or Transformer-based models, to further push the boundaries of efficient deep learning.

Summary updated

Abstract

DenseNet has strong expressiveness in many computer vision tasks, but the complexity of its model makes it difficult to deploy to devices with limited computing resources. In this paper, we propose S-DenseNet, a compact model of DenseNet which makes the extracting features of DenseNet more comprehensively and reduces the parameter redundancy at the same time. Firstly, we investigate the parameter redundancy of DenseNet and show the possibility of compressing DenseNet. For example, we reduced its parameters by 10% as keeping in the same accuracy. Secondly, we propose an algorithm based on the Skyline method for converting standard convolution into group convolution automatically. It can choose the filter weights automatically and restore the standard convolution filter with a high sparsity one called S-Conv. Finally, we combine S-Conv with dense connections and formed a more effective network architecture, that is S-DenseNet. Experiments show that in Cifar10, Cifar100 and SVHN datasets, we use only 40% parameters and 20% FLOPs of DenseNet, and the accuracy is improved by 1%. Compared with the compression models of DenseNet and other CNNs, we use less complexity in the model and achieve higher or similar accuracy. On ImageNet dataset, compared with lightweight CNN models, traditional CNNs and their compression models, we achieve the higher or similar Top-1 accuracy with less complexity.
BibTeX
@inproceedings{yu2019s,
  author = {Changyong Yu and Xin He and Haitao Ma and Xin Qi and Jiarui Lu and Yuhai Zhao},
  title = {{S-DenseNet: A DenseNet Compression Model Based on Convolution Grouping Strategy Using Skyline Method}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=s5GrOEA9ID},
}
Figure