ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution

Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, Qiang Qiu

Introduction

Modern Convolutional Neural Networks (CNNs) are characterized by extreme over-parameterization, often containing millions of redundant parameters. This "weight bloat" hinders deployment on edge devices, increases computational costs, and complicates model interpretation. ACDC (Atom-Coefficient Decomposed Convolution) addresses this by questioning the independence of convolutional kernels across layers. If layers perform similar feature extraction tasks, they should be able to share the underlying "recipe" for their filters.

Methodology: Atom-Coefficient Decomposition

The fundamental innovation of ACDC is the mathematical decomposition of the convolutional kernel $K$ into two distinct components: $K = DA$.

  1. Dictionary Atoms ($D$): A small set of trainable 2D spatial filters representing fundamental "shapes" or patterns (e.g., 8 or 16 atoms).
  2. Coefficients ($A$): Weights that determine how to linearly combine these atoms to form the final convolutional filters.
The ACDC framework decomposes kernels into shared coefficients and unique dictionary atoms across layers.

ACDC introduces Coefficient Sharing across different architectural levels to maximize efficiency:

  • ACDC-net: Shares the same set of coefficients across the entire network.
  • ACDC-block: Shares coefficients within specific blocks (e.g., all layers operating at the same spatial resolution).
  • ACDC-group: Breaks a single layer into groups that share coefficients, utilizing "channel shuffling" to maintain information flow between groups.

To improve training robustness and prevent over-reliance on specific spatial patterns, the authors introduce Atom-drop. Similar to Dropout, this regularization technique randomly zeroes out dictionary atoms during the training phase.

Theoretical Foundation and Interpretability

The feasibility of weight sharing is supported by Canonical Correlation Analysis (CCA), which reveals that coefficients in standard CNNs naturally reside in a low-rank structure. This empirical finding provides the theoretical justification for the ACDC decomposition, suggesting that the expressiveness of the network is not significantly compromised by sharing weights.

Furthermore, ACDC enhances global interpretability. Since coefficients are shared across layers, Class Activation Mapping (CAM) can be applied consistently from shallow to deep layers using the same weighting scheme. Experiments on the CUB-200 bird classification dataset demonstrate how this shared structure allows for a clear visualization of how the network refines its focus from noisy edges in early layers to distinct object features in deeper layers.

Experimental Results

The authors validated ACDC across several standard benchmarks, including CIFAR-10/100, TinyImageNet, and miniImageNet. The results demonstrate that ACDC can drastically reduce parameter counts while maintaining or even improving classification accuracy.

Performance comparison of ACDC variants against baseline architectures on CIFAR-10.

As shown in the table above, ACDC-net applied to VGG16 reduces parameters from 14.72M to 4.21M (with $m=16$) while actually decreasing the error rate from 6.2% to 5.44%. For ResNet-18, ACDC-net achieves comparable performance to the baseline with only 5% of the original parameters. The "group" variants (e.g., ACDC-g-layer) offer even more extreme compression, reducing VGG16 to 0.80M parameters with a 5.67% error rate. Additionally, the grouping variants significantly reduce Floating Point Operations (FLOPs), cutting VGG16's computational load by nearly 4x.

Beyond standard classification, ACDC shows superior task adaptation in few-shot learning scenarios (e.g., on miniImageNet). The shared structure likely acts as a form of inductive bias that prevents the model from "forgetting" general features while fine-tuning on very small datasets.

Conclusion

ACDC provides a highly efficient, plug-and-play replacement for standard convolutional layers. By exploiting the low-rank structure of filter coefficients and enforcing sharing, it achieves up to 98% parameter reduction. While currently focused on classification, future work will explore ACDC's performance in dense prediction tasks like object detection and semantic segmentation, as well as optimizing its "wall-clock" execution speed on modern hardware.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) are known to be significantly over-parametrized, and difficult to interpret, train and adapt. In this paper, we introduce a structural regularization across convolutional kernels in a CNN. In our approach, each convolution kernel is first decomposed as 2D dictionary atoms linearly combined by coefficients. The widely observed correlation and redundancy in a CNN hint a common low-rank structure among the decomposed coefficients, which is here further supported by our empirical observations. We then explicitly regularize CNN kernels by enforcing decomposed coefficients to be shared across sub-structures, while leaving each sub-structure only its own dictionary atoms, a few hundreds of parameters typically, which leads to dramatic model reductions. We explore models with sharing across different sub-structures to cover a wide range of trade-offs between parameter reduction and expressiveness. Our proposed regularized network structures open the door to better interpreting, training and adapting deep models. We validate the flexibility and compatibility of our method by image classification experiments on multiple datasets and underlying network structures, and show that CNNs now maintain performance with dramatic reduction in parameters and computations, e.g., only 5\% parameters are used in a ResNet-18 to achieve comparable performance. Further experiments on few-shot classification show that faster and more robust task adaptation is obtained in comparison with models with standard convolutions.
BibTeX
@misc{wang2020acdc,
  author = {Ze Wang and Xiuyuan Cheng and Guillermo Sapiro and Qiang Qiu},
  title = {{ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution}},
  year = {2020},
  eprint = {2009.02386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.02386},
}
Figure