Introduction
Modern Convolutional Neural Networks (CNNs) are characterized by extreme over-parameterization, often containing millions of redundant parameters. This "weight bloat" hinders deployment on edge devices, increases computational costs, and complicates model interpretation. ACDC (Atom-Coefficient Decomposed Convolution) addresses this by questioning the independence of convolutional kernels across layers. If layers perform similar feature extraction tasks, they should be able to share the underlying "recipe" for their filters.
Methodology: Atom-Coefficient Decomposition
The fundamental innovation of ACDC is the mathematical decomposition of the convolutional kernel $K$ into two distinct components: $K = DA$.
- Dictionary Atoms ($D$): A small set of trainable 2D spatial filters representing fundamental "shapes" or patterns (e.g., 8 or 16 atoms).
- Coefficients ($A$): Weights that determine how to linearly combine these atoms to form the final convolutional filters.
ACDC introduces Coefficient Sharing across different architectural levels to maximize efficiency:
- ACDC-net: Shares the same set of coefficients across the entire network.
- ACDC-block: Shares coefficients within specific blocks (e.g., all layers operating at the same spatial resolution).
- ACDC-group: Breaks a single layer into groups that share coefficients, utilizing "channel shuffling" to maintain information flow between groups.
To improve training robustness and prevent over-reliance on specific spatial patterns, the authors introduce Atom-drop. Similar to Dropout, this regularization technique randomly zeroes out dictionary atoms during the training phase.
Theoretical Foundation and Interpretability
The feasibility of weight sharing is supported by Canonical Correlation Analysis (CCA), which reveals that coefficients in standard CNNs naturally reside in a low-rank structure. This empirical finding provides the theoretical justification for the ACDC decomposition, suggesting that the expressiveness of the network is not significantly compromised by sharing weights.
Furthermore, ACDC enhances global interpretability. Since coefficients are shared across layers, Class Activation Mapping (CAM) can be applied consistently from shallow to deep layers using the same weighting scheme. Experiments on the CUB-200 bird classification dataset demonstrate how this shared structure allows for a clear visualization of how the network refines its focus from noisy edges in early layers to distinct object features in deeper layers.
Experimental Results
The authors validated ACDC across several standard benchmarks, including CIFAR-10/100, TinyImageNet, and miniImageNet. The results demonstrate that ACDC can drastically reduce parameter counts while maintaining or even improving classification accuracy.
As shown in the table above, ACDC-net applied to VGG16 reduces parameters from 14.72M to 4.21M (with $m=16$) while actually decreasing the error rate from 6.2% to 5.44%. For ResNet-18, ACDC-net achieves comparable performance to the baseline with only 5% of the original parameters. The "group" variants (e.g., ACDC-g-layer) offer even more extreme compression, reducing VGG16 to 0.80M parameters with a 5.67% error rate. Additionally, the grouping variants significantly reduce Floating Point Operations (FLOPs), cutting VGG16's computational load by nearly 4x.
Beyond standard classification, ACDC shows superior task adaptation in few-shot learning scenarios (e.g., on miniImageNet). The shared structure likely acts as a form of inductive bias that prevents the model from "forgetting" general features while fine-tuning on very small datasets.
Conclusion
ACDC provides a highly efficient, plug-and-play replacement for standard convolutional layers. By exploiting the low-rank structure of filter coefficients and enforcing sharing, it achieves up to 98% parameter reduction. While currently focused on classification, future work will explore ACDC's performance in dense prediction tasks like object detection and semantic segmentation, as well as optimizing its "wall-clock" execution speed on modern hardware.
Summary updated