Abstract
Conventional transforms such as the Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have long been staples of image processing but have seen limited use within neural network architectures. This paper proposes applying these transforms to pointwise convolutions, demonstrating that they can significantly reduce computational complexity without sacrificing accuracy. Notably, the DWHT requires no floating-point multiplications—only additions and subtractions—and its fast algorithm reduces complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(n \log n)$. Our results show that a DWHT-based MobileNet-V1 model achieved a 1.49% accuracy increase on CIFAR-100 while reducing parameters by 79.1% and FLOPs by 48.4%.
Introduction
In the pursuit of efficient, mobile-friendly deep learning, architectures like MobileNet and ShuffleNet popularized depthwise separable convolutions. This technique splits standard convolutions into spatial filters (depthwise) and channel mixers (pointwise). However, in modern optimized networks, the pointwise convolution (PC) has become a new bottleneck, often accounting for the vast majority of a model's parameters and floating-point operations (FLOPs).
This research investigates whether these channel-mixing weights must be learned or if they can be replaced by fixed, non-parametric mathematical transforms. By introducing Conventional Transform Pointwise Convolutions (CTPC), we show that fixed transforms can effectively capture cross-channel correlations with zero learnable parameters.
Methodology
The proposed CTPC replaces standard $1 \times 1$ convolutions with either the Discrete Cosine Transform (DCT) or the Discrete Walsh-Hadamard Transform (DWHT).
Computational Efficiency
The primary advantage of these transforms lies in their efficiency. The DWHT is particularly notable because its basis consists entirely of $+1$ and $-1$, rendering it entirely multiplication-free. Furthermore, both transforms utilize a "butterfly" structure (similar to the Fast Fourier Transform) to reduce computational complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$, where $N$ is the number of channels.
Architectural Integration
The authors found that the placement of these transforms is critical. They observed that applying a ReLU activation immediately after a CTPC layer is detrimental, as these transforms distribute information across both positive and negative values; ReLU's clipping effectively discards half of the signal. Additionally, the surrounding learnable layers, such as $3 \times 3$ depthwise convolutions, appear to "compensate" for the fixed transform by becoming more active.
Experimental Results
The CTPC approach was evaluated on the CIFAR-100 dataset using MobileNet-V1 and ShuffleNet-V2 as baselines. The experiments revealed a strong sensitivity to layer depth.
As shown in the figures above, replacing "high-level" (deeper) layers with CTPC yields the best results. In these layers, channel counts are highest—maximizing the impact of the $\mathcal{O}(N \log N)$ speedup—and features are more abstract. Conversely, replacing "low-level" layers (early in the network) leads to a significant drop in accuracy, suggesting that early feature extraction still requires the flexibility of learnable weights.
Key performance metrics for the DWHT-based MobileNet-V1 include:
- Accuracy: +1.49% Top-1 increase.
- Parameters: 79.1% reduction.
- FLOPs: 48.4% reduction.
Conclusion and Future Work
This work demonstrates that non-parametric transforms can replace learnable pointwise convolutions to create extremely efficient networks. While the results on CIFAR-100 are compelling, future research is needed to validate these findings on larger datasets like ImageNet and more complex tasks like object detection. Additionally, while theoretically multiplication-free, realizing the full hardware speedup of DWHT will require custom kernel implementations optimized for addition-only operations rather than standard matrix multiplication frameworks.
Summary updated