New pointwise convolution in Deep Neural Networks through Extremely Fast and Non Parametric Transforms

Joonhyun Jeong, Sung-Ho BaearXiv 2019 · OpenReview

Abstract

Conventional transforms such as the Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have long been staples of image processing but have seen limited use within neural network architectures. This paper proposes applying these transforms to pointwise convolutions, demonstrating that they can significantly reduce computational complexity without sacrificing accuracy. Notably, the DWHT requires no floating-point multiplications—only additions and subtractions—and its fast algorithm reduces complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(n \log n)$. Our results show that a DWHT-based MobileNet-V1 model achieved a 1.49% accuracy increase on CIFAR-100 while reducing parameters by 79.1% and FLOPs by 48.4%.

Introduction

In the pursuit of efficient, mobile-friendly deep learning, architectures like MobileNet and ShuffleNet popularized depthwise separable convolutions. This technique splits standard convolutions into spatial filters (depthwise) and channel mixers (pointwise). However, in modern optimized networks, the pointwise convolution (PC) has become a new bottleneck, often accounting for the vast majority of a model's parameters and floating-point operations (FLOPs).

This research investigates whether these channel-mixing weights must be learned or if they can be replaced by fixed, non-parametric mathematical transforms. By introducing Conventional Transform Pointwise Convolutions (CTPC), we show that fixed transforms can effectively capture cross-channel correlations with zero learnable parameters.

Methodology

The proposed CTPC replaces standard $1 \times 1$ convolutions with either the Discrete Cosine Transform (DCT) or the Discrete Walsh-Hadamard Transform (DWHT).

Computational Efficiency

The primary advantage of these transforms lies in their efficiency. The DWHT is particularly notable because its basis consists entirely of $+1$ and $-1$, rendering it entirely multiplication-free. Furthermore, both transforms utilize a "butterfly" structure (similar to the Fast Fourier Transform) to reduce computational complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$, where $N$ is the number of channels.

Complexity comparison showing the dramatic reduction in multiplication operations for fast transform-based PC compared to conventional PC.

Architectural Integration

The authors found that the placement of these transforms is critical. They observed that applying a ReLU activation immediately after a CTPC layer is detrimental, as these transforms distribute information across both positive and negative values; ReLU's clipping effectively discards half of the signal. Additionally, the surrounding learnable layers, such as $3 \times 3$ depthwise convolutions, appear to "compensate" for the fixed transform by becoming more active.

Experimental Results

The CTPC approach was evaluated on the CIFAR-100 dataset using MobileNet-V1 and ShuffleNet-V2 as baselines. The experiments revealed a strong sensitivity to layer depth.

Comparison of Top-1 Accuracy vs. Parameters across different layer depths, showing that high-level layer replacement (left) yields the best efficiency gains.

As shown in the figures above, replacing "high-level" (deeper) layers with CTPC yields the best results. In these layers, channel counts are highest—maximizing the impact of the $\mathcal{O}(N \log N)$ speedup—and features are more abstract. Conversely, replacing "low-level" layers (early in the network) leads to a significant drop in accuracy, suggesting that early feature extraction still requires the flexibility of learnable weights.

Key performance metrics for the DWHT-based MobileNet-V1 include:

  • Accuracy: +1.49% Top-1 increase.
  • Parameters: 79.1% reduction.
  • FLOPs: 48.4% reduction.

Conclusion and Future Work

This work demonstrates that non-parametric transforms can replace learnable pointwise convolutions to create extremely efficient networks. While the results on CIFAR-100 are compelling, future research is needed to validate these findings on larger datasets like ImageNet and more complex tasks like object detection. Additionally, while theoretically multiplication-free, realizing the full hardware speedup of DWHT will require custom kernel implementations optimized for addition-only operations rather than standard matrix multiplication frameworks.

Summary updated

Abstract

Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms to pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy performance degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from $\mathcal{O}(n^2)$ to $\mathcal{O}(n\log n)$. These nice properties construct extremely efficient networks in the number parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49\% accuracy increase with 79.1\% reduced parameters and 48.4\% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.
BibTeX
@misc{jeong2019new,
  author = {Joonhyun Jeong and Sung-Ho Bae},
  title = {{New pointwise convolution in Deep Neural Networks through Extremely Fast and Non Parametric Transforms}},
  year = {2019},
  eprint = {1906.12172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.12172},
}

Loading related content...

Figure