Abstract
Deep neural networks have revolutionized feature-based image matching, but their high computational cost and model size hinder deployment on mobile devices. This paper addresses these challenges by analyzing the state-of-the-art HardNet model and identifying significant redundancy in its learned weights. We propose the Convolution-Depthwise-Pointwise (CDP) layer, a novel architectural block that interpolates between standard and depthwise separable convolutions. By strategically partitioning input channels, the CDP layer achieves an 8x reduction in parameters and a 13x reduction in computational complexity for HardNet, with less than 1% loss in accuracy on HPatches benchmarks. We further demonstrate the generalizability of this approach on the SuperPoint model, significantly reducing its footprint with minimal performance degradation.
Introduction
Local feature descriptors are critical for computer vision applications like Augmented Reality (AR), drone navigation, and 3D reconstruction. However, modern deep-learning models like HardNet are often too resource-intensive for mobile hardware, requiring massive memory and trillions of floating-point operations (FLOPs). Existing compression methods, such as depthwise-separable convolutions or Tucker decomposition, frequently sacrifice too much accuracy for speed. This work introduces the CDP layer to provide a "Goldilocks" solution: significant compression without sacrificing top-tier performance.
The CDP Layer
The core contribution is the Convolution-Depthwise-Pointwise (CDP) layer. This approach is based on the observation that in pre-trained models, only a small subset of input channels exhibits high variance and complex structures, while the rest are relatively redundant.
Instead of applying the same operation to every channel, the CDP layer splits the input tensor $\mathcal{X}$ into two paths:
- Dense Path: A small subset of channels ($\alpha$) undergoes a standard, "dense" convolution to preserve high-end representational power.
- Efficient Path: The remaining channels undergo a "depthwise" convolution, which is computationally cheap but less expressive.
- Aggregation: The outputs from both paths are concatenated and fused using a pointwise ($1 \times 1$) convolution.
To further reduce the parameter count, the authors use Variational Bayesian Matrix Factorization (VBMF) to find an optimal low-rank approximation for the $1 \times 1$ kernels, creating a Pointwise Linear Bottleneck.
Insights: Structured Redundancy
The motivation for the CDP layer comes from visualizing weight slices in standard descriptor models. The researchers observed that many channels are "lazy"—they do not contribute much unique information.
By identifying a "cut-off point" (offset), the network can relegate redundant channels to cheaper operations. This hybridization allows the model to retain its "intelligence" via the dense core while shedding significant weight.
Key Results
The proposed methods were evaluated on industry-standard benchmarks including HPatches, MS-COCO, and the Brown Dataset. The results demonstrate that efficiency does not have to come at the cost of accuracy:
- HardNet Compression: Achieved an 8x reduction in parameters and a 13x reduction in FLOPs, with a mean Average Precision (mAP) drop of less than 1%.
- SuperPoint Generalization: Yielded a 3.2x reduction in parameters and a 1.9x reduction in FLOPs with virtually no degradation in matching scores.
- Mobile Viability: These compressed models can run in real-time on resource-constrained devices, a capability previously reserved for simpler, handcrafted descriptors like SIFT.
Conclusion
The CDP layer provides a flexible means of interpolating between dense and efficient convolutions. While highly effective, future work could involve making the channel split parameter ($\alpha$) learnable, allowing the network to automatically optimize its own compression-accuracy tradeoff during training. Additionally, hardware-specific optimizations for mobile GPUs/NPUs would further enhance on-device latency.
Summary updated