Abstract
Deep neural networks have demonstrated state-of-the-art performance for feature-based image matching through the advent of new large and diverse datasets. However, there has been little work on evaluating the computational cost, model size, and matching accuracy tradeoffs for these models. This paper explicitly addresses these practical metrics by considering the state-of-the-art HardNet model. We observe a significant redundancy in the learned weights, which we exploit through the use of depthwise separable layers and an efficient Tucker decomposition. We demonstrate that a combination of these methods is very effective, but still sacrifices the top-end accuracy. To resolve this, we propose the Convolution-Depthwise-Pointwise (CDP) layer, which provides a means of interpolating between the standard and depthwise separable convolutions. With this proposed layer, we can achieve an 8 times reduction in the number of parameters on the HardNet model, 13 times reduction in the computational complexity, while sacrificing less than 1% on the overall accuracy across the HPatches benchmarks. To further demonstrate the generalisation of this approach, we apply it to the state-of-the-art SuperPoint model, where we can significantly reduce the number of parameters and floating-point operations, with minimal degradation in the matching accuracy.
Introduction
Local feature descriptors are the fundamental building blocks for modern computer vision applications, including augmented reality (AR), robotics, and 3D reconstruction. While deep learning has significantly improved the accuracy of these descriptors, state-of-the-art models like HardNet are often too computationally expensive and memory-intensive for real-time deployment on mobile devices. Standard compression techniques, such as depthwise-separable convolutions, often result in a significant drop in "top-end" accuracy for these precise matching tasks. This research addresses the need for a middle ground: a compression method that drastically reduces a model's footprint while preserving the fine-grained details necessary for high-quality image matching.
The Convolution-Depthwise-Pointwise (CDP) Layer
The primary innovation of this work is the Convolution-Depthwise-Pointwise (CDP) layer. This hybrid architecture is based on the observation that in pre-trained descriptor models, only a small subset of input channels exhibits high variance in their weights, while the remainder are relatively redundant.
As illustrated in the architecture diagram, the CDP layer operates through three main stages:
- Partitioning: The input features are split into two groups based on a tunable "offset" parameter ($\alpha$).
- Hybrid Processing: The first $\alpha$ channels undergo a standard, dense convolution to preserve complex spatial and channel-wise relationships. The remaining channels are processed using a computationally efficient depthwise convolution.
- Aggregation: The outputs are concatenated and fused using a pointwise ($1 \times 1$) convolution to maintain the original output dimensions.
Analysis of Weight Redundancy
The motivation for the CDP layer stems from an analysis of weight variance in standard convolutional layers. By visualizing weight slices, a clear "cut-off" point is observed where weight variance drops significantly. This suggests that dense connectivity is not required for every channel to maintain high performance.
By keeping a small "dense" path, the CDP layer ensures that critical features are processed with full complexity, preventing the accuracy "cliff" often seen in aggressive compression methods like MobileNet's pure depthwise-separable approach.
Experimental Results
The CDP layer was evaluated on the HPatches benchmark using both HardNet and SuperPoint architectures. The results demonstrate significant efficiency gains with negligible accuracy loss:
- HardNet Compression: Achieved an 8× reduction in parameters and a 13× reduction in FLOPs (Floating Point Operations) with less than 1% loss in mAP (mean Average Precision).
- SuperPoint Generalization: Achieved a 3.2× reduction in parameters and a 1.9× reduction in FLOPs.
- Mobile Performance: The compressed models operate within the 10-150 MFLOPs range, enabling real-time performance on modern mobile hardware.
Conclusion
The CDP layer provides a direct, drop-in replacement for standard convolutional layers, offering a flexible way to trade off computational cost and accuracy. While the current approach requires manual selection of the offset parameter ($\alpha$), future work will explore automated methods to determine optimal partitioning during training. This logic of "dense for some, depthwise for others" holds potential for various high-precision computer vision tasks beyond local descriptors.
Summary updated