Compression of descriptor models for mobile applications

Roy Miles, Krystian Mikolajczyk

Abstract

Deep neural networks have revolutionized feature-based image matching, but their high computational cost and model size hinder deployment on mobile devices. This paper addresses these challenges by analyzing the state-of-the-art HardNet model and identifying significant redundancy in its learned weights. We propose the Convolution-Depthwise-Pointwise (CDP) layer, a novel architectural block that interpolates between standard and depthwise separable convolutions. By strategically partitioning input channels, the CDP layer achieves an 8x reduction in parameters and a 13x reduction in computational complexity for HardNet, with less than 1% loss in accuracy on HPatches benchmarks. We further demonstrate the generalizability of this approach on the SuperPoint model, significantly reducing its footprint with minimal performance degradation.

Introduction

Local feature descriptors are critical for computer vision applications like Augmented Reality (AR), drone navigation, and 3D reconstruction. However, modern deep-learning models like HardNet are often too resource-intensive for mobile hardware, requiring massive memory and trillions of floating-point operations (FLOPs). Existing compression methods, such as depthwise-separable convolutions or Tucker decomposition, frequently sacrifice too much accuracy for speed. This work introduces the CDP layer to provide a "Goldilocks" solution: significant compression without sacrificing top-tier performance.

The CDP Layer

The core contribution is the Convolution-Depthwise-Pointwise (CDP) layer. This approach is based on the observation that in pre-trained models, only a small subset of input channels exhibits high variance and complex structures, while the rest are relatively redundant.

The CDP layer architecture splits input channels between a dense convolutional path and an efficient depthwise path before fusing them with a pointwise convolution.

Instead of applying the same operation to every channel, the CDP layer splits the input tensor $\mathcal{X}$ into two paths:

  1. Dense Path: A small subset of channels ($\alpha$) undergoes a standard, "dense" convolution to preserve high-end representational power.
  2. Efficient Path: The remaining channels undergo a "depthwise" convolution, which is computationally cheap but less expressive.
  3. Aggregation: The outputs from both paths are concatenated and fused using a pointwise ($1 \times 1$) convolution.

To further reduce the parameter count, the authors use Variational Bayesian Matrix Factorization (VBMF) to find an optimal low-rank approximation for the $1 \times 1$ kernels, creating a Pointwise Linear Bottleneck.

Insights: Structured Redundancy

The motivation for the CDP layer comes from visualizing weight slices in standard descriptor models. The researchers observed that many channels are "lazy"—they do not contribute much unique information.

Visualization of weight slices showing structured redundancy in the learned kernels of the descriptor model.

By identifying a "cut-off point" (offset), the network can relegate redundant channels to cheaper operations. This hybridization allows the model to retain its "intelligence" via the dense core while shedding significant weight.

Key Results

The proposed methods were evaluated on industry-standard benchmarks including HPatches, MS-COCO, and the Brown Dataset. The results demonstrate that efficiency does not have to come at the cost of accuracy:

  • HardNet Compression: Achieved an 8x reduction in parameters and a 13x reduction in FLOPs, with a mean Average Precision (mAP) drop of less than 1%.
  • SuperPoint Generalization: Yielded a 3.2x reduction in parameters and a 1.9x reduction in FLOPs with virtually no degradation in matching scores.
  • Mobile Viability: These compressed models can run in real-time on resource-constrained devices, a capability previously reserved for simpler, handcrafted descriptors like SIFT.

Conclusion

The CDP layer provides a flexible means of interpolating between dense and efficient convolutions. While highly effective, future work could involve making the channel split parameter ($\alpha$) learnable, allowing the network to automatically optimize its own compression-accuracy tradeoff during training. Additionally, hardware-specific optimizations for mobile GPUs/NPUs would further enhance on-device latency.

Summary updated

Abstract

Deep neural networks have demonstrated state-of-the-art performance for feature-based image matching through the advent of new large and diverse datasets. However, there has been little work on evaluating the computational cost, model size, and matching accuracy tradeoffs for these models. This paper explicitly addresses these practical metrics by considering the state-of-the-art HardNet model. We observe a significant redundancy in the learned weights, which we exploit through the use of depthwise separable layers and an efficient Tucker decomposition. We demonstrate that a combination of these methods is very effective, but still sacrifices the top-end accuracy. To resolve this, we propose the Convolution-Depthwise-Pointwise(CDP) layer, which provides a means of interpolating between the standard and depthwise separable convolutions. With this proposed layer, we can achieve an 8 times reduction in the number of parameters on the HardNet model, 13 times reduction in the computational complexity, while sacrificing less than 1% on the overall accuracy across theHPatchesbenchmarks. To further demonstrate the generalisation of this approach, we apply it to the state-of-the-art SuperPoint model, where we can significantly reduce the number of parameters and floating-point operations, with minimal degradation in the matching accuracy.
BibTeX
@misc{miles2020compression,
  author = {Roy Miles and Krystian Mikolajczyk},
  title = {{Compression of descriptor models for mobile applications}},
  year = {2020},
  eprint = {2001.03102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.03102},
}
Figure