Compressing Local Descriptor Models for Mobile Applications

Roy Miles, Krystian MikolajczykICASSP 2021 · OpenReview

TL;DR

This paper tackles the challenge of deploying powerful deep learning-based local feature descriptors on resource-constrained mobile devices. They introduce a novel "Convolution-Depthwise-Pointwise" (CDP) layer that efficiently compresses these models by exploiting the inherent structure in convolutional weights, achieving significant reductions in size and computation with minimal accuracy loss.

What Problem Does This Paper Address?

Deep learning has revolutionized image matching and local feature description, leading to highly accurate models. However, these state-of-the-art models are computationally expensive and large, making them impractical for deployment on mobile devices with limited memory and processing power. While general CNN compression techniques exist (like those used in MobileNets), they haven't been effectively applied to the specific needs of local descriptor models, which require a delicate balance between accuracy and efficiency. This paper aims to bridge this gap by developing a method to compress these descriptor models without sacrificing their critical matching performance.

Methods

The core contribution is the Convolution-Depthwise-Pointwise (CDP) layer. This novel layer is inspired by the observation that in convolutional weights, a significant portion of the variance is concentrated in a subset of input channels. The CDP layer exploits this by partitioning the convolutional weights:

  • Low-rank decomposition: A portion of the input channels are processed using a standard, dense convolution.
  • Efficient depthwise convolution: The remaining channels are processed using a more efficient depthwise convolution.
  • Pointwise aggregation: The outputs from both branches are then concatenated and aggregated using a pointwise convolution.

This approach allows for a controlled trade-off between dense connectivity (preserving accuracy) and efficient computation. The authors also explore combining this with Tucker decomposition and pointwise linear bottlenecks for further compression.

Datasets

  • Training: The HardNet++ model weights were trained on the Liberty, Yosemite, and Notredame datasets. For their proposed methods, training was done solely on the Liberty dataset from random initialization.
  • Evaluation: The HPatches benchmark [31] was used for evaluating descriptor performance. This dataset consists of local patches from 116 different scenes, featuring variations in illumination and viewpoints.

Key Insights and Ideas

  • Exploiting Weight Structure: The paper's key insight is that convolutional weights in descriptor models exhibit a natural structure where some input channels are more "important" (have higher variance) than others. This structure can be leveraged for compression.
  • Controlled Dense Connectivity: The CDP layer's ability to retain a subset of dense connections is crucial for maintaining high-end accuracy, a limitation often seen in purely depthwise separable or Tucker decomposed layers.
  • Generalizability: The proposed CDP layer is demonstrated to be applicable not just to HardNet but also to other models like SuperPoint, showcasing its broad utility.
  • Mobile-Friendly Performance: The compression achieved pushes models into a performance range suitable for real-time applications on mobile phones (10-150 MFLOPs).

Key Results

  • HardNet Compression: The CDP layer achieved an 8x reduction in parameters and a 13x reduction in computational complexity on the HardNet model, with less than 1% loss in overall accuracy on the HPatches benchmark.
  • SuperPoint Compression: For the SuperPoint model, CDP layers resulted in a ~3.2x reduction in parameters and a ~1.9x reduction in FLOPS with minimal loss in matching score.
  • Accuracy vs. Compression Trade-off: The results consistently show that as long as some dense channel connectivity is maintained (offset $\alpha \ge 5$), high accuracy is preserved, unlike methods that rely solely on depthwise separable or Tucker decomposition.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method significantly reduces computational cost and model size, there's still a trade-off, albeit a very favorable one, between compression and accuracy. The exact optimal partitioning (offset $\alpha$) might require careful tuning for different layers and models.
  • Future Work: The authors suggest that their approach can be applied to a wider range of descriptor models and potentially other CNN architectures for mobile applications. Further exploration of optimal partitioning strategies and combinations with other compression techniques could yield even greater improvements.

Summary updated

Abstract

Feature-based image matching has been significantly improved through the use of deep learning and new large datasets. However, there has been little work addressing the computational cost, model size, and matching accuracy tradeoffs for the state of the art models. In this paper, we consider these practical aspects and improve the state-of-the-art HardNet model through the use of depthwise separable layers and an efficient tensor decomposition. We propose the Convolution-Depthwise-Pointwise (CDP) layer, which partitions the weights into a low and full rank decomposition to exploit the naturally emergent structure in the convolutional weights. We can achieve an 8× reduction in the number of parameters on the HardNet model, 13× reduction in the computational complexity, while sacrificing less than 1% on the overall accuracy across the HPatches benchmarks. To further demonstrate the generalisation of this approach, we apply it to other state-of-the-art descriptor models, where we are able to a significant performance improvement.
BibTeX
@inproceedings{miles2021compressing,
  author = {Roy Miles and Krystian Mikolajczyk},
  title = {{Compressing Local Descriptor Models for Mobile Applications}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=_8elaJ7rsUm},
}
Figure