Highly Efficient Mobile Visual Search Algorithm

Chuang Zhu, Xiaofeng Huang, Guoqing Xiang, Huihui Dong, Jiawen SongIEICE Trans. Information and Systems 2018 · OpenReview

Abstract

Mobile Visual Search (MVS) allows users to identify objects or locations via mobile photography, but the computational demands of feature extraction often strain device resources. This paper introduces a streamlined MVS framework that significantly reduces processing time by utilizing a coarse-to-fine feature detection strategy and an enhanced reranking algorithm. Experimental results demonstrate up to a 23.5% reduction in processing time with negligible loss in retrieval accuracy.

Introduction

The industry standard for MVS is MPEG CDVS (Compact Descriptors for Visual Search). A primary bottleneck in the CDVS pipeline is the construction of a scale-space pyramid, which involves repeatedly blurring and sub-sampling images to identify features at various scales. Traditionally, this process treats all image regions with equal importance, leading to wasted computational cycles on uninformative areas. This research proposes a more efficient approach by identifying "important" regions early and focusing resources where they matter most.

Methodology

1. Low-Complexity Feature Detection

The authors propose a coarse-to-fine strategy to optimize descriptor extraction on mobile devices. Instead of full-scale processing, the algorithm first detects key points in lower-resolution "coarse" octaves. These points are used to generate an Expanded Important Area (EIA) mask.

Generation of the Expanded Important Area (EIA) mask using coarse-to-fine guidance.

As shown in the figure above, keypoints from coarse scales (a) are aggregated into a binary mask (b). When processing the high-resolution "fine" octave (c), the algorithm skips expensive Gaussian and Laplacian operations for pixels outside this mask. Furthermore, the pipeline is optimized by moving feature selection before orientation computing, discarding low-relevance points early to save power and time.

2. Enhanced Reranking (LDSS)

On the server side, the system must efficiently rank candidate images. The authors developed the Local Descriptor Similarity Score (LDSS), which integrates tf-idf (term frequency-inverse document frequency) weighting.

Proposed server-side reranking framework incorporating Local Descriptor Similarity Score (LDSS).

By treating visual features as "words," the LDSS down-weights common, non-distinctive features (e.g., sky) and up-weights rare, informative ones. The final ranking is determined by a weighted combination of global and local similarity scores, with the balance ($\lambda$) optimized through a training phase to ensure accuracy even at low bitrates.

Experimental Results

The algorithm was evaluated against the CDVS reference software (TM 11.0) using standard benchmarks including the University of Kentucky (UK) dataset, INRIA Holidays, and MPEG-7 CDVS subsets (Graphics, Paintings, Buildings).

  • Efficiency: Descriptor extraction time was reduced by an average of 18.5% on the UK dataset and 23.5% on the Buildings dataset.
  • Accuracy: The reduction in Mean Average Precision (mAP) was typically less than 1%, maintaining state-of-the-art performance despite the lower complexity.
  • Reranking Performance: The LDSS-based reranking consistently outperformed standard CDVS methods and established spatial verification techniques.

Limitations

The authors noted a specific limitation with the INRIA Holidays dataset, where mAP dropped by approximately 2.1%. This is attributed to the dataset's lack of clearly "unimportant" areas, making the masking strategy less effective and slightly more prone to missing features.

Conclusion

This research demonstrates that leveraging relationships between image scales can significantly optimize mobile visual search. By gating fine-scale processing with coarse-scale "previews" and employing intelligent reranking, the proposed framework offers a high-performance, battery-friendly solution for mobile devices. Future work will explore adaptive partitioning and hybrid feature sets combining hand-crafted and deep learning descriptors.

Summary updated

BibTeX
@article{zhu2018highly,
  author = {Chuang Zhu and Xiaofeng Huang and Guoqing Xiang and Huihui Dong and Jiawen Song},
  title = {{Highly Efficient Mobile Visual Search Algorithm}},
  journal = {IEICE Trans. Information and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=JyDZCghLzls},
}
Figure