Feature Fusion for Image Retrieval With Adaptive Bitrate Allocation and Hard Negative Mining

Chuang Zhu, Huihui Dong, Shanghang ZhangIEEE Access 2019 · OpenReview

Abstract

By combining Convolutional Neural Network (CNN) descriptors and Compact Descriptors for Visual Search (CDVS), visual search performance can be significantly boosted. However, redundancies still exist in standard CDVS representations, and hard negative mining—a critical step in training CNN embeddings—is often inaccurate in early stages. This paper proposes a high-performance image retrieval scheme based on descriptor fusion. Specifically, we introduce a more compact CDVS database building scheme through adaptive bitrate allocation to reduce redundancy. Furthermore, we propose a highly accurate CDVS-guided hard negative mining strategy where CDVS descriptors and CNN embeddings are adaptively weighted to improve training decisions. Finally, retrieval results are refined through CDVS local descriptor matching. Experimental results demonstrate that this method remarkably outperforms recent hybrid methods and established anchors.

Introduction

Image retrieval—identifying images of the same object or scene from a massive database—is a cornerstone of mobile augmented reality and visual search. Current state-of-the-art approaches typically rely on either hand-crafted descriptors like the MPEG-standardized Compact Descriptors for Visual Search (CDVS) or deep learning-based CNN embeddings. While CDVS offers standardized compactness and geometric robustness, CNNs excel at capturing high-level semantic information.

However, existing hybrid systems face two major inefficiencies. First, standard CDVS typically allocates a fixed bitrate (e.g., 4KB) to every image, leading to "bitrate waste" where distinctive images are over-described and complex images are under-described. Second, training CNNs for retrieval requires "hard negative mining"—selecting challenging non-matching examples. Early in training, inaccurate CNN embeddings often lead to poor selection choices, creating a "training noise" problem that hinders model convergence.

Proposed Methodology

The proposed framework introduces a three-pronged approach to fuse these features effectively while addressing the aforementioned inefficiencies.

1. Adaptive Bitrate Allocation (BA)

The authors introduce the concept of a Retrieval Hardness Degree (RHD) for each image. By calculating the distance between an image and its neighbors in feature space, the system identifies "hard" images that require more descriptive power to be distinguished. Bits are dynamically redistributed: images with high RHD receive a larger portion of the local descriptor budget, while "easy" images are stored more compactly. This optimization improves the overall discriminative power of the database without increasing the total storage footprint.

The standard CDVS extraction pipeline, which the proposed method optimizes through adaptive bitrate allocation.

2. CDVS-Guided Hard Negative Mining (AHM)

To overcome the "cold start" problem in CNN training, the framework utilizes reliable, hand-crafted CDVS global descriptors to guide the selection of hard negatives. An adaptive weighting scheme is employed: early in the training process, the system relies more heavily on CDVS descriptors. As the CNN's loss decreases and its embeddings become more discriminative, the weight shifts toward the CNN's own internal representations. This "mentor" approach ensures the network is consistently challenged by high-quality negative examples.

3. Retrieval Refinement (RR)

The final stage involves a reranking process. After an initial search using fused global features, the system performs geometric consistency checks using CDVS local descriptors (SIFT-like features). This step prunes irrelevant images that may have high global similarity but lack fine-grained local matches, ensuring the top results are highly accurate.

Experimental Results

The method was evaluated on industry-standard benchmarks, including Oxford5k, Paris6k, and the MPEG CDVS dataset.

  • Accuracy Gains: The full pipeline (AHM+G+BA+RR) achieved significant mAP (Mean Average Precision) improvements: +1.4% on Oxford5k, +3.1% on Paris6k, and +2.5% on the CDVS dataset compared to previous state-of-the-art hybrid schemes.
  • Top-1 Performance: On the CDVS dataset, the system reached a near-perfect 94.6% Top-1 accuracy.
  • Storage Efficiency: The adaptive bitrate allocation (BA) alone provided a 0.8% mAP boost simply by redistributing storage resources, proving that intelligent data allocation is as critical as the descriptors themselves.

Conclusion and Future Work

This research demonstrates that hand-crafted features like CDVS remain vital in the deep learning era, serving as both a stable training guide and a robust geometric verifier. While the current framework significantly improves retrieval performance, it is currently limited by fixed-length CNN embeddings. Future research will explore "deep descriptors" with variable lengths and extend these strategies to video-based retrieval (CDVA).

Summary updated

Abstract

By combining Convolutional Neural Network (CNN) descriptor and Compact Descriptors for Visual Search (CDVS), the visual search performance can be boosted. However, some redundancies still exist in the CDVS representation and the hard negative mining is not very accurate when training CNN embeddings. In this paper, we propose a high performance image retrieval scheme based on descriptor fusion. In detail, we first propose a more compact CDVS descriptor database building scheme through bitrate allocation, which can reduce information redundancy and boost image retrieval performance. We then propose a highly accurate CDVS-guided hard negative mining scheme when training CNN embeddings. In the hard negative selection, the CDVS descriptor and the CNN embedding are adaptively weighted together to achieve more precise decisions. Finally, the retrieval result is further refined through CDVS local descriptor matching by removing the irrelevant targets from the top positions. Extensive experimental results show that the proposed method outperforms the recent hybrid method and several other anchors remarkably, and produces better visual search performance. Codes and some models are available at https://github.com/WendyDong/ImageRetrieval_DF_CDVS.
BibTeX
@inproceedings{zhu2019feature,
  author = {Chuang Zhu and Huihui Dong and Shanghang Zhang},
  title = {{Feature Fusion for Image Retrieval With Adaptive Bitrate Allocation and Hard Negative Mining}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=oVXRug2HJyu},
}
Figure