Query Adaptive Similarity Measure for RGB-D Object Recognition

Yanhua Cheng, Rui Cai, Chi Zhang, Zhiwei Li, Xin Zhao, Kaiqi Huang, Yong RuiICCV 2015 · OpenReview

Abstract

This paper studies the problem of improving the top-1 accuracy of RGB-D object recognition. Despite the impressive top-5 accuracies achieved by existing methods, their top-1 accuracies are often unsatisfactory. The reasons are two-fold: (1) existing similarity measures are sensitive to object pose and scale changes, as well as intra-class variations, and (2) effectively fusing RGB and depth cues remains an open problem. To address these issues, this paper first proposes a new similarity measure based on dense matching, through which objects in comparison are warped and aligned to better tolerate variations. Regarding RGB and depth fusion, we argue that a constant "golden weight" does not exist; the two modalities have varying contributions across different categories. To capture this dynamic characteristic, a group of matchers equipped with various fusion weights is constructed to explore responses under different configurations. All response scores are finally merged using a learning-to-combination approach, providing strong generalization. The proposed approach achieves 92.7% top-1 test accuracy on the Washington RGB-D object dataset, a 5.1% improvement over the state-of-the-art.

Introduction

While RGB-D cameras like the Kinect have revolutionized object recognition, a persistent performance gap remains. Current state-of-the-art models often achieve excellent top-5 accuracy (near 98%) but struggle with top-1 accuracy, which frequently hovers around 85%. This research identifies two primary culprits for this discrepancy: geometric sensitivity and the "fusion dilemma."

Standard feature extraction is often brittle when faced with changes in an object's pose, scale, or intra-class variations (e.g., distinguishing between different models of a pitcher). Furthermore, static fusion strategies fail to account for the fact that the relative importance of color and depth is not constant. For some objects, color is the primary identifier, while for others, geometric shape (depth) is the deciding factor. This paper introduces a query-adaptive framework to resolve these challenges.

Methodology

Dense Matching and Warping

Instead of relying on static feature vectors, the proposed method utilizes dense matching (inspired by SIFT flow) to perform a "warping" step. When comparing a query image to a reference database image, the reference is warped to align its semantically corresponding parts with the query. For instance, if a query pitcher is tilted differently than a reference pitcher, the algorithm aligns the handles and spouts before calculating similarity. This makes the resulting similarity measure adaptive to the specific geometric configuration of the query.

The dense matching process aligns a reference object to the query to calculate similarity.

Learning-to-Combination

To address the fusion dilemma, the authors propose that the contribution of RGB and depth should be dynamic. They construct a group of dense matchers, each assigned a different fusion weight ($\theta$) ranging from purely depth-based to purely color-based. Using a Ranking SVM, the system learns a global weight vector that combines the scores from these different matchers. This allows the model to observe how similarity scores fluctuate across different fusion configurations, effectively learning which modality is more reliable for specific types of object categories and potential confusions.

Experimental Results

The approach was evaluated on two major benchmarks: the Washington RGB-D Object Dataset (300 objects in 51 categories) and the 2D3D Object Dataset (156 objects in 14 categories).

On the Washington dataset, the method achieved a 92.7% top-1 accuracy, outperforming the previous state-of-the-art CNN-RNN model by 5.1%. The method also demonstrated superior stability, showing the smallest standard deviation across different test trials.

Classification accuracy across 51 categories on the Washington RGB-D dataset.

Qualitative results confirm the model's robustness. For example, where color-heavy models might mistake a black pitcher for a black cap, the proposed method correctly identifies the object by prioritizing shape alignment and geometric residuals over misleading color cues.

Conclusion and Future Work

By introducing query-adaptive alignment and dynamic modality fusion, this work significantly closes the gap in top-1 recognition accuracy. However, the primary limitation remains computational cost; calculating dense matching for every pair currently takes approximately 9 seconds on a single-threaded CPU. Future research will focus on optimizing these algorithms through GPU acceleration and extending the similarity measure to the more complex task of RGB-D object detection in cluttered scenes.

Summary updated

Abstract

This paper studies the problem of improving the top-1 accuracy of RGB-D object recognition. Despite of the impressive top-5 accuracies achieved by existing methods, their top-1 accuracies are not very satisfactory. The reasons are in two-fold: (1) existing similarity measures are sensitive to object pose and scale changes, as well as intra-class variations, and (2) effectively fusing RGB and depth cues is still an open problem. To address these problems, this paper first proposes a new similarity measure based on dense matching, through which objects in comparison are warped and aligned, to better tolerate variations. Towards RGB and depth fusion, we argue that a constant and golden weight doesn't exist. The two modalities have varying contributions when comparing objects from different categories. To capture such a dynamic characteristic, a group of matchers equipped with various fusion weights is constructed, to explore the responses of dense matching under different fusion configurations. All the response scores are finally merged following a learning-to-combination way, which provides quite good generalization ability in practice. The proposed approach win the best results on several public benchmarks, e.g., achieves 92.7% top-1 test accuracy on the Washington RGB-D object dataset, with a 5.1% improvement over the state-of-the-art.
BibTeX
@inproceedings{cheng2015query,
  author = {Yanhua Cheng and Rui Cai and Chi Zhang and Zhiwei Li and Xin Zhao and Kaiqi Huang and Yong Rui},
  title = {{Query Adaptive Similarity Measure for RGB-D Object Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=Bkb3YlGuZS},
}
Figure