Semi-Supervised Multimodal Deep Learning for RGB-D Object Recognition

Yanhua Cheng, Xin Zhao, Rui Cai, Zhiwei Li, Kaiqi Huang, Yong RuiIJCAI 2016 · OpenReview

Abstract

This paper studies the problem of RGB-D object recognition. Inspired by the great success of deep convolutional neural networks (DCNN) in AI, researchers have tried to apply it to improve the performance of RGB-D object recognition. However, DCNN always requires a large-scale annotated dataset to supervise its training. Manually labeling such a large RGB-D dataset is expensive and time consuming, which prevents DCNN from quickly promoting this research area. To address this problem, we propose a semi-supervised multimodal deep learning framework to train DCNN effectively based on very limited labeled data and massive unlabeled data. The core of our framework is a novel diversity preserving co-training algorithm, which can successfully guide DCNN to learn from the unlabeled RGB-D data by making full use of the complementary cues of the RGB and depth data in object representation. Experiments on the benchmark RGB-D dataset demonstrate that, with only 5% labeled training data, our approach achieves competitive performance for object recognition compared with those state-of-the-art results reported by fully-supervised methods.

Introduction

Deep Convolutional Neural Networks (DCNNs) have revolutionized computer vision, but they are notoriously data-hungry. For RGB-D object recognition—which utilizes both standard color images and depth information—the primary bottleneck is the scarcity of massive, high-quality annotated datasets. Manually labeling thousands of 3D-synced images is both expensive and time-consuming.

While unlabeled RGB-D data is plentiful, standard semi-supervised learning often fails because models tend to become "over-confident" in a narrow subset of data. For example, a model might only learn to recognize "red apples," eventually ignoring green ones because they were absent from the initial small labeled set. This paper introduces a semi-supervised framework that achieves state-of-the-art performance using only 5% labeled data by leveraging a "diversity-preserving" co-training algorithm.

Methods

The proposed multimodal framework utilizes two DCNNs (one for RGB, one for Depth) and operates in three distinct phases:

  1. Initialization via Reconstruction: Instead of relying solely on generic pre-training (like ImageNet), the authors build reconstruction networks. These act as autoencoders, learning to compress and then reconstruct original images. This allows the models to learn meaningful features from all available data (labeled and unlabeled) before classification training begins.
  2. Diversity-Preserving Co-Training: In standard co-training, the RGB model labels unlabeled data for the Depth model, and vice versa. To prevent models from focusing only on easy, similar examples, the authors use Convex Clustering to identify "latent attributes" (sub-types) within a category. The system ensures that newly labeled samples added in each round are spread across these diverse attributes.
  3. Multitask Learning: The DCNNs are trained to predict both the object category and the latent attribute simultaneously. Finally, a fusion layer combines features from both modalities for the final prediction.
Architecture of the proposed semi-supervised multimodal framework.

Diversity-Preserving Co-Training

The "secret sauce" of this framework is the diversity-preserving component. Most semi-supervised methods suffer from "concept drift" or bias. By forcing the model to pick confident samples that also represent diverse intra-class attributes, the authors prevent the model from narrowing its focus too much. For instance, as shown in the clustering visualization, the model identifies different types of mugs (with handles, without handles, different shapes) to ensure the training set remains representative of the entire category.

Illustration of convex clustering identifying latent attributes for diversity preservation.

Experimental Results

The researchers evaluated their approach on the Washington RGB-D dataset, a standard benchmark containing 300 household objects across 51 categories. To test the semi-supervised aspect, they randomly labeled only 5% of the training data (approx. 1,750 images) and left the remaining 95% (approx. 33,250 images) unlabeled.

  • High Efficiency: With only 5% labeled data, the proposed method achieved an accuracy of 89.2%.
  • Competitive Performance: This result is remarkably close to state-of-the-art fully supervised methods, which typically hover around the 90-92% range using 100% of the labels.
  • Modality Boost: The combined (RGB+D) accuracy was significantly higher than either modality alone (82.6% for Depth, 85.5% for RGB), validating the multimodal approach and the power of complementarity.

Conclusion

This research demonstrates that RGB and Depth are ideal for co-training because they provide independent, complementary views: RGB captures texture and color, while Depth captures geometry and shape. By maintaining intra-class variety through diversity-preserving clustering, the framework avoids the "echo chamber" effect common in semi-supervised learning. Future work may involve scaling this approach to even larger "in-the-wild" datasets, where the abundance of unlabeled data could potentially allow semi-supervised models to surpass their fully supervised counterparts.

Summary updated

Abstract

This paper studies the problem of RGB-D object recognition. Inspired by the great success of deep convolutional neural networks (DCNN) in AI, researchers have tried to apply it to improve the performance of RGB-D object recognition. However, DCNN always requires a large-scale annotated dataset to supervise its training. Manually labeling such a large RGB-D dataset is expensive and time consuming, which prevents DCNN from quickly promoting this research area. To address this problem, we propose a semi-supervised multimodal deep learning framework to train DCNN effectively based on very limited labeled data and massive unlabeled data. The core of our framework is a novel diversity preserving co-training algorithm, which can successfully guide DCNN to learn from the unlabeled RGB-D data by making full use of the complementary cues of the RGB and depth data in object representation. Experiments on the benchmark RGB-D dataset demonstrate that, with only 5% labeled training data, our approach achieves competitive performance for object recognition compared with those state-of-the-art results reported by fully-supervised methods.
BibTeX
@inproceedings{cheng2016semi,
  author = {Yanhua Cheng and Xin Zhao and Rui Cai and Zhiwei Li and Kaiqi Huang and Yong Rui},
  title = {{Semi-Supervised Multimodal Deep Learning for RGB-D Object Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=Hk-kW4MdbS},
}
Figure