Introduction
Modern visual recognition systems increasingly rely on multi-sensory hardware, such as RGB-D (color + depth) or RGB-IR (color + infrared) cameras. While multi-modal data is highly beneficial during training, real-world deployment often faces the "missing-view-in-test-data" problem. Auxiliary sensors (like depth or IR) are frequently prone to hardware failure, bandwidth limitations, or environmental noise, meaning they may be completely absent during testing. This research introduces a method to leverage these "ghost" modalities during training to build a more robust classifier that requires only the primary sensor at test time.
Methodology: Discriminative Canonical Correlation Analysis (DCCA)
To address the missing-view problem, the authors propose Discriminative Canonical Correlation Analysis (DCCA). This approach maps data from different modalities into a shared semantic latent space where classification is performed.
DCCA improves upon standard Canonical Correlation Analysis (CCA) in two critical ways:
- Labels as a "View": DCCA treats training class labels as an additional sensory modality, encoded via a k-Nearest Neighbors (k-NN) graph. This forces the latent space to be highly discriminative rather than just correlated.
- Simultaneous Optimization: Unlike many generalized CCA algorithms that calculate projection vectors iteratively, DCCA optimizes all projection directions at once. This ensures numerical stability and is backed by a proof of monotonic convergence.
During training, the main view, auxiliary view, and label view are projected into this common space to train a standard SVM. At test time, the lone main view is projected into this pre-calculated space for prediction.
Experimental Results
The framework was evaluated on four diverse benchmarks: NYU Depth V1, the RGBD Object Dataset, the Multi-Spectral Scene Dataset, and the Binghamton University 3D Facial Expression Dataset.
Key quantitative insights include:
- Superior Accuracy: On the RGBD Object Dataset, DCCA achieved an average accuracy of 86.6%, significantly outperforming the baseline SVM (81.3%) and other CCA variants. Using advanced HMP features, the accuracy reached as high as 94.62%.
- Effective Knowledge Transfer: Training a model to map the main view into a space shaped by an auxiliary view effectively "bakes" the missing sensor's geometric or spectral insights into the main view's processing pipeline.
- Synergy of Views: Ablation studies showed that the most significant performance gains occur only when both the auxiliary view and the label view are combined through DCCA's simultaneous optimization.
Conclusion and Future Work
DCCA provides a powerful solution for visual recognition when sensor availability is inconsistent between training and deployment. By constructing a discriminative latent space, the method ensures that the benefits of multi-modal training are preserved even in single-modal testing environments. While the current method relies on hand-crafted features, future work could involve integrating DCCA constraints directly into end-to-end deep learning architectures to further untangle complex, cluttered scenes.
Summary updated