Auxiliary Information Regularized Machine for Multiple Modality Feature Learning

Yang Yang, Han-Jia Ye, De-Chuan Zhan, Yuan Jiang

Introduction

In multi-modal machine learning, data sources such as text, images, and sensor data are often treated as equally informative. However, real-world scenarios frequently involve modalities of varying quality. For instance, in identity recognition, a fingerprint serves as a "strong" modality due to its high distinctiveness, whereas a facial image may be a "weak" modality, susceptible to environmental noise like lighting or resolution issues. While strong modalities are more reliable, they are often harder or more expensive to obtain.

The Auxiliary Information Regularized Machine (ARM) addresses this discrepancy by using the rich information inherent in a strong modality to supervise the feature learning process of a weak modality. This approach aims to "unlock" hidden discriminative features in noisy data by leveraging paired, often unlabeled, multi-modal information.

The ARM Framework

ARM is a semi-supervised framework that jointly optimizes two primary tasks:

  1. Feature Extraction for Weak Modality: Identifying a lower-dimensional subspace where the weak modality's features are most discriminative.
  2. Strong Modality Regularization: Training a predictor for the strong modality while ensuring its predictions remain consistent with the structure of the weak modality.

A core component of ARM is its manifold regularizer, which ensures that instances appearing similar in the weak modality's learned feature space receive similar predictions from the strong modality's classifier. To solve the non-convex objective function, the authors employ an alternating descent algorithm that iteratively updates the weak modality's projection matrix and the strong modality's predictor.

Experimental Evaluation

The researchers evaluated ARM across diverse domains, including image categorization (Nus-WIDE, MSRA-MM), webpage classification (WebKB), and biometrics (pairing faces with gait sequences). In these tests, one modality was designated as "strong" (e.g., citation links or faces) and the other as "weak" (e.g., webpage content or gait).

Key Results and Discussion

ARM demonstrated superior performance across all tested datasets. Notably, the strong modality predictor achieved high accuracy (e.g., 97.3% on the Biometric dataset). More importantly, the weak modality itself saw a significant performance boost after training with ARM, outperforming standard techniques like PCA and LDA. For example, on the MSRA dataset, ARM improved weak feature accuracy to 81.4%, compared to just 67.2% for PCA.

Convergence and Stability

The iterative optimization process is highly efficient. As shown in Figure 1, the objective function typically converges within a few iterations across various datasets, confirming the algorithm's practicality for real-world use.

Convergence of objective function and ARM accuracy across iterations for Msra, Wisconsin, Cornell, and WebKB datasets.

Furthermore, ARM exhibits robustness to hyperparameter selection. Figure 2 illustrates that the accuracy for both weak and strong modalities remains stable across a wide range of values for the regularization parameter $\lambda_2$, indicating that the model is not overly sensitive to precise parameter tuning.

Accuracy vs. regularization parameter lambda_2 for the Wisconsin dataset, showing stable performance for both modalities.

Conclusion

ARM provides a robust framework for asymmetric multi-modal learning, effectively using high-quality "teacher" modalities to improve feature extraction in noisier "student" modalities. While currently limited to two modalities, future work could extend this to integrate multiple data streams and automate the identification of modality strength.

Summary updated

Abstract

In real world applications, data are often with multiple modalities. Previous works assumed that each modality contains sufficient information for target and can be treated with equal importance. However, it is often that different modalities are of various importance in real tasks, e.g., the facial feature is weak modality and the fingerprint feature is strong modality in ID recognition. In this paper, we point out that different modalities should be treated with different strategies and propose the Auxiliary information Regularized Machine (ARM), which works by extracting the most discriminative feature subspace of weak modality while regularizing the strong modal predictor. Experiments on binary and multi-class datasets demonstrate the advantages of our proposed approach ARM.
BibTeX
@inproceedings{yang2015auxiliary,
  author = {Yang Yang and Han-Jia Ye and De-Chuan Zhan and Yuan Jiang},
  title = {{Auxiliary Information Regularized Machine for Multiple Modality Feature Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure