Latent Low-Rank Transfer Subspace Learning for Missing Modality Recognition

Zhengming Ding, Ming Shao, Yun Fu

Abstract

This paper introduces Latent Low-Rank Transfer Subspace Learning ($L^2TSL$), a novel framework designed to solve the "Missing Modality Problem" in transfer learning. By leveraging an auxiliary database and transferring knowledge in two directions—across modalities and across databases—the method can successfully recognize target data even when an entire modality (such as infrared images or sketches) is completely missing during the training phase.

Introduction

In many real-world recognition tasks, researchers deal with multi-modal data, such as visible light (VIS) vs. near-infrared (NIR) photos, or high-resolution (HR) vs. low-resolution (LR) images. Standard transfer learning helps bridge the gap between a "source" domain and a "target" domain, but typically assumes that at least some target data is available during training.

The authors identify a more difficult scenario called the Missing Modality Problem: what if you have a large database of VIS images, but the NIR images for your target subjects are only available at runtime? Traditional methods fail here because there is no target-modality data to "align" with. The proposed $L^2TSL$ framework uses a third, "auxiliary" database containing both modalities to act as a bridge.

Overview of the Missing Modality Problem and the proposed two-directional transfer framework.

Methodology

The core of the $L^2TSL$ approach is a mathematical framework that learns a common low-dimensional subspace where different modalities and databases can be aligned. It relies on four key pillars:

  1. Low-Rank Constraints: The model assumes that data from the same class across different domains should lie in the same low-dimensional subspace, using low-rank representation to filter out noise.
  2. Latent Factor Recovery: Since one modality is missing in the target database, the authors introduce a "latent factor" ($L$). This allows the model to mathematically "uncover" the structure of the missing data by observing how the available data relates to the auxiliary database.
  3. Common Dictionary Learning: The model learns a shared dictionary ($D$) to ensure that intrinsic features common to both modalities are preserved during the transfer process.
  4. Two-Directional Transfer: The algorithm iteratively transfers knowledge via $T(D)$ (Database Transfer) from the auxiliary database to the current database, and $T(M)$ (Modality Transfer) from the available modality to the missing one.
Mathematical framework of L2TSL incorporating low-rank and dictionary constraints.

The optimization is solved using the Inexact Augmented Lagrange Multiplier (ALM) algorithm, which efficiently handles the complex constraints including SVD steps and Sylvester equation solvers.

Experimental Evaluation

The authors evaluated their method on two distinct multi-modal challenges:

  • VIS-NIR Face Recognition: Using the BUAA and OULU face databases.
  • Resolution Transfer: Using CMU-PIE and Yale B face databases (HR vs. LR images).

Key Results

The $L^2TSL$ method consistently outperformed state-of-the-art transfer learning methods like LTSL, RDALR, and GFK. In VIS-NIR experiments, $L^2TSL$ often improved recognition rates by 5-10% over the next best competitor, achieving 57.2% accuracy where standard LTSL only managed 47.2%. The method also showed superior robustness across various subspace projection settings (PCA, LDA, ULPP/SLPP) and demonstrated fast convergence, typically stabilizing within the first 10-20 iterations.

Conclusion

$L^2TSL$ demonstrates that missing modality information is recoverable through the synergy of low-rank reconstruction and dictionary learning. While the method is robust, future work is needed to address potential dimensionality decay during long iterations and the computational cost of SVD steps in extremely large-scale datasets. The framework holds promise for extension to multi-modality scenarios and cross-media retrieval tasks.

Summary updated

Abstract

We consider an interesting problem in this paper that uses transfer learning in two directions to compensate missing knowledge from the target domain. Transfer learning tends to be exploited as a powerful tool that mitigates the discrepancy between different databases used for knowledge transfer. It can also be used for knowledge transfer between different modalities within one database. However, in either case, transfer learning will fail if the target data are missing. To overcome this, we consider knowledge transfer between different databases and modalities simultaneously in a single framework, where missing target data from one database are recovered to facilitate recognition task. We referred to this framework as Latent Low-rank Transfer Subspace Learning method (L2TSL). We first propose to use a low-rank constraint as well as dictionary learning in a learned subspace to guide the knowledge transfer between and within different databases. We then introduce a latent factor to uncover the underlying structure of the missing target data. Next, transfer learning in two directions is proposed to integrate auxiliary database for transfer learning with missing target data. Experimental results of multi-modalities knowledge transfer with missing target data demonstrate that our method can successfully inherit knowledge from the auxiliary database to complete the target domain, and therefore enhance the performance when recognizing data from the modality without any training data.
BibTeX
@inproceedings{ding2014latent,
  author = {Zhengming Ding and Ming Shao and Yun Fu},
  title = {{Latent Low-Rank Transfer Subspace Learning for Missing Modality Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8905},
}
Figure