Abstract
This paper introduces Latent Low-Rank Transfer Subspace Learning ($L^2TSL$), a novel framework designed to solve the "Missing Modality Problem" in transfer learning. By leveraging an auxiliary database and transferring knowledge in two directions—across modalities and across databases—the method can successfully recognize target data even when an entire modality (such as infrared images or sketches) is completely missing during the training phase.
Introduction
In many real-world recognition tasks, researchers deal with multi-modal data, such as visible light (VIS) vs. near-infrared (NIR) photos, or high-resolution (HR) vs. low-resolution (LR) images. Standard transfer learning helps bridge the gap between a "source" domain and a "target" domain, but typically assumes that at least some target data is available during training.
The authors identify a more difficult scenario called the Missing Modality Problem: what if you have a large database of VIS images, but the NIR images for your target subjects are only available at runtime? Traditional methods fail here because there is no target-modality data to "align" with. The proposed $L^2TSL$ framework uses a third, "auxiliary" database containing both modalities to act as a bridge.
Methodology
The core of the $L^2TSL$ approach is a mathematical framework that learns a common low-dimensional subspace where different modalities and databases can be aligned. It relies on four key pillars:
- Low-Rank Constraints: The model assumes that data from the same class across different domains should lie in the same low-dimensional subspace, using low-rank representation to filter out noise.
- Latent Factor Recovery: Since one modality is missing in the target database, the authors introduce a "latent factor" ($L$). This allows the model to mathematically "uncover" the structure of the missing data by observing how the available data relates to the auxiliary database.
- Common Dictionary Learning: The model learns a shared dictionary ($D$) to ensure that intrinsic features common to both modalities are preserved during the transfer process.
- Two-Directional Transfer: The algorithm iteratively transfers knowledge via $T(D)$ (Database Transfer) from the auxiliary database to the current database, and $T(M)$ (Modality Transfer) from the available modality to the missing one.
The optimization is solved using the Inexact Augmented Lagrange Multiplier (ALM) algorithm, which efficiently handles the complex constraints including SVD steps and Sylvester equation solvers.
Experimental Evaluation
The authors evaluated their method on two distinct multi-modal challenges:
- VIS-NIR Face Recognition: Using the BUAA and OULU face databases.
- Resolution Transfer: Using CMU-PIE and Yale B face databases (HR vs. LR images).
Key Results
The $L^2TSL$ method consistently outperformed state-of-the-art transfer learning methods like LTSL, RDALR, and GFK. In VIS-NIR experiments, $L^2TSL$ often improved recognition rates by 5-10% over the next best competitor, achieving 57.2% accuracy where standard LTSL only managed 47.2%. The method also showed superior robustness across various subspace projection settings (PCA, LDA, ULPP/SLPP) and demonstrated fast convergence, typically stabilizing within the first 10-20 iterations.
Conclusion
$L^2TSL$ demonstrates that missing modality information is recoverable through the synergy of low-rank reconstruction and dictionary learning. While the method is robust, future work is needed to address potential dimensionality decay during long iterations and the computational cost of SVD steps in extremely large-scale datasets. The framework holds promise for extension to multi-modality scenarios and cross-media retrieval tasks.
Summary updated