Deep Low-Rank Coding for Transfer Learning

Zhengming Ding, Ming Shao, Yun Fu

Introduction

Transfer learning aims to leverage knowledge from a data-rich source domain to improve performance in a data-starved target domain. A significant challenge in this field is the "domain gap"—the discrepancy in data distributions between the two domains. While traditional "shallow" methods seek a single common subspace for alignment, they often fail to capture the complex, hierarchical discriminative information present in modern datasets. Conversely, standard deep learning architectures frequently overlook the target domain during feature extraction, potentially discarding information vital for bridging the domain gap.

To address these issues, researchers from Northeastern University proposed Deep Low-Rank Coding (DLRC). This framework integrates the representational power of deep learning with the structural constraints of low-rank modeling to jointly learn discriminative features and align domains in a layer-by-layer, coarse-to-fine manner.

The DLRC Framework

The core of the DLRC architecture is the Single-layer Low-Rank Coding (SLRC) module. By stacking these modules, the model builds a deep hierarchy that refines features from basic shapes and textures at lower layers to complex semantic concepts at higher layers.

Figure 1: The DLRC architecture illustrating the layer-by-layer transformation of source and target data through low-rank coding and marginalized denoising.

Each SLRC layer incorporates three key mechanisms:

  1. Low-Rank Constraint: The model identifies a coding ($Z$) that can reconstruct the data while ensuring the reconstruction is locality-aware. This facilitates knowledge transfer between similar local spaces in the source and target domains.
  2. Marginalized Denoising: Drawing from Marginalized Stacked Denoising Autoencoders (mSDA), the framework learns a transformation matrix ($W$) robust to noise. It achieves this by training the model to reconstruct clean data from an infinite number of corrupted versions.
  3. Iterative Structure Learning: To mitigate conditional distribution mismatches, the model uses a nearest-neighbor classifier to assign "pseudo-labels" to the most confident target domain samples. These labels guide a structure matrix ($H$), forcing the low-rank coding to preserve class-specific information.

Experimental Results and Analysis

The authors evaluated DLRC against state-of-the-art baselines (e.g., TCA, GFK, mSDA) on diverse benchmarks including Office + Caltech-256, Reuters-215782, and digit datasets (USPS + MNIST). DLRC consistently outperformed these methods. For instance, in the "Amazon to DSLR" task of the Office + Caltech-256 benchmark, DLRC achieved 41.8% accuracy, surpassing the GUMA baseline's 38.2%.

Empirical analysis highlights the efficiency and structural benefits of the DLRC framework.

Figure 4: Empirical analysis of DLRC showing (a) rapid convergence of the objective value, (b) parameter sensitivity, and (c) the impact of layer depth on recognition rates.

Efficiency and Convergence

Despite its deep architecture, DLRC is computationally efficient. Utilizing an Augmented Lagrange Multiplier optimization strategy, the model typically converges within 10 iterations per layer, as shown in Figure 4(a).

The Impact of Depth

A critical finding is that depth significantly enhances recognition performance. As illustrated in Figure 4(c), recognition rates generally improve as the number of layers increases, typically peaking around 5 layers. This confirms that deep structures are superior to shallow ones for complex transfer tasks.

Limitations and Future Outlook

While depth is beneficial, the authors noted the risk of "negative transfer" if the architecture becomes too deep (e.g., beyond 7 layers). In such cases, the model may overfit or attempt to force non-existent alignments, leading to a performance drop. Additionally, the framework's success is partially dependent on the initial accuracy of pseudo-labels; poor initial labeling can propagate errors through the hierarchy.

Future research directions include exploring non-linear kernels and integrating low-rank coding principles directly into Convolutional Neural Networks (CNNs) to further enhance cross-domain feature extraction.

Summary updated

Abstract

Recent researches on transfer learning exploit deep structures for discriminative feature representation to tackle cross-domain disparity. However, few of them are able to joint feature learning and knowledge transfer in a unified deep framework. In this paper, we develop a novel approach, called Deep Low-Rank Coding (DLRC), for transfer learning. Specifically, discriminative low-rank coding is achieved in the guidance of an iterative supervised structure term for each single layer. In this way, both marginal and conditional distributions between two domains intend to be mitigated. In addition, a marginalized denoising feature transformation is employed to guarantee the learned single-layer low-rank coding to be robust despite of corruptions or noises. Finally, by stacking multiple layers of low-rank codings, we manage to learn robust cross-domain features from coarse to fine. Experimental results on several benchmarks have demonstrated the effectiveness of our proposed algorithm on facilitating the recognition performance for the target domain.
BibTeX
@inproceedings{ding2015deep,
  author = {Zhengming Ding and Ming Shao and Yun Fu},
  title = {{Deep Low-Rank Coding for Transfer Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure