Algorithm and Theoretical Analysis for Domain Adaptation Feature Learning with Linear Classifiers

Wenhao Jiang, Feiping Nie, Korris Fu-Lai Chung, Heng Huang

Domain Adaptation (DA) is a critical challenge in machine learning, occurring when the distribution of training data (source) differs from that of the test data (target). A common example is a sentiment classifier trained on book reviews failing when applied to electronics reviews. While many modern approaches rely on complex neural networks, recent research demonstrates that a mathematically grounded linear approach can be remarkably effective.

The Theoretical Foundation: Matching Second Moments

The core insight of this work is that for linear classifiers to generalize across domains, the second moments (covariance structures) of the source and target distributions must be aligned. The authors provide a theoretical proof that the error bound on a target domain is directly related to the difference between these second moments.

By minimizing this difference, one can learn a feature transformation $P$ that maps data into a space where the domain gap is significantly reduced. This leads to the FLAMM (Feature LeArning with second Moment Matching) algorithm. FLAMM optimizes an objective function balancing reconstruction error (to preserve information), feature regularization, and explicit second-moment matching.

Figure 2: Convergence of the second-moment difference as a function of depth K for different regularization parameters across datasets.

As shown in Figure 2, the discrepancy between source and target second moments ($\|\Delta M\|_F$) decreases rapidly as the number of layers (or iterations) $K$ increases. This empirical evidence supports the theoretical claim that stacking linear transformations can progressively close the distribution gap.

Experimental Evaluation

The authors evaluated FLAMM on two major benchmarks: the Amazon Review Dataset and the ECML/PKDD 2006 Spam Dataset. On the Amazon Review tasks, FLAMM achieved an average accuracy of 83.29%, outperforming standard baselines like PCA and CORAL, as well as the popular Marginalized Stacked Denoising Auto-encoders (mSDA). In the spam filtering task, the improvement was even more pronounced, with FLAMM reaching 93.78% accuracy compared to just 68.85% for raw features.

Figure 1: Accuracy comparison between FLAMM and SFL across different depths K on the Amazon and Spam datasets.

Figure 1 illustrates the performance of FLAMM compared to Stacked Feature Learning (SFL). In the Amazon dataset, FLAMM maintains higher accuracy as depth increases, whereas SFL's performance degrades more sharply after $K=2$. In the spam dataset, both methods benefit from increased depth, but FLAMM consistently maintains a performance lead. This highlights that while depth is beneficial, the explicit second-moment matching in FLAMM provides a more robust feature representation.

Conclusion

The FLAMM algorithm proves that "extremely easy" linear methods can rival or exceed the performance of more complex models in domain adaptation tasks. By focusing on the theoretical necessity of matching second moments, the authors provide a transparent and computationally efficient framework for bridging the domain gap. While future work may integrate these principles into non-linear deep architectures, FLAMM stands as a powerful testament to the efficacy of mathematically grounded simplicity.

Summary updated

BibTeX
@inproceedings{jiang2017algorithm,
  author = {Wenhao Jiang and Feiping Nie and Korris Fu-Lai Chung and Heng Huang},
  title = {{Algorithm and Theoretical Analysis for Domain Adaptation Feature Learning with Linear Classifiers}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/272},
}
Figure