Marginalized Denoising Autoencoders for Domain Adaptation

Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, Fei Sha

Abstract

This paper introduces Marginalized Stacked Denoising Autoencoders (mSDA), a breakthrough approach for domain adaptation that replaces slow, iterative neural network training with a simple closed-form solution. By mathematically "marginalizing out" noise during training, mSDA achieves a massive 200x to 700x speedup over traditional methods while maintaining or exceeding state-of-the-art accuracy on sentiment analysis tasks.

Introduction

In machine learning, domain adaptation is the challenge of training a model on one type of data (the "source," like book reviews) and getting it to perform well on a different but related type (the "target," like kitchen appliance reviews).

Stacked Denoising Autoencoders (SDAs) have become a standard for this task because they learn robust features by reconstructing original data from corrupted versions. However, SDAs face significant computational hurdles: they rely on slow, iterative stochastic gradient descent (SGD) and struggle to scale with high-dimensional data, such as large text vocabularies.

Methodology

The authors propose a "marginalized" version of the SDA to address these pain points. The technical approach involves:

  • Linear Denoisers: Using linear mappings as basic building blocks instead of complex non-linear layers.
  • Marginalization: Instead of training on a finite set of noisy samples, mSDA mathematically calculates the expected corruption over an infinite number of noisy versions.
  • Closed-Form Solution: By using linear units and marginalizing noise, the optimal weights can be computed directly via a system of linear equations ($W = E[P]E[Q]^{-1}$), bypassing backpropagation and SGD.
  • Stacking and Non-linearity: To retain the benefits of deep learning, these linear layers are stacked with a non-linear "squashing" function (like tanh) applied between them.
  • High-Dimensional Handling: For massive datasets, features are divided into sub-problems using "pivot features" to maintain matrix efficiency.

Experimental Setup

The researchers evaluated mSDA on the Amazon Reviews benchmark:

  • Small Set: 4 product domains (Books, DVDs, Electronics, Kitchen Appliances), creating 12 transfer tasks.
  • Complete Set: A massive collection of over 340,000 reviews across 20 different product categories, resulting in 380 possible transfer tasks.

Key Results and Discussion

Speed and Accuracy Trade-offs

mSDA provides a dramatic improvement in efficiency. On small tasks, it is roughly 180x faster than standard SDA. On the full Amazon dataset, training time was reduced from days to just minutes—a ~700x speedup for a single layer. Despite this speed, mSDA achieves accuracy comparable to or better than traditional SDAs and other popular methods like Structural Correspondence Learning (SCL) and PCA.

Transfer Ratio vs. Training Time on Amazon Benchmarks

Scalability to High-Dimensional Data

One of the primary advantages of mSDA is its ability to handle large feature sets. Traditional SDAs are often forced to truncate data to the top 5,000 features to remain computationally feasible. In contrast, mSDA scales successfully to 40,000+ features. Including these "rare" features (low-frequency words) significantly improves model accuracy.

Scalability of mSDA vs. SDA with increasing feature dimensions

Conclusion

mSDA represents a significant advancement in domain adaptation. By replacing iterative optimization with a closed-form solution, it enables the training of deep, robust feature extractors in a fraction of the time previously required. This efficiency not only speeds up research but also allows for the use of more comprehensive feature sets, leading to better performance on complex real-world tasks. Future work may explore applying this marginalization technique to other neural network architectures.

Summary updated

Abstract

Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks.
BibTeX
@misc{chen2012marginalized,
  author = {Minmin Chen and Zhixiang Eddie Xu and Kilian Q. Weinberger and Fei Sha},
  title = {{Marginalized Denoising Autoencoders for Domain Adaptation}},
  year = {2012},
  eprint = {1206.4683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1206.4683},
}
Figure