Abstract
This paper introduces Marginalized Stacked Denoising Autoencoders (mSDA), a breakthrough approach for domain adaptation that replaces slow, iterative neural network training with a simple closed-form solution. By mathematically "marginalizing out" noise during training, mSDA achieves a massive 200x to 700x speedup over traditional methods while maintaining or exceeding state-of-the-art accuracy on sentiment analysis tasks.
Introduction
In machine learning, domain adaptation is the challenge of training a model on one type of data (the "source," like book reviews) and getting it to perform well on a different but related type (the "target," like kitchen appliance reviews).
Stacked Denoising Autoencoders (SDAs) have become a standard for this task because they learn robust features by reconstructing original data from corrupted versions. However, SDAs face significant computational hurdles: they rely on slow, iterative stochastic gradient descent (SGD) and struggle to scale with high-dimensional data, such as large text vocabularies.
Methodology
The authors propose a "marginalized" version of the SDA to address these pain points. The technical approach involves:
- Linear Denoisers: Using linear mappings as basic building blocks instead of complex non-linear layers.
- Marginalization: Instead of training on a finite set of noisy samples, mSDA mathematically calculates the expected corruption over an infinite number of noisy versions.
- Closed-Form Solution: By using linear units and marginalizing noise, the optimal weights can be computed directly via a system of linear equations ($W = E[P]E[Q]^{-1}$), bypassing backpropagation and SGD.
- Stacking and Non-linearity: To retain the benefits of deep learning, these linear layers are stacked with a non-linear "squashing" function (like
tanh) applied between them. - High-Dimensional Handling: For massive datasets, features are divided into sub-problems using "pivot features" to maintain matrix efficiency.
Experimental Setup
The researchers evaluated mSDA on the Amazon Reviews benchmark:
- Small Set: 4 product domains (Books, DVDs, Electronics, Kitchen Appliances), creating 12 transfer tasks.
- Complete Set: A massive collection of over 340,000 reviews across 20 different product categories, resulting in 380 possible transfer tasks.
Key Results and Discussion
Speed and Accuracy Trade-offs
mSDA provides a dramatic improvement in efficiency. On small tasks, it is roughly 180x faster than standard SDA. On the full Amazon dataset, training time was reduced from days to just minutes—a ~700x speedup for a single layer. Despite this speed, mSDA achieves accuracy comparable to or better than traditional SDAs and other popular methods like Structural Correspondence Learning (SCL) and PCA.
Scalability to High-Dimensional Data
One of the primary advantages of mSDA is its ability to handle large feature sets. Traditional SDAs are often forced to truncate data to the top 5,000 features to remain computationally feasible. In contrast, mSDA scales successfully to 40,000+ features. Including these "rare" features (low-frequency words) significantly improves model accuracy.
Conclusion
mSDA represents a significant advancement in domain adaptation. By replacing iterative optimization with a closed-form solution, it enables the training of deep, robust feature extractors in a fraction of the time previously required. This efficiency not only speeds up research but also allows for the use of more comprehensive feature sets, leading to better performance on complex real-world tasks. Future work may explore applying this marginalization technique to other neural network architectures.
Summary updated