Abstract
Multi-domain text classification (MDTC) aims to leverage resources from multiple domains to learn a predictive model that generalizes well across all of them. While adversarial learning and shared-private paradigms have advanced the state-of-the-art, they suffer from alignment uncertainty, poor feature discriminability, and over-confident predictions on unlabeled data. We propose Co-Regularized Adversarial Learning (CRAL) to address these issues. CRAL constructs two diverse shared latent spaces, performs domain alignment in each, and punishes disagreements between their predictions on unlabeled data. Additionally, virtual adversarial training (VAT) with entropy minimization is incorporated for consistency regularization. Experiments on two major benchmarks demonstrate that CRAL outperforms existing state-of-the-art methods.
Introduction: The Challenges of Domain Alignment
In Multi-Domain Text Classification, the primary hurdle is "domain shift"—where the same features (words) may have different meanings across contexts. For instance, "predictable" is a criticism for a movie but might be a compliment for an electronic device.
Current methods often use adversarial learning to extract domain-invariant features. However, these approaches face three critical limitations:
- Alignment Uncertainty: Minimizing marginal distribution divergence does not guarantee that the domains are correctly aligned for the classification task. Multiple alignments might minimize divergence, but only some preserve class boundaries.
- Feature Discriminability: Distribution alignment alone doesn't ensure that the resulting features remain useful for distinguishing between classes (e.g., positive vs. negative sentiment).
- Over-confidence: Standard entropy minimization on unlabeled data can lead to over-confident, incorrect predictions, which degrades the quality of the learned decision boundaries.
The CRAL Framework
To overcome these challenges, the CRAL framework introduces a multi-branch architecture designed to find a robust "consensus" alignment.
Dual Shared Latent Spaces
Instead of a single shared feature space, CRAL employs two independent branches. Each branch consists of its own shared feature extractor, domain-specific extractors, and a classifier. This dual-view approach is the foundation for co-regularization.
Co-Regularization and Diversity
The core innovation is the co-regularization loss ($\mathcal{L}_d$), which penalizes disagreements between the two branches' predictions on unlabeled data. To prevent the two branches from collapsing into identical solutions, a diversity loss ($\mathcal{L}_\Delta$) is applied to push their feature centroids apart. This "agreement-under-diversity" constraint significantly narrows the space of possible domain alignments to those that are consistent across different feature perspectives.
Consistency Regularization with VAT
To address the issue of over-confidence and improve discriminability, CRAL integrates Virtual Adversarial Training (VAT) with entropy minimization. By adding small adversarial perturbations to inputs and enforcing prediction consistency, the model is encouraged to place decision boundaries in low-density regions of the feature space, adhering to the "cluster assumption."
Experimental Results
The authors evaluated CRAL on the Amazon Review Dataset (4 domains) and the FDU-MTL Dataset (16 domains).
- State-of-the-Art Performance: CRAL achieved an average accuracy of 88.00% on the Amazon dataset and 90.2% on FDU-MTL, surpassing previous leaders like CAN and DACL.
- Robustness to Unseen Domains: In Multi-Source Unsupervised Domain Adaptation (MS-UDA) tests, where the model is evaluated on a completely new domain, CRAL outperformed competitors by an average of 2.22%. This suggests that the co-regularized features are truly domain-invariant and generalizable.
- Ablation Study: The disagreement penalty ($\mathcal{L}_d$) proved to be the most vital component; removing it led to a performance drop of over 2% on the Amazon benchmark.
Conclusion
CRAL provides a powerful mechanism for multi-domain text classification by leveraging the principle of co-regularization. By training two diverse "views" of the data and forcing them to agree on unlabeled samples, the model achieves a more reliable and discriminative domain alignment. While the architecture adds some computational complexity and hyperparameter tuning requirements, the significant gains in accuracy and transferability make it a compelling new standard for MDTC tasks.
Summary updated