Co-Regularized Adversarial Learning for Multi-Domain Text Classification

Yuan Wu, Diana Inkpen, Ahmed El-Roby

Abstract

Multi-domain text classification (MDTC) aims to leverage resources from multiple domains to learn a predictive model that generalizes well across all of them. While adversarial learning and shared-private paradigms have advanced the state-of-the-art, they suffer from alignment uncertainty, poor feature discriminability, and over-confident predictions on unlabeled data. We propose Co-Regularized Adversarial Learning (CRAL) to address these issues. CRAL constructs two diverse shared latent spaces, performs domain alignment in each, and punishes disagreements between their predictions on unlabeled data. Additionally, virtual adversarial training (VAT) with entropy minimization is incorporated for consistency regularization. Experiments on two major benchmarks demonstrate that CRAL outperforms existing state-of-the-art methods.

Introduction: The Challenges of Domain Alignment

In Multi-Domain Text Classification, the primary hurdle is "domain shift"—where the same features (words) may have different meanings across contexts. For instance, "predictable" is a criticism for a movie but might be a compliment for an electronic device.

Current methods often use adversarial learning to extract domain-invariant features. However, these approaches face three critical limitations:

  1. Alignment Uncertainty: Minimizing marginal distribution divergence does not guarantee that the domains are correctly aligned for the classification task. Multiple alignments might minimize divergence, but only some preserve class boundaries.
  2. Feature Discriminability: Distribution alignment alone doesn't ensure that the resulting features remain useful for distinguishing between classes (e.g., positive vs. negative sentiment).
  3. Over-confidence: Standard entropy minimization on unlabeled data can lead to over-confident, incorrect predictions, which degrades the quality of the learned decision boundaries.
Conceptual illustration of alignment uncertainty (b) and how co-regularization (c) helps identify the correct alignment by leveraging multiple views.

The CRAL Framework

To overcome these challenges, the CRAL framework introduces a multi-branch architecture designed to find a robust "consensus" alignment.

Dual Shared Latent Spaces

Instead of a single shared feature space, CRAL employs two independent branches. Each branch consists of its own shared feature extractor, domain-specific extractors, and a classifier. This dual-view approach is the foundation for co-regularization.

Co-Regularization and Diversity

The core innovation is the co-regularization loss ($\mathcal{L}_d$), which penalizes disagreements between the two branches' predictions on unlabeled data. To prevent the two branches from collapsing into identical solutions, a diversity loss ($\mathcal{L}_\Delta$) is applied to push their feature centroids apart. This "agreement-under-diversity" constraint significantly narrows the space of possible domain alignments to those that are consistent across different feature perspectives.

The CRAL architecture featuring dual shared latent spaces, classifiers ($C$), discriminators ($D$), and the various loss terms including co-regularization ($\mathcal{L}_d$) and diversity ($\mathcal{L}_\Delta$).

Consistency Regularization with VAT

To address the issue of over-confidence and improve discriminability, CRAL integrates Virtual Adversarial Training (VAT) with entropy minimization. By adding small adversarial perturbations to inputs and enforcing prediction consistency, the model is encouraged to place decision boundaries in low-density regions of the feature space, adhering to the "cluster assumption."

Experimental Results

The authors evaluated CRAL on the Amazon Review Dataset (4 domains) and the FDU-MTL Dataset (16 domains).

  • State-of-the-Art Performance: CRAL achieved an average accuracy of 88.00% on the Amazon dataset and 90.2% on FDU-MTL, surpassing previous leaders like CAN and DACL.
  • Robustness to Unseen Domains: In Multi-Source Unsupervised Domain Adaptation (MS-UDA) tests, where the model is evaluated on a completely new domain, CRAL outperformed competitors by an average of 2.22%. This suggests that the co-regularized features are truly domain-invariant and generalizable.
  • Ablation Study: The disagreement penalty ($\mathcal{L}_d$) proved to be the most vital component; removing it led to a performance drop of over 2% on the Amazon benchmark.

Conclusion

CRAL provides a powerful mechanism for multi-domain text classification by leveraging the principle of co-regularization. By training two diverse "views" of the data and forcing them to agree on unlabeled samples, the model achieves a more reliable and discriminative domain alignment. While the architecture adds some computational complexity and hyperparameter tuning requirements, the significant gains in accuracy and transferability make it a compelling new standard for MDTC tasks.

Summary updated

Abstract

Multi-domain text classification (MDTC) aims to leverage all available resources from multiple domains to learn a predictive model that can generalize well on these domains. Recently, many MDTC methods adopt adversarial learning, shared-private paradigm, and entropy minimization to yield state-of-the-art results. However, these approaches face three issues: (1) Minimizing domain divergence can not fully guarantee the success of domain alignment; (2) Aligning marginal feature distributions can not fully guarantee the discriminability of the learned features; (3) Standard entropy minimization may make the predictions on unlabeled data over-confident, deteriorating the discriminability of the learned features. In order to address the above issues, we propose a co-regularized adversarial learning (CRAL) mechanism for MDTC. This approach constructs two diverse shared latent spaces, performs domain alignment in each of them, and punishes the disagreements of these two alignments with respect to the predictions on unlabeled data. Moreover, virtual adversarial training (VAT) with entropy minimization is incorporated to impose consistency regularization to the CRAL method. Experiments show that our model outperforms state-of-the-art methods on two MDTC benchmarks.
BibTeX
@inproceedings{wu2022co,
  author = {Yuan Wu and Diana Inkpen and Ahmed El-Roby},
  title = {{Co-Regularized Adversarial Learning for Multi-Domain Text Classification}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=aiyuAFiofz},
}
Figure