Co-Regularized Adversarial Learning for Multi-Domain Text Classification

Yuan Wu, Diana Inkpen, Ahmed El-Roby

Abstract

Multi-Domain Text Classification (MDTC) aims to leverage data from multiple source domains to improve performance across all domains. Traditional adversarial methods often suffer from alignment uncertainty and label mismatch. We propose Co-Regularized Adversarial Learning (CRAL), which utilizes two diverse latent spaces to ensure more robust domain alignment. By forcing two independent "views" of the data to agree on unlabeled samples while maintaining their unique perspectives, CRAL achieves state-of-the-art results on major sentiment analysis benchmarks.

Introduction

In Multi-Domain Text Classification (MDTC), the objective is to train a model on data from several distinct sources (e.g., book, electronics, and movie reviews) such that it generalizes well across all of them. A primary challenge in this field is that language is highly context-dependent; for instance, the word "easy" may be positive in the context of a camera's usability but negative when describing a movie's predictable plot.

Current state-of-the-art methods typically employ Adversarial Learning to extract "domain-invariant" features by stripping away domain-specific noise. However, the authors identify three critical gaps in this approach:

  1. Alignment Uncertainty: Minimizing statistical distance between domains does not guarantee that features are correctly aligned for the specific task.
  2. Label Mismatch: Perfect alignment of data "shapes" can still result in mapping "positive" samples from one domain onto "negative" samples of another.
  3. Over-confidence: Techniques like entropy minimization can lead to brittle decision boundaries by making the model too certain of incorrect predictions on unlabeled data.
Conceptual illustration of (a) domain misalignment, (b) label mismatch, and (c) correct alignment achieved by CRAL.

The CRAL Framework

The core innovation of this work is the Co-regularized Adversarial Learning (CRAL) mechanism. Instead of a single shared feature space, the model constructs two diverse shared latent spaces through a two-branch architecture.

The CRAL architecture featuring two diverse branches for robust domain alignment.

Architecture and Loss Functions

Each branch in the CRAL architecture contains its own shared feature extractor, domain-specific extractors, a domain discriminator, and a task classifier. The framework is governed by three key components:

  • Diversity Regularization ($\mathcal{L}_\Delta$): To prevent the two branches from collapsing into identical representations, a penalty is applied to push the centroids of the two latent spaces apart. This ensures the model analyzes the data from two distinct "angles."
  • Co-Regularization ($\mathcal{L}_d$): This component penalizes disagreements between the two branches when predicting labels for unlabeled data. By forcing two different perspectives to reach a consensus, the model constrains the search space of possible alignments to those most likely to be correct.
  • Virtual Adversarial Training (VAT): To address over-confidence, VAT adds calculated perturbations to input data and requires the model to maintain consistent predictions. This effectively smooths the decision boundary in high-density data regions.

Experimental Evaluation

The authors evaluated CRAL on two standard MDTC benchmarks:

  1. Amazon Review Dataset: Contains 4 domains (Books, DVDs, Electronics, Kitchen) with 2,000 labeled samples each, pre-processed into bag-of-features.
  2. FDU-MTL Dataset: A more challenging dataset with 16 domains using raw text, allowing for the use of Convolutional Neural Networks (CNNs) to capture local word patterns.

Results and Analysis

CRAL consistently outperformed previous state-of-the-art models, such as CAN and DACL:

  • Amazon Dataset: Achieved an average accuracy of 88.00%, a 1.3% improvement over the previous best.
  • FDU-MTL Dataset: Achieved an average accuracy of 90.2%, outperforming the baseline on 12 out of 16 domains.
  • Generalization: In tests on completely unseen domains (MS-UDA), CRAL outperformed the baseline by 2.22%, demonstrating superior generalization capabilities.

Ablation studies revealed that the co-regularization term ($\mathcal{L}_d$) was the most impactful component, confirming that diversity-driven agreement is key to robust alignment.

Conclusion

CRAL demonstrates that "two heads are better than one" in the context of domain adaptation. By maintaining diverse perspectives and enforcing agreement on unlabeled data, the model achieves more reliable domain-invariant features. Future work may involve integrating this strategy with large-scale pre-trained transformers and optimizing the computational overhead associated with the multi-branch architecture.

Summary updated

Abstract

Multi-domain text classification (MDTC) aims to leverage all available resources from multiple domains to learn a predictive model that can generalize well on these domains. Recently, many MDTC methods adopt adversarial learning, shared-private paradigm, and entropy minimization to yield state-of-the-art results. However, these approaches face three issues: (1) Minimizing domain divergence can not fully guarantee the success of domain alignment; (2) Aligning marginal feature distributions can not fully guarantee the discriminability of the learned features; (3) Standard entropy minimization may make the predictions on unlabeled data over-confident, deteriorating the discriminability of the learned features. In order to address the above issues, we propose a co-regularized adversarial learning (CRAL) mechanism for MDTC. This approach constructs two diverse shared latent spaces, performs domain alignment in each of them, and punishes the disagreements of these two alignments with respect to the predictions on unlabeled data. Moreover, virtual adversarial training (VAT) with entropy minimization is incorporated to impose consistency regularization to the CRAL method. Experiments show that our model outperforms state-of-the-art methods on two MDTC benchmarks.
BibTeX
@inproceedings{wu2022co,
  author = {Yuan Wu and Diana Inkpen and Ahmed El-Roby},
  title = {{Co-Regularized Adversarial Learning for Multi-Domain Text Classification}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=aiyuAFiofz},
}
Figure