Dual Adversarial Co-Learning for Multi-Domain Text Classification

Yuan Wu, Yuhong Guo

Abstract

With the advent of deep learning, the performance of text classification models has been improved significantly. Nevertheless, the successful training of a good classification model requires a sufficient amount of labeled data, while it is always expensive and time consuming to annotate data. With the rapid growth of digital data, similar classification tasks can typically occur in multiple domains, while the availability of labeled data can largely vary across domains. Some domains may have abundant labeled data, while in some other domains there may only exist a limited amount (or none) of labeled data. Meanwhile text classification tasks are highly domain-dependent — a text classifier trained in one domain may not perform well in another domain. In order to address these issues, in this paper we propose a novel dual adversarial co-learning approach for multi-domain text classification (MDTC). The approach learns shared-private networks for feature extraction and deploys dual adversarial regularizations to align features across different domains and between labeled and unlabeled data simultaneously under a discrepancy based co-learning framework, aiming to improve the classifiers' generalization capacity with the learned features. We conduct experiments on multi-domain sentiment classification datasets. The results show the proposed approach achieves the state-of-the-art MDTC performance.

Introduction

In Natural Language Processing (NLP), the meaning of words is often tied to their specific context. For instance, the word "fast" serves as a positive attribute for a sports car but can be a negative one for a digital camera's battery consumption. This inherent domain-dependency makes creating a universal text classifier a significant challenge. A model trained on book reviews often fails when applied to electronics or apparel.

To address this, researchers Yuan Wu and Yuhong Guo introduced Dual Adversarial Co-Learning (DACL) at the AAAI-20 conference. DACL is designed to bridge the gap between multiple domains by learning both shared and domain-specific features while leveraging unlabeled data through a unique "tug-of-war" adversarial strategy.

The Challenge of Multi-Domain Text Classification

The primary goal of Multi-Domain Text Classification (MDTC) is to utilize knowledge from domains with abundant labeled data (source domains) to improve performance in domains with scarce or no labeled data (target domains). Traditional approaches often struggle with two main issues:

  1. Feature Interference: Domain-specific nuances can leak into the shared feature space, degrading the quality of domain-invariant representations.
  2. Generalization Gap: Models often overfit to the limited labeled data available, failing to generalize to the vast amounts of unlabeled data present in real-world scenarios.

The DACL Framework

The DACL architecture utilizes a Shared-Private Network design. It employs a shared feature extractor ($F_s$) to capture universal sentiment patterns and multiple private extractors ($F_d$) to isolate domain-specific characteristics.

The architecture of the proposed Dual Adversarial Co-Learning (DACL) framework.

The "Dual" aspect of the framework refers to two distinct adversarial training loops:

  • Domain Alignment: A multinomial discriminator ($\mathcal{D}$) attempts to identify the source domain of the input text. The shared extractor is trained to generate features that "fool" this discriminator, ensuring they are truly domain-invariant.
  • Labeled-Unlabeled Alignment (Co-Learning): This mechanism uses two separate classifiers ($C_1$ and $C_2$). During training, these classifiers are encouraged to maximize their disagreement on unlabeled data. Simultaneously, the feature extractors are updated to produce representations that minimize this disagreement. This adversarial process identifies "hard" examples near the decision boundary, forcing the model to learn more robust and generalizable features.

To prevent the shared and private features from overlapping, a Separation Regularizer ($\mathcal{L}_\Delta$) is employed to ensure the orthogonality of the two feature spaces.

Experimental Evaluation

The researchers evaluated DACL on two major benchmarks: the Amazon Review Dataset (4 domains) and the FDU-MTL Dataset (16 domains).

Performance comparison of DACL against state-of-the-art models on the FDU-MTL dataset.

As shown in the results, DACL consistently outperforms previous state-of-the-art models such as MAN (Multinomial Adversarial Networks) and ASP-MTL. On the FDU-MTL dataset, DACL achieved an average accuracy of 89.1%, surpassing the next best model (MAN-NLL) by nearly a full percentage point. It achieved the highest accuracy in 12 out of the 16 individual domains. Similarly, on the Amazon dataset, DACL reached an average accuracy of 86.59%, a notable improvement over the previous benchmark of 85.66%.

Key Insights

  • Dual Alignment Strategy: By aligning both across domains and between labeled/unlabeled data distributions, DACL effectively prevents overfitting to specific domain labels.
  • Disagreement-Based Learning: Utilizing the disagreement between two classifiers provides a stronger training signal for unlabeled data than a single-classifier approach.
  • Robust Feature Extraction: The combination of shared-private networks and adversarial regularization ensures that the learned features are both discriminative and transferable.

Conclusion

DACL represents a significant step forward in multi-domain text classification. By integrating shared-private feature extraction with dual adversarial regularizations, it effectively addresses the challenges of domain shift and data scarcity. While the framework's complexity requires careful tuning, its ability to achieve state-of-the-art results across diverse domains demonstrates its potential for real-world NLP applications. Future work may explore integrating DACL with large-scale pre-trained language models like BERT to further enhance its performance.

Summary updated

Abstract

With the advent of deep learning, the performance of text classification models have been improved significantly. Nevertheless, the successful training of a good classification model requires a sufficient amount of labeled data, while it is always expensive and time consuming to annotate data. With the rapid growth of digital data, similar classification tasks can typically occur in multiple domains, while the availability of labeled data can largely vary across domains. Some domains may have abundant labeled data, while in some other domains there may only exist a limited amount (or none) of labeled data. Meanwhile text classification tasks are highly domain-dependent — a text classifier trained in one domain may not perform well in another domain. In order to address these issues, in this paper we propose a novel dual adversarial co-learning approach for multi-domain text classification (MDTC). The approach learns shared-private networks for feature extraction and deploys dual adversarial regularizations to align features across different domains and between labeled and unlabeled data simultaneously under a discrepancy based co-learning framework, aiming to improve the classifiers' generalization capacity with the learned features. We conduct experiments on multi-domain sentiment classification datasets. The results show the proposed approach achieves the state-of-the-art MDTC performance.
BibTeX
@inproceedings{wu2020dual,
  author = {Yuan Wu and Yuhong Guo},
  title = {{Dual Adversarial Co-Learning for Multi-Domain Text Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=SFxsCs3LEXt},
}
Figure