Abstract
Multi-domain text classification (MDTC) has obtained remarkable achievements due to the advent of deep learning. Recently, many endeavors are devoted to applying adversarial learning to extract domain-invariant features to yield state-of-the-art results. However, these methods still face one challenge: transforming original features to be domain-invariant distorts the distributions of the original features, degrading the discriminability of the learned features. To address this issue, we first investigate the structure of the batch classification output matrix and theoretically justify that the discriminability of the learned features has a positive correlation with the Frobenius norm of the batch output matrix. Based on this finding, we propose a maximum batch Frobenius norm (MBF) method to boost the feature discriminability for MDTC. Experiments on two MDTC benchmarks show that our MBF approach can effectively advance the performance of the state-of-the-art.
Introduction
Multi-domain text classification (MDTC) aims to leverage knowledge from multiple source domains to improve classification performance on a target domain. A common strategy involves adversarial learning, where a feature extractor and a domain discriminator engage in a minimax game. The goal is to learn features that are domain-invariant, making it impossible for the discriminator to distinguish between domains.
However, a significant drawback of this approach is the "transferability-discriminability" trade-off. While adversarial training successfully aligns feature distributions across domains (improving transferability), it often distorts the original feature space. This distortion can lead to a loss of discriminability, where the features become less effective for the primary task of classification, such as distinguishing between positive and negative sentiment.
Methodology: Maximum Batch Frobenius Norm (MBF)
To mitigate the loss of discriminability, the authors introduce the Maximum Batch Frobenius Norm (MBF) method. This approach is built upon a shared-private architecture, utilizing shared feature extractors for domain-invariant information and domain-specific extractors for unique characteristics.
The core innovation lies in the use of the batch output matrix ($A$) from the classifier for unlabeled data. This matrix represents the predicted class probabilities for a batch of samples. The authors theoretically demonstrate that the Frobenius norm of this matrix is inversely related to Shannon entropy. Consequently, maximizing the Frobenius norm forces the model to produce more "confident" predictions (probabilities closer to 0 or 1). This sharpening of decision boundaries directly enhances the discriminability of the learned features without sacrificing domain invariance. The MBF term is integrated as a regularization loss within the overall objective function, balanced by hyperparameters $\alpha$ and $\beta$.
Experimental Setup
The MBF method was evaluated on two standard MDTC benchmarks:
- Amazon Review Dataset: A classic benchmark featuring four domains: Books, DVDs, Electronics, and Kitchen. Each domain contains 2,000 labeled samples.
- FDU-MTL Dataset: A more extensive dataset comprising 16 domains, including 14 product categories (e.g., apparel, toys, baby) and two movie review sources (IMDB and MR). Unlike the Amazon dataset, FDU-MTL uses raw text, providing a more realistic challenge.
Results and Discussion
The experimental results demonstrate that MBF consistently outperforms existing state-of-the-art methods across both benchmarks.
On the Amazon Review Dataset, MBF achieved an average accuracy of 87.71%, a 1.01% improvement over the previous best model (CAN). The gains were particularly notable in the "Kitchen" and "Electronics" domains, where the model significantly boosted performance compared to baselines like CMSC and MAN.
On the more complex FDU-MTL dataset, MBF reached an average accuracy of 90.1%, surpassing the state-of-the-art by 0.7%. It achieved the highest performance in 9 out of the 16 individual domains, showcasing its robustness across diverse text categories. The success of MBF highlights that the Frobenius norm serves as an efficient and theoretically grounded proxy for feature discriminability. By leveraging unlabeled data to "sharpen" predictions, the model effectively balances the need for domain-invariant features with the requirement for high classification accuracy.
Conclusion
The Maximum Batch Frobenius Norm (MBF) method provides a novel solution to the transferability-discriminability trade-off in multi-domain text classification. By maximizing the Frobenius norm of the batch output matrix, the model learns highly discriminative features that remain transferable across domains. While the results are promising, future work could explore the sensitivity of the model to its hyperparameters and investigate the applicability of the Frobenius norm strategy in other modalities, such as computer vision or audio processing.
Summary updated