ZeroAE: Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification

Kaihao Guo, Hang Yu, Cong Liao, Jianguo Li, Haipeng ZhangACL 2023 · DOI

Abstract

Zero-shot learning (ZSL) aims to classify text into categories not seen during training. Traditional methods often struggle with high uncertainty or low-quality synthetic data. We present ZeroAE, a novel autoencoder framework for Transductive Zero-shot Learning (TZSL) that bridges the gap between discriminative encoders and generative decoders. ZeroAE disentangles label-relevant features from stylistic noise and utilizes an Indirect Uncertainty-Aware Sampling (IUAS) strategy to leverage unlabeled target domain data. Our approach achieves state-of-the-art performance across multiple benchmarks, often surpassing models trained with labeled data.

Introduction

Zero-shot text classification is a challenging task where models must generalize to unseen classes. Current approaches typically fall into two categories: encoder-based discriminative models (e.g., BERT) and decoder-based generative models (e.g., GPT-2). Discriminative models often face high uncertainty in new domains, while generative models may produce noisy synthetic data that degrades classifier performance. Furthermore, many methods fail to exploit unlabeled data available in the target domain—a paradigm known as Transductive ZSL (TZSL). ZeroAE addresses these issues by integrating both paradigms into a unified autoencoder framework.

Methodology

ZeroAE is the first end-to-end autoencoder designed specifically for TZSL. Its architecture is composed of several key components that work in tandem to improve classification robustness.

The ZeroAE architecture overview.

Disentangled Latent Spaces

ZeroAE employs two separate BERT-based encoders to decompose text into two distinct latent spaces:

  1. Label-Relevant Encoder: Captures the semantic content necessary for classification.
  2. Label-Irrelevant Encoder: Captures stylistic features such as syntax and punctuation. A discriminator is used to ensure the independence of these spaces, preventing stylistic noise from interfering with the classification task.

Prompt-based GPT-2 Decoder

The decoder, based on GPT-2, reconstructs the original text from the latent representations. It also serves a dual purpose: generating new labeled data for unseen domains. By prompting the decoder with specific class labels, the model can "teach" the encoder about new categories.

Indirect Uncertainty-Aware Sampling (IUAS)

To effectively utilize unlabeled data without the pitfalls of standard pseudo-labeling (which can lead to error accumulation), ZeroAE introduces IUAS. This strategy identifies "hard" samples with high model uncertainty. Instead of directly labeling them, the GPT-2 decoder is used to simulate similar, labeled samples, helping the encoder refine decision boundaries for difficult cases.

Vector Quantization (VQ)

To prevent posterior collapse—where the decoder ignores latent variables—ZeroAE discretizes the latent space using a codebook (VQ-VAE technique). This ensures the decoder remains grounded in the latent representations provided by the encoders.

Experimental Evaluation

The model was evaluated on four diverse benchmarks: Topic (Yahoo news), Situation (news events), Emotion (tweets/tales), and Complaint (Alipay customer service). Two scenarios were tested: label-partially-unseen (some training labels available) and label-fully-unseen (no training labels).

Key Results and Discussion

ZeroAE established new state-of-the-art (SOTA) benchmarks across all datasets.

Performance comparison in the label-partially-unseen scenario.

As shown in the results, ZeroAE outperformed existing methods by an average of 15.93% in the label-partially-unseen scenario and 8.70% in the label-fully-unseen scenario. Notably, ZeroAE trained without any labels often surpassed SOTA methods that utilized labeled data.

Ablation studies highlighted the importance of each component. Removing the IUAS strategy resulted in the most significant performance drop (~10.56%), confirming that the method of sampling and utilizing unlabeled data is critical for TZSL success. Additionally, the disentanglement of style and substance proved vital; without it, the classifier's robustness significantly decreased.

Conclusion

ZeroAE demonstrates that the mutual promotion of encoders and decoders can overcome the limitations of traditional zero-shot classification. By focusing on hard samples through IUAS and stripping away stylistic noise via disentanglement, ZeroAE provides a robust framework for transductive learning. Future work will explore optimizing the balance between the various loss components and reducing the computational overhead associated with running multiple large pre-trained language models.

Summary updated

BibTeX
@inproceedings{guo2023zeroae,
  author = {Kaihao Guo and Hang Yu and Cong Liao and Jianguo Li and Haipeng Zhang},
  title = {{ZeroAE: Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.200},
}
Figure