Complementary Auxiliary Classifiers for Label-Conditional Text Generation

Yuan Li, Chunyuan Li, Yizhe Zhang, Xiujun Li, Guoqing Zheng, Lawrence Carin, Jianfeng GaoAAAI 2020 · OpenReview

Abstract

Learning to generate text with a given label is a challenging task because natural language sentences are highly variable and ambiguous. It renders difficulties in trade-off between sentence quality and label fidelity. In this paper, we present CARA to alleviate the issue, where two auxiliary classifiers work simultaneously to ensure that (1) the encoder learns disentangled features and (2) the generator produces label-related sentences. Two practical techniques are further proposed to improve the performance, including annealing the learning signal from the auxiliary classifier, and enhancing the encoder with pre-trained language models. To establish a comprehensive benchmark fostering future research, we consider a suite of four datasets, and systematically reproduce three representative methods. CARA shows consistent improvement over the previous methods on the task of label-conditional text generation, and achieves state-of-the-art on the task of attribute transfer.

Introduction

The CARA (Complementary Auxiliary classifier Regularized Auto-encoder) framework addresses a major hurdle in conditional text generation: the tendency of models to ignore input labels (such as sentiment or topic) in favor of latent features. Generating text conditioned on a specific label is a core challenge in NLP. Most modern approaches use Latent Variable Models (LVMs) like Variational Autoencoders (VAEs) to "disentangle" features, aiming for a latent code ($z$) that captures content and a label ($y$) that captures the target attribute.

The Non-Identifiability Problem

The authors identify a "non-identifiability" issue in standard disentanglement approaches. Even if a model successfully strips label information from the latent code, the generator often learns a "shortcut." Because it aims to reconstruct training data perfectly, it might ignore the label $y$ entirely and rely solely on the latent code to generate the sentence.

Illustration of the non-identifiability issue in label-conditional text generation.

As illustrated above, different configurations of latent variables and labels can lead to the same observed sentences ($x_1, x_2$). This is especially problematic with non-parallel data, leading to models that produce high-quality text but fail to actually change the sentiment or topic as requested. Disentanglement alone does not guarantee controllability; an explicit signal in the output space is required to enforce fidelity.

The CARA Framework

CARA builds on the Adversarially Regularized Autoencoder (ARAE) by adding a second, "complementary" layer of supervision through dual auxiliary classifiers:

  1. Latent Space Classifier ($C$): Operates on the encoder's output to predict the label from the latent code. The encoder is trained to "fool" this classifier, ensuring the latent code is label-agnostic.
  2. Observation Space Classifier ($C^{MI}$): This complementary classifier looks at the generated text and predicts the label. The generator is trained to make this classifier's job easy, forcing the output text to manifest the intended attribute. This maximizes the mutual information between the generated sentence and its label.

Practical Techniques for Stability

To prevent the model from sacrificing text quality for label accuracy (which can lead to repetitive, low-diversity text), the authors propose two techniques:

  • Annealing Schedule: The influence of the second classifier is gradually increased during training to balance the trade-off between sounding natural and matching the label.
  • BERT Encoder: Replacing standard RNN encoders with a pre-trained BERT model provides more robust latent representations, helping the model avoid "posterior collapse."

Experimental Results

CARA was evaluated on four datasets: Personality Captioning, Style Captioning, Yahoo Questions (topic-based), and Yelp (sentiment-based). The researchers used the G-score (geometric mean of BLEU for quality and Accuracy for label fidelity) as their primary metric.

  • Label Fidelity: CARA significantly outperformed baselines. On the Yahoo dataset, CARA achieved a label accuracy of 69.67% compared to ARAE's 50.05%.
  • Text Quality: Despite stronger label enforcement, CARA improved BLEU scores. On the Yelp sentiment transfer task, CARA achieved a BLEU score of 37.23, nearly double that of the original ARAE (20.18).
  • State-of-the-Art: CARA achieved SOTA results on the Yelp attribute transfer task, outperforming specialized models like StyleTransformer and DeleteAndRetrieve.

Conclusion

CARA demonstrates that effective label-conditional generation requires balancing latent disentanglement with explicit output-space supervision. By preventing the generator from taking "shortcuts," CARA ensures high-quality text that remains strictly faithful to its intended attributes. Future work will explore multi-label conditioning and applications in other domains like image-to-image translation.

Summary updated

Abstract

Learning to generate text with a given label is a challenging task because natural language sentences are highly variable and ambiguous. It renders difficulties in trade-off between sentence quality and label fidelity. In this paper, we present CARA to alleviate the issue, where two auxiliary classifiers work simultaneously to ensure that (1) the encoder learns disentangled features and (2) the generator produces label-related sentences. Two practical techniques are further proposed to improve the performance, including annealing the learning signal from the auxiliary classifier, and enhancing the encoder with pre-trained language models. To establish a comprehensive benchmark fostering future research, we consider a suite of four datasets, and systematically reproduce three representative methods. CARA shows consistent improvement over the previous methods on the task of label-conditional text generation, and achieves state-of-the-art on the task of attribute transfer.
BibTeX
@inproceedings{li2020complementary,
  author = {Yuan Li and Chunyuan Li and Yizhe Zhang and Xiujun Li and Guoqing Zheng and Lawrence Carin and Jianfeng Gao},
  title = {{Complementary Auxiliary Classifiers for Label-Conditional Text Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=9CuQtC6NJ},
}
Figure