TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks

Md. Akmal Haidar, Mehdi RezagholizadehCanadian ai 2019 · DOI

TL;DR

This paper introduces TextKD-GAN, a novel approach to text generation using Generative Adversarial Networks (GANs) by incorporating knowledge distillation. By using an autoencoder to provide a smooth, continuous representation of text instead of discrete one-hot encodings, TextKD-GAN makes the discriminator's task harder, leading to improved generator performance and more coherent text generation.

What Problem Does This Paper Address?

Text generation, a crucial task in Natural Language Processing (NLP) with applications like machine translation and summarization, has traditionally relied on Recurrent Neural Networks (RNNs) trained with Maximum Likelihood Estimation (MLE). However, MLE-based methods suffer from "exposure bias," where models are trained on ground truth but tested on their own potentially erroneous predictions. Generative Adversarial Networks (GANs) have shown great success in image generation, but their application to text generation is challenging due to the discrete nature of words. Standard GANs struggle to backpropagate gradients through discrete outputs, leading to difficulties in training. Existing GAN-based text generation methods often resort to techniques like reinforcement learning or latent space representations, which can be computationally expensive or require pre-training. This paper aims to bridge this gap by proposing a GAN-based text generation method that effectively handles the discrete nature of text without requiring pre-training.

Methods

The core innovation of TextKD-GAN lies in its use of knowledge distillation combined with an autoencoder (AE) to create a smoother representation of text.

  • Autoencoder (Teacher): An AE is trained to encode input sentences into a latent space and then reconstruct them. The AE's reconstructed output, which is a "softened" representation (assigning non-zero probabilities to multiple words), serves as a continuous representation of the original text.
  • Knowledge Distillation: This softened representation from the AE is then "distilled" to train the GAN. Instead of feeding the discriminator discrete one-hot vectors for real text, TextKD-GAN feeds it the continuous, smooth representations from the AE.
  • GAN Architecture (Student): The GAN generator (the "student") is trained to mimic this softened representation from the AE. The discriminator is trained to distinguish between the AE's softened outputs and the generator's outputs. This setup makes the discrimination task more challenging for the discriminator, providing a richer signal to the generator.
  • Training: The AE and the GAN are trained simultaneously. The training objective includes a reconstruction loss for the AE, a gradient penalty for the discriminator (borrowed from improved Wasserstein GANs), and an adversarial loss for the generator.

Datasets

The experiments were conducted on two datasets:

  • Google 1 Billion Benchmark Language Modeling Data: The first 1 million sentences were used, with the most frequent 100 characters forming the vocabulary.
  • Stanford Natural Language Inference (SNLI) Corpus: The entire preprocessed training data (714,667 sentences) was used, with a vocabulary of 86 characters.

Key Insights and Ideas

  • Smooth Representations for Discrete Data: The key insight is that by using a softened, continuous representation (from an AE) instead of discrete one-hot encodings, the GAN's discriminator faces a more challenging task. This difficulty forces the generator to learn a more robust and nuanced data distribution.
  • Knowledge Distillation as a Bridge: Knowledge distillation acts as a crucial bridge, allowing the GAN to learn from the rich, continuous representations generated by the AE, effectively bypassing the direct challenges of discrete text generation.
  • Improved Discriminator Signal: The harder discrimination task leads to a more powerful discriminator, which in turn provides a better learning signal for the generator, resulting in higher quality generated text.
  • Character-Level Focus: The paper focuses on character-level text generation, which simplifies some aspects of vocabulary handling but still presents the core challenge of discrete outputs.

Key Results

The proposed TextKD-GAN approach demonstrated significant improvements over existing GAN-based text generation methods like IWGAN and ARAE, particularly on the SNLI dataset.

  • BLEU Scores: TextKD-GAN achieved higher BLEU-2, BLEU-3, and BLEU-4 scores compared to IWGAN and ARAE on both datasets. For instance, on the SNLI dataset, TextKD-GAN achieved BLEU-2 of 0.62, BLEU-3 of 0.50, and BLEU-4 of 0.38, outperforming IWGAN (0.57, 0.44, 0.30) and ARAE (0.37, 0.27, 0.17).
  • Jensen-Shannon Distance (JSD): The training curves showed that TextKD-GAN further minimized JSD between generated and real sentences compared to other methods, indicating that the generated data distribution is closer to the real data distribution.
  • Qualitative Results: Example generated sentences from the SNLI experiment showed that TextKD-GAN produced more meaningful and coherent text compared to IWGAN.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Character-Level vs. Word-Level: The current evaluation is primarily at the character level. Investigating the performance of TextKD-GAN at the word level is a key future direction.
  • Generator Architecture: The current implementation uses a CNN-based generator. Exploring RNN-based generators could potentially improve performance.
  • Long Sentence Generation: Similar to other pure GAN-based text generation techniques, TextKD-GAN struggles with generating very long sentences. The authors suggest that incorporating Reinforcement Learning (RL) could help address this weakness.
  • Capacity for Semantics: Pure GAN-based text generation techniques, including TextKD-GAN, are still in their early stages and may lack the capacity to fully capture the semantics of complex datasets and long sentences, possibly due to limitations of CNNs in capturing long-term dependencies.

Summary updated

BibTeX
@inproceedings{haidar2019textkd,
  author = {Md. Akmal Haidar and Mehdi Rezagholizadeh},
  title = {{TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks}},
  booktitle = {Canadian ai},
  year = {2019},
  doi = {10.1007/978-3-030-18305-9\_9},
}
Figure