Learning Implicit Text Generation via Feature Matching

Inkit Padhi, Pierre L. Dognin, Ke Bai, Cícero Nogueira dos Santos, Vijil Chenthamarakshan, Youssef Mroueh, Payel Das

Abstract

Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer.

Introduction

Generating high-quality text using implicit models like Generative Adversarial Networks (GANs) has historically been a significant challenge. The discrete nature of text prevents direct backpropagation from a discriminator through a generator. While techniques like Reinforcement Learning (REINFORCE) or the Gumbel-softmax trick offer workarounds, they are frequently plagued by training instability, mode collapse, and a heavy reliance on Maximum Likelihood Estimation (MLE) pre-training.

Furthermore, the adversarial minimax game in GANs is inherently unstable. If a discriminator learns to distinguish real from synthetic data too quickly, the generator's learning signal vanishes. SeqGFMN addresses these issues by introducing a "discriminator-free" approach that matches the feature statistics of generated text to those of real data using fixed, pre-trained models like BERT.

Methodology

The proposed SeqGFMN (Sequence Generative Feature Matching Network) replaces the learned discriminator with a fixed, pre-trained feature extractor. This transformation converts a volatile adversarial game into a straightforward minimization problem.

The SeqGFMN architecture: A generator produces sequences which are then passed through a fixed feature extractor to calculate a moment-matching loss.

The core mechanism is Moment Matching:

  1. Feature Extraction: Real text sequences are passed through a pre-trained model (e.g., BERT, GloVe) to compute the mean and variance of hidden representations at each token position.
  2. Soft Generation: The generator produces a sequence of probability distributions (soft one-hot representations) over the vocabulary.
  3. Statistical Alignment: These "soft" tokens are passed through the same fixed feature extractor. The generator is then trained to minimize the $L_2$ distance between the statistics (mean/variance) of its generated features and those of the real data.

Because the feature extractor is fixed and the generator outputs differentiable "soft" embeddings, the entire pipeline can be optimized using standard stochastic gradient descent without reinforcement learning.

Experimental Setup

The effectiveness of SeqGFMN was evaluated across three tasks:

  • Unconditional Text Generation: Evaluated on MSCOCO (image captions) and EMNLP 2017 WMT News.
  • Class-Conditional Generation: Tested on Yelp Reviews to generate positive vs. negative sentiment text.
  • Unsupervised Text Style Transfer: Also utilizing Yelp Reviews to transform the sentiment of existing text without paired examples.

Key Results and Discussion

SeqGFMN demonstrates superior stability and performance compared to traditional adversarial text models. A critical finding is that SeqGFMN does not require MLE pre-training; it can converge to meaningful linguistic structures starting from random weights.

Performance comparison of SeqGFMN against various GAN baselines on COCO and WMT News datasets.

As shown in the results above, SeqGFMN using BERT features significantly outperformed traditional GANs. On the COCO dataset, it achieved a BLEU-4 score of 0.277, compared to SeqGAN's 0.012. The quality of the feature extractor proved paramount: context-aware models like BERT provided a much stronger training signal than non-contextual embeddings like GloVe or FastText.

In style transfer tasks, SeqGFMN combined with back-translation (BT) achieved a 96.4% sentiment transfer accuracy on Yelp reviews, outperforming complex adversarial baselines. Additionally, the model's latent space appears well-structured; interpolating between noise vectors produces sentences that transition smoothly in meaning, suggesting the model learns a genuine map of language rather than simple memorization.

Conclusion

SeqGFMN provides a stable and effective alternative to GANs for text generation by leveraging the rich representations of pre-trained models. While the approach is currently limited by its dependency on the domain coverage of the pre-trained feature extractor and a fixed sequence length, it eliminates the need for complex RL-based training. Future research may focus on adapting the architecture for variable-length sequences and exploring even more diverse feature extraction sources.

Summary updated

Abstract

Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer.
BibTeX
@inproceedings{padhi2020learning,
  author = {Inkit Padhi and Pierre L. Dognin and Ke Bai and Cícero Nogueira dos Santos and Vijil Chenthamarakshan and Youssef Mroueh and Payel Das},
  title = {{Learning Implicit Text Generation via Feature Matching}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.354},
}
Figure