Abstract
Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer.
Introduction
Generating high-quality text using implicit models like Generative Adversarial Networks (GANs) has historically been a significant challenge. The discrete nature of text prevents direct backpropagation from a discriminator through a generator. While techniques like Reinforcement Learning (REINFORCE) or the Gumbel-softmax trick offer workarounds, they are frequently plagued by training instability, mode collapse, and a heavy reliance on Maximum Likelihood Estimation (MLE) pre-training.
Furthermore, the adversarial minimax game in GANs is inherently unstable. If a discriminator learns to distinguish real from synthetic data too quickly, the generator's learning signal vanishes. SeqGFMN addresses these issues by introducing a "discriminator-free" approach that matches the feature statistics of generated text to those of real data using fixed, pre-trained models like BERT.
Methodology
The proposed SeqGFMN (Sequence Generative Feature Matching Network) replaces the learned discriminator with a fixed, pre-trained feature extractor. This transformation converts a volatile adversarial game into a straightforward minimization problem.
The core mechanism is Moment Matching:
- Feature Extraction: Real text sequences are passed through a pre-trained model (e.g., BERT, GloVe) to compute the mean and variance of hidden representations at each token position.
- Soft Generation: The generator produces a sequence of probability distributions (soft one-hot representations) over the vocabulary.
- Statistical Alignment: These "soft" tokens are passed through the same fixed feature extractor. The generator is then trained to minimize the $L_2$ distance between the statistics (mean/variance) of its generated features and those of the real data.
Because the feature extractor is fixed and the generator outputs differentiable "soft" embeddings, the entire pipeline can be optimized using standard stochastic gradient descent without reinforcement learning.
Experimental Setup
The effectiveness of SeqGFMN was evaluated across three tasks:
- Unconditional Text Generation: Evaluated on MSCOCO (image captions) and EMNLP 2017 WMT News.
- Class-Conditional Generation: Tested on Yelp Reviews to generate positive vs. negative sentiment text.
- Unsupervised Text Style Transfer: Also utilizing Yelp Reviews to transform the sentiment of existing text without paired examples.
Key Results and Discussion
SeqGFMN demonstrates superior stability and performance compared to traditional adversarial text models. A critical finding is that SeqGFMN does not require MLE pre-training; it can converge to meaningful linguistic structures starting from random weights.
As shown in the results above, SeqGFMN using BERT features significantly outperformed traditional GANs. On the COCO dataset, it achieved a BLEU-4 score of 0.277, compared to SeqGAN's 0.012. The quality of the feature extractor proved paramount: context-aware models like BERT provided a much stronger training signal than non-contextual embeddings like GloVe or FastText.
In style transfer tasks, SeqGFMN combined with back-translation (BT) achieved a 96.4% sentiment transfer accuracy on Yelp reviews, outperforming complex adversarial baselines. Additionally, the model's latent space appears well-structured; interpolating between noise vectors produces sentences that transition smoothly in meaning, suggesting the model learns a genuine map of language rather than simple memorization.
Conclusion
SeqGFMN provides a stable and effective alternative to GANs for text generation by leveraging the rich representations of pre-trained models. While the approach is currently limited by its dependency on the domain coverage of the pre-trained feature extractor and a fixed sequence length, it eliminates the need for complex RL-based training. Future research may focus on adapting the architecture for variable-length sequences and exploring even more diverse feature extraction sources.
Summary updated