TL;DR
Training sequence generators with Generative Adversarial Networks (GANs) is notoriously unstable because the "reward" signal—provided by a constantly updating discriminator—is a moving target. This paper proposes Expert-based Reward Function Training, a method that replaces the volatile GAN framework with a stable, fixed reward function trained on "near-miss" sequences generated from a proposal distribution. This approach yields significantly more stable training and outperforms established baselines like SeqGAN and RankGAN on synthetic benchmarks.
The Problem: Adversarial Instability
In the realm of sequence generation (such as text or music), combining GANs with Reinforcement Learning (RL) has become a standard approach. In a typical setup like SeqGAN, a Generator creates sequences, and a Discriminator tries to distinguish them from real "expert" data. The Generator then uses the Discriminator’s feedback as a "reward" to improve itself via policy gradients.
However, this introduces a major bottleneck: instability. Because the Discriminator is trained on the Generator's ever-changing (and often initially poor) output, the reward signal constantly shifts. If the Generator is too weak, the Discriminator easily overpowers it and learns nothing useful; if the Discriminator is too strong, the Generator gets discouraged by a "perfect" critic. This adversarial dance frequently leads to training collapse.
Methods: A Non-Adversarial Alternative
The authors propose a "non-GAN" alternative that prioritizes stability by decoupling the reward function's training from the generator's performance. Their method, Expert-based Reward Function Training, follows a distinct process:
- Create a Proposal Distribution: Instead of using the Generator to create negative samples, the authors define a mathematical "proposal distribution" $q(x|x^*)$. This distribution takes a real expert sequence ($x^*$) and creates a slightly corrupted version of it. They use negative Hamming distance to determine how much to change the original sequence.
- Train a Fixed Reward Function: A reward function (such as a CNN-based classifier) is trained to distinguish between the real expert sequences and the corrupted sequences from the proposal distribution. Crucially, the Generator is not involved in this step.
- Train the Generator: Once the reward function is trained, its parameters are frozen. The Generator is then trained using Policy Gradient (RL) to maximize the score given by this fixed, stable reward function.
By decoupling the reward training from the Generator's performance, the authors ensure the Generator always has a consistent, high-quality "teacher" to learn from.
Key Insights and Ideas
- Stability is King: The core insight is that for sequence generation, an adversarial framework might be overkill or even a hindrance. A pre-trained, static reward function provides a smoother, more reliable gradient for the Generator to follow.
- The "Goldilocks" Corruptions: The hyperparameter $\tau$ (tau) controls the "smoothness" of the proposal distribution. If the corrupted sequences are too similar to the real ones, the reward function cannot distinguish them; if they are too different, the reward signal doesn't help the Generator learn the fine nuances of the data. As shown in the distribution charts, varying $\tau$ significantly alters the balance of the generated classes. The authors found that $\tau = 1.0$ provided the optimal balance for training.
- Density Ratio Estimation: Mathematically, the reward function is essentially estimating the density ratio between the expert distribution and the proposal distribution. This provides a principled, mathematically grounded signal for the Generator to move toward the expert data.
Datasets and Key Results
The paper evaluates the method using a standard Synthetic Data Benchmark (the "Oracle" test). A random Recurrent Neural Network (the "Oracle RNN") is used as the ground truth to generate a dataset of sequences (length 20, vocabulary size 5,000). Models are evaluated using Oracle Negative Log-Likelihood (NLL), which measures how closely the trained Generator's distribution matches the original Oracle RNN's distribution.
The proposed model demonstrated significant improvements over traditional GAN-based sequence generators:
| Model | Oracle NLL (Lower is better) |
|---|---|
| MLE (Standard training) | 9.03 |
| SeqGAN | 8.61 |
| RankGAN | 8.25 |
| Our Model (Expert-based) | 7.55 |
| LeakGAN | 7.04 |
While LeakGAN still holds the top spot, the Expert-based method significantly outperformed the widely used SeqGAN and the strong RankGAN baseline. This proves that superior results can be achieved without relying on a fragile adversarial framework.
Limitations and Future Work
- Synthetic vs. Real World: The paper focuses exclusively on synthetic data. While the Oracle test is a standard sanity check, real-world language has semantic complexities that simple Hamming distance (which only looks at character/word swaps) might fail to capture.
- Fixed Reward Limitations: Because the reward function is fixed before the Generator starts training, it cannot adapt if the Generator finds a way to "game" the reward function—a common problem in RL known as reward hacking.
- Future Directions: The authors suggest that exploring different types of proposal distributions or prior reward functions (beyond Hamming distance) could further improve the quality of the learned reward signal for more complex, real-world tasks.
Summary updated