Abstract
Sequence generative adversarial networks (SeqGAN) have been used to improve conditional sequence generation tasks, for example, chit-chat dialogue generation. To stabilize the training of SeqGAN, Monte Carlo tree search (MCTS) or reward at every generation step (REGS) is used to evaluate the goodness of a generated subsequence. MCTS is computationally intensive, but the performance of REGS is worse than MCTS. In this paper, we propose stepwise GAN (StepGAN), in which the discriminator is modified to automatically assign scores quantifying the goodness of each subsequence at every generation step. StepGAN has significantly less computational costs than MCTS. We demonstrate that StepGAN outperforms previous GAN-based methods on both synthetic experiment and chit-chat dialogue generation.
Introduction
Training Generative Adversarial Networks (GANs) for discrete sequences, such as text, is notoriously difficult due to the sparse reward problem. In standard SeqGAN frameworks, the discriminator typically provides a single scalar score only after an entire sequence is generated. This lack of intermediate feedback makes the training signal weak and unstable, often leading models to produce safe but generic responses (e.g., "I don't know").
To address this, researchers have previously employed Monte Carlo Tree Search (MCTS) to estimate the value of partial sequences by rolling out multiple future completions. While effective, MCTS is computationally prohibitive. Alternative approaches like Reward at Every Generation Step (REGS) offer speed but often sacrifice performance. StepGAN bridges this gap by providing high-quality, stepwise rewards with nearly zero extra computational cost.
Methodology
The core innovation of StepGAN lies in its redefined interaction between the discriminator and the generator, moving away from holistic evaluation toward incremental assessment.
Stepwise Discriminator
Unlike traditional discriminators that output a single final score, the StepGAN discriminator is modified to output an estimated Q-value (state-action value) $\hat{Q}(s_t, y_t)$ at every time step $t$. The final discriminator score $D(x, y)$ is calculated as the average of these stepwise scores. This allows the model to receive immediate feedback on each token generated, approximating the expected return of the full sequence without expensive roll-outs.
Policy Gradient and Stability
The generator is updated via policy gradients. To stabilize training and reduce variance, the authors incorporate a Value Network ($V$) that predicts the expected reward for a given state, serving as a baseline. Furthermore, the authors introduce Weighted StepGAN (StepGAN-W), which applies decaying weights to rewards, emphasizing the importance of earlier tokens in determining sequence quality.
Key Insights and Results
The "One-to-Many" Property and Critical Words
Dialogue is inherently non-deterministic; a single prompt can have many valid responses. GANs are particularly well-suited for this "one-to-many" mapping compared to Maximum Likelihood Estimation (MLE), which tends to converge on a single "ground truth" and produce boring, repetitive text.
A key finding is that StepGAN naturally learns to focus on "critical words" that define the semantic quality of a response. For instance, in a response like "I'm John," the discriminator assigns higher importance to the name "John" than to the generic prefix "I'm."
Performance and Efficiency
- Quantitative Success: In synthetic tasks, StepGAN-W achieved a precision of 93.04% and the lowest combined KL divergence score (6.973), outperforming MLE, SeqGAN, and MCTS.
- Human Evaluation: In dialogue tasks, StepGAN-W significantly outperformed MLE in Semantics Human Score (60.4 vs 42.6), producing more diverse and semantically rich responses.
- Computational Efficiency: StepGAN is approximately 5x faster than MCTS-based methods and nearly as fast as vanilla SeqGAN, making it feasible for large-scale training.
Conclusion
StepGAN successfully addresses the sparse reward problem in sequence GANs by providing efficient stepwise evaluation. While it significantly improves the informativeness and diversity of generated text, a trade-off remains regarding logical coherence compared to MLE. Future work will explore applying StepGAN to other conditional sequence tasks such as machine translation, abstractive summarization, and video captioning.
Summary updated