OptiGAN: Generative Adversarial Networks for Goal Optimized Sequence Generation

Mahmoud Hossam, Trung Le, Viet Huynh, Michael Papasimeon, Dinh Phung

TL;DR

This paper introduces OptiGAN, a novel framework that combines Generative Adversarial Networks (GANs) with Reinforcement Learning (RL) to generate sequences that not only mimic real data but also optimize for specific desired goals. OptiGAN addresses the limitations of existing models by achieving high goal scores without sacrificing output diversity, demonstrating its effectiveness in both text and trajectory generation tasks.

What Problem Does This Paper Address?

Generating realistic sequences is a cornerstone of many AI applications, from text generation and drug design to robotics and music synthesis. While Generative Adversarial Networks (GANs) have shown great promise in mimicking data distributions, they often struggle with optimizing generated outputs for specific, task-relevant properties or goals. Current models primarily focus on "resembling" the training data, which is insufficient when applications require sequences with particular attributes (e.g., solubility of a drug molecule, specific sentiment in text, or tactical quality in an air combat maneuver). This gap limits the practical utility of generative models in real-world scenarios where goal-oriented generation is crucial.

Methods

OptiGAN proposes a principled integration of GANs and Reinforcement Learning (RL) using policy gradients. The framework starts with a standard autoregressive generative model, enhanced with a GAN component to improve data mimicry and combat mode collapse. To address the goal optimization challenge, policy gradient RL is incorporated.

Key methodological innovations include:

  • Hybrid GAN and Maximum Likelihood: To mitigate GANs' notorious mode-collapse issue and ensure diversity, OptiGAN combines a GAN objective with a maximum likelihood objective. This approach theoretically ensures that both Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences between real and generated distributions are simultaneously maximized, leading to better diversity.
  • Policy Gradient RL for Goal Optimization: The RL component, specifically policy gradients, is used to directly optimize a desired goal score. The generator acts as the RL agent, learning a policy to produce sequences that maximize a reward signal derived from the specific goal metric.
  • Variance Reduction Techniques: To stabilize training, which is often problematic with policy gradients due to high variance, OptiGAN employs two techniques:
  • Monte Carlo Rollout: Similar to existing methods, but with a modification to directly compute rewards based on the chosen score.
  • Policy Gradient with Baseline: A fixed baseline value (average computed rewards) is used to reduce the variance of the policy gradient estimates.

The overall training involves alternating updates to the discriminator (for GAN training) and the generator (which optimizes both the GAN loss and the RL loss).

Datasets

The paper evaluates OptiGAN on two distinct types of sequential data:

  • Text Generation:
    • MS-COCO image captions dataset: Used for training and evaluating text generation.
  • EMNLP2017 WMT News dataset: Another dataset for text generation evaluation.
  • Real-valued Sequence Generation:
    • ACE-Zero simulator [37]: A flight simulator used to generate air combat trajectories for training and evaluation. The data specifically focuses on the "Stern Conversion" maneuver.

No new benchmarks were introduced; instead, existing datasets were used to demonstrate the model's versatility.

Key Insights and Ideas

  • GANs Alone Aren't Enough for Goal Optimization: Pure RL, while capable of optimizing for a score, can lead to unrealistic or structurally unsound sequences (as seen in Table IV with repetitive n-grams). Conversely, GANs alone focus on data mimicry and don't inherently optimize for external goals.
  • The Power of Hybridization: Combining GANs (for realism and diversity) with RL (for goal optimization) is crucial. OptiGAN demonstrates that this synergy allows for generating sequences that are both high-quality and goal-achieving.
  • Diversity is Not Sacrificed: A key finding is that OptiGAN successfully optimizes for desired scores without compromising the diversity of generated outputs, a common pitfall for GANs. The hybrid maximum likelihood GAN component is credited for this.
  • RL Can "Cheat" Without GAN Guidance: When pure RL is used to maximize a score like BLEU, the model can learn to exploit the metric by generating repetitive or nonsensical sequences, rather than truly high-quality ones. The GAN component acts as a regularizer, keeping the generation process grounded in realistic data distributions.

Key Results

  • Text Generation: OptiGAN achieved competitive or superior BLEU scores compared to baselines like SeqGAN and OptiGAN-OnlyRL on both MS-COCO and EMNLP News datasets. Crucially, it maintained competitive Negative Log-Likelihood (NLL) values, indicating good output diversity, unlike SeqGAN which suffered from poor NLL.
  • Air Combat Trajectory Generation: OptiGAN significantly outperformed baselines (LSTM, OptiGAN-OnlyGAN) in terms of the McGrew score, a metric for tactical quality in air combat maneuvers. It achieved scores closest to those generated by the ACE-Zero simulator itself, demonstrating effective goal optimization for trajectory planning.
  • Qualitative Results: Generated text samples were described as "meaningful, structured and diverse." Trajectory samples showed the blue aircraft successfully maneuvering to get behind the red aircraft, as intended by the "Stern Conversion" maneuver.

Limitations and Future Work

  • Limitations:
    • The paper notes that pure RL can lead to unrealistic repetitions and shorter sentence lengths in text generation when not guided by a GAN component.
  • While OptiGAN-OnlyRL achieved high BLEU scores, it exhibited very high variance.
  • Future Work:
    • Improve the realism and physical constraints of real-valued outputs like trajectories.
  • Incorporate latent space modeling for discrete data generation, enabling guidance along learned disentangled features.
  • The authors suggest OptiGAN's framework is general and can be applied to any GAN model for goal optimization.

Summary updated

Abstract

One of the challenging problems in sequence generation tasks is the optimized generation of sequences with specific desired goals. Current sequential generative models mainly generate sequences to closely mimic the training data, without direct optimization of desired goals or properties specific to the task. We introduce OptiGAN, a generative model that incorporates both Generative Adversarial Networks (GAN) and Reinforcement Learning (RL) to optimize desired goal scores using policy gradients. We apply our model to text and real-valued sequence generation, where our model is able to achieve higher desired scores out-performing GAN and RL baselines, while not sacrificing output sample diversity.
BibTeX
@inproceedings{hossam2020optigan,
  author = {Mahmoud Hossam and Trung Le and Viet Huynh and Michael Papasimeon and Dinh Phung},
  title = {{OptiGAN: Generative Adversarial Networks for Goal Optimized Sequence Generation}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=95eqrT_AuA2},
}
Figure