InitialGAN: A Language GAN With Completely Random Initialization
Lists
Only you see this
Abstract
Text generative models trained via maximum likelihood estimation (MLE) suffer from the notorious exposure bias problem, and generative adversarial networks (GANs) are shown to have potential to tackle this problem. The existing language GANs adopt estimators, such as REINFORCE or continuous relaxations to model word probabilities. The inherent limitations of such estimators lead current models to rely on pretraining techniques (MLE pretraining or pretrained embeddings). Representation modeling methods (RMMs), which are free from those limitations, however, are seldomly explored because of their poor performance in previous attempts. Our analyses reveal that invalid sampling methods and unhealthy gradients are the main contributors to such unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized long short-term memory network (LSTM). Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized in full. Besides, we introduce a new evaluation metric, least coverage rate (LCR), to better evaluate the quality of generated samples. The experimental results demonstrate that the InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without using any pretraining techniques.
BibTeX
@article{ren2024initialgan,
author = {Da Ren and Qing Li},
title = {{InitialGAN: A Language GAN With Completely Random Initialization}},
journal = {IEEE Trans. Neural Networks Learn. Syst.},
year = {2024},
doi = {10.1109/tnnls.2023.3315778},
}| Training Language GANs from Scratch | NeurIPS · 2019 |
| Language Generation with Recurrent Generative Adversarial Networks without Pre-training | arXiv · 2017 |
| End-to-End Differentiable GANs for Text Generation | ICBINB @ NeurIPS · 2020 |
| ColdGANs: Taming Language GANs with Cautious Sampling Strategies | arXiv · 2020 |
| Making Use of Latent Space in Language GANs for Generating Diverse Text without Pre-training | Student Research Workshop @ EACL · 2021 |
| Maximum-Likelihood Augmented Discrete Generative Adversarial Networks | arXiv · 2017 |