Eval all, trust a few, do wrong to none: Comparing sentence generation models

Ondrej Cífka, Aliaksei Severyn, Enrique Alfonseca, Katja FilippovaarXiv 2018 · arXiv

Abstract

Evaluating neural text generation models remains a significant challenge due to inconsistent metrics and a lack of focus on the dual requirements of sampling and reconstruction. This paper establishes a rigorous evaluation framework and introduces the Fréchet InferSent Distance (FID), a text-based adaptation of the Fréchet Inception Distance used in computer vision. Our findings reveal a fundamental trade-off between a model's ability to generate fluent new sentences and its ability to accurately reconstruct existing ones. Furthermore, we demonstrate that simple regularization techniques, such as constraining autoencoder embeddings to a unit sphere, allow basic models to perform competitively with more complex state-of-the-art architectures.

Introduction

In Natural Language Generation (NLG), researchers often claim their models produce "fluent and meaningful" text, yet evaluation methodologies are frequently inconsistent. While computer vision has standardized metrics for Generative Adversarial Networks (GANs), NLG evaluation often focuses narrowly on either unconditional generation (sampling) or reconstruction, but rarely both.

This narrow focus is problematic. A model that generates high-quality random sentences might suffer from "posterior collapse," effectively ignoring its input during reconstruction. Conversely, a model with perfect reconstruction might possess a "broken" latent space that fails to generate coherent new samples. This paper addresses this gap by comparing three major generative model families and introducing a robust automatic metric for distribution matching.

Methods

We evaluate three primary generative architectures:

  1. Variational Autoencoders (VAE): Utilizing a KL-divergence penalty to smooth the latent space.
  2. Adversarial Autoencoders (AAE): Employing a discriminator to align latent representations with a prior distribution.
  3. Adversarially Regularized Autoencoders (ARAE): Learning a flexible prior rather than imposing a fixed one.

Additionally, we investigate simple regularization for standard Autoencoders (AEs), specifically constraining embeddings to a unit sphere (AE-SPH) and injecting noise during training.

To evaluate these models, we introduce the Fréchet InferSent Distance (FID). By using a pre-trained InferSent model to map sentences into a vector space, FID calculates the statistical distance between the distributions of real and generated text. A lower FID indicates that the generated text more closely matches the characteristics of the training data.

Results and The Great Trade-off

Our experiments on a corpus of 200,000 news summaries reveal a fundamental tension between sampling fluency and reconstruction accuracy.

Latent space visualizations for different models. (a) AAE-GAUSS-DET shows highly clustered regions. (b) VAE displays a smooth, continuous distribution. (c) AE-SPH and (d) AAE-SPH show intermediate structures.

As shown in the latent space visualizations above, models like the VAE (b) create a smooth, continuous space ideal for sampling but often lose the fine-grained detail required for reconstruction. In contrast, deterministic models like AAE-GAUSS-DET (a) create highly clustered spaces that reconstruct well but contain "holes" that produce gibberish when sampled.

Quantitative Analysis

The quantitative results in the table below highlight this trade-off across various metrics.

Performance metrics for sampling and reconstruction across various models.
  • Sampling Fluency: The VAE achieved the best FID (0.6562) and high human fluency scores, but suffered from poor reconstruction (BLEU 10.27, NLL 66.3).
  • Reconstruction Accuracy: The ARAE and AE-GAUSS-DET models dominated reconstruction metrics, with BLEU scores exceeding 71, but performed poorly in sampling tasks.
  • The Balanced All-Rounder: The AE-SPH (with noise $\sigma=0.1$) provided the most robust balance, maintaining a solid reconstruction BLEU (37.56) while achieving competitive sampling quality.

Discussion and Conclusion

One of the most significant insights is that simplicity often wins. The AE-SPH model, despite its architectural simplicity, matched or outperformed the more complex ARAE in several metrics. This suggests that the field may be over-engineering architectures when simple latent space regularization is sufficient.

Furthermore, the FID proved to be a viable "compass" for researchers, correlating well with human judgments of quality. We urge the community to adopt this "dual-track" evaluation—measuring both sampling and reconstruction—to ensure that generative models are learning truly useful and representative language features.

Summary updated

Abstract

In this paper, we study recent neural generative models for text generation related to variational autoencoders. Previous works have employed various techniques to control the prior distribution of the latent codes in these models, which is important for sampling performance, but little attention has been paid to reconstruction error. In our study, we follow a rigorous evaluation protocol using a large set of previously used and novel automatic and human evaluation metrics, applied to both generated samples and reconstructions. We hope that it will become the new evaluation standard when comparing neural generative models for text.
BibTeX
@misc{cifka2018eval,
  author = {Ondrej Cífka and Aliaksei Severyn and Enrique Alfonseca and Katja Filippova},
  title = {{Eval all, trust a few, do wrong to none: Comparing sentence generation models}},
  year = {2018},
  eprint = {1804.07972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.07972},
}
Figure