Evaluating and Explaining Natural Language Generation with GenX

Kayla Duskin, Shivam Sharma, Ji Young Yun, Emily Saldanha, Dustin ArendtNAACL 2021

TL;DR

This paper introduces GenX, an interactive tool designed to help data scientists evaluate natural language generation (NLG) models by detecting memorization of training data. GenX goes beyond traditional metrics by visualizing sentence-level similarities to training data, enabling a deeper understanding of model creativity and potential biases.

What Problem Does This Paper Address?

The rapid advancement of Natural Language Generation (NLG) models, like GPT-3, has led to outputs that are often indistinguishable from human-written text. However, current evaluation methods primarily focus on text quality (fluency, realism) and often overlook two critical aspects: model creativity and the extent to which models memorize their training data. A model that simply reproduces large chunks of its training data might appear high-quality but lacks genuine generative ability. This memorization can also lead to inflated performance metrics, especially when training and testing datasets have overlapping content. Detecting this memorization, particularly partial or sub-document duplication, is challenging for data scientists who may not be intimately familiar with the training corpus.

Methods

The core of this paper is the GenX tool, implemented as a Jupyter Notebook widget. GenX operates by:

  • Sentence Embeddings: It takes raw text and pre-computed sentence-level embeddings (e.g., from Sentence-BERT) for both training and testing data.
  • Nearest Neighbor Search: For each sentence in the test set, GenX finds its 10 nearest neighbors in the training set based on cosine distance.
  • Visualizing Similarity:
    • Sentence-Level Highlighting: GenX uses dynamic time warping and Levenshtein distance to identify and highlight verbatim (strong underline) or near-verbatim (weaker underline) matches within test sentences against their nearest training neighbors.
  • Nearest Neighbor Bar Charts: For each test sentence, a bar chart visualizes the distances to its nearest neighbors, sorted by distance, providing an immediate sense of how unique or similar the sentence is to the training data.
  • Document Navigation: A step line chart visualizes the source document of the nearest neighbor for each sentence in a generated document, helping to identify if repetitions occur within the same source document.
  • Novel Metrics: To quantify memorization at a document level, GenX introduces two metrics:
  • Distinctiveness: The average distance of the first nearest neighbor to each test sentence in a document. Low distinctiveness suggests sentences are similar to training data.
  • Diversity: The number of unique source documents for the nearest neighbors of sentences in a test document. Low diversity indicates similarity to a single source document.

Datasets

The paper uses two datasets for demonstration:

  • Phishing Emails: A composite dataset initially containing 60,705 emails, which was de-duplicated to 9,234 emails for training/testing. This dataset highlights issues with test set contamination and formulaic text.
  • ACL Abstracts: 17,903 abstracts from the ACL anthology were used as the training data, with 2,000 for validation and 2,000 for testing. This dataset is used to demonstrate models with high creativity and low memorization.

Key Insights and Ideas

  • Memorization vs. Creativity Trade-off: The research strongly suggests a trade-off: models that memorize heavily often produce coherent text, while models that are more creative (less memorization) may struggle with coherence, especially when using sub-optimal parameters.
  • Limitations of Perplexity: Perplexity alone is insufficient for evaluating NLG models. The ACL abstract generation case shows a model with higher perplexity producing more coherent and creative outputs than a model with lower perplexity.
  • Importance of Interactive Tools: Traditional aggregate metrics fail to capture nuanced behaviors. Interactive tools like GenX are crucial for qualitative analysis and understanding why a model behaves a certain way.
  • Test Set Contamination is Pervasive: The phishing dataset analysis reveals significant test set contamination, where identical or near-identical sentences exist in both training and testing splits, artificially inflating performance metrics.
  • Formulaic Nature of Phishing: Phishing emails, due to their nature, often contain repeated phrases and sentences, making them a challenging domain for NLG models and a good test case for memorization detection.

Key Results

  • GenX successfully identified significant test set contamination in the phishing dataset, revealing that many generated emails were near-verbatim copies of training data.
  • In the phishing use case, models trained for more epochs showed increased memorization.
  • For the ACL abstract generation, GenX demonstrated that models achieved low memorization and high coherence, producing novel and creative outputs.
  • The distinctiveness and diversity metrics, visualized through scatter plots, effectively distinguished between models with high memorization and those with high creativity.

Limitations and Future Work

  • Scalability: GenX's nearest neighbor approach becomes computationally intractable for massive datasets like CommonCrawl. Future work aims to incorporate approximate neighbor techniques.
  • Diversity Metric Refinement: In the phishing use case, identical sentences across multiple training documents artificially inflated diversity. The authors plan to use a minimum set cover algorithm to improve diversity score accuracy.
  • Broader Impact: While the tool can help detect memorization, the authors acknowledge the ethical considerations of conditioning NLG models on malicious content like phishing emails. They do not plan to release phishing-specific models or code.

Summary updated

BibTeX
@inproceedings{duskin2021evaluating,
  author = {Kayla Duskin and Shivam Sharma and Ji Young Yun and Emily Saldanha and Dustin Arendt},
  title = {{Evaluating and Explaining Natural Language Generation with GenX}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure