Improving Language Generation with Sentence Coherence Objective

Ruixiao Sun, Jie Yang, M. YousefzadeharXiv 2020 · DOI · arXiv

Abstract

Conditional story generation and contextual text continuation have become increasingly popular topics in the NLP community. Existing models are often prone to output paragraphs of text that gradually diverge from the given prompt. Although the generated text may have reasonable perplexity and diversity, it can easily be identified by humans as incoherent. This project aims to improve the coherence and consistency across sentences in a language-generation model. We first train a sentence pair coherence classifier using a GPT-2 pretrained model and then co-train the GPT-2 language model with this new coherence objective using a method analogous to the REINFORCE algorithm. This fine-tuned language model is able to generate lengthy paragraphs conditioned on a given topic without significant divergence. The simplicity of this model allows it to be applicable to a variety of underlying language model architectures since it only modifies the final layer of the pre-trained model.

Introduction

While large language models (LLMs) like GPT-2 produce remarkably human-like text, they often suffer from a lack of "global" intent. In conditional story generation, a user provides a prompt (e.g., "The new PlayStation 5 is releasing soon"), and the model continues the text. However, after a few sentences, vanilla GPT-2 often loses the thread. It might pivot to a specific TV show or a random historical event simply because the local word-to-word transitions seem plausible.

This phenomenon is a failure of coherence. Standard language models are trained to predict the next word based on immediate context, but they aren't explicitly optimized to ensure the entire paragraph relates back to the original prompt. This research bridges that gap by making the prompt a constant "guiding star" for the generation process.

Methodology

The proposed approach consists of a two-stage pipeline designed to enforce topical consistency.

1. The Coherence Classifier

First, a GPT-2 model is fine-tuned to act as a judge. Given two sentences, the classifier predicts whether they belong in the same paragraph. It utilizes a [CLS] token to create sentence embeddings ($e_1, e_2$) and a double dot-product regression model to calculate a coherence score.

Architecture of the sentence coherence classifier using GPT-2 embeddings.

2. Co-training with REINFORCE

Once a reliable "judge" is established, the text generation model is modified. A feed-forward neural network is added to the final layer of GPT-2. This layer takes the concatenation of the current hidden state and the original prompt’s embedding as input.

The generation model architecture featuring prompt embedding injection into the final layers.

3. Objective Function and Training

The model is trained on a dual objective: minimizing standard cross-entropy loss (for fluency) while maximizing a "similarity reward" (for coherence). Because sampling words is non-differentiable, the REINFORCE algorithm—a policy gradient method—is used to update weights based on the coherence score of the generated paragraph. The researchers utilized the WikiText-2 dataset, segmenting the corpus into 200-token chunks for training.

Key Insights and Results

  • The "Shortcut" Architecture: Explicitly injecting the prompt's embedding into every hidden state calculation before the final vocabulary projection acts as a persistent guidance signal.
  • Non-linearity is Essential: Experiments showed that a simple linear projection led to repetitive, "degenerated" text. Incorporating a non-linear activation function (tanh) in the feed-forward layer was crucial for maintaining variety and quality.
  • High-Fidelity Judging: The coherence classifier achieved 94.21% accuracy, outperforming the human baseline of 92%. This provided a robust reward signal for the reinforcement learning phase.
  • Qualitative Success: In side-by-side comparisons, vanilla GPT-2 drifted from a PlayStation 5 prompt into a tangent about The X-Files. In contrast, the proposed model stayed focused on gaming and Sony throughout the entire passage.
  • Convergence: Despite the noise inherent in REINFORCE gradients, the model showed a downward trend in coherence loss, with perplexity stabilizing at approximately 15.

Limitations and Future Work

A primary limitation is the fluency trade-off. Because the model is heavily incentivized to stay on topic, it can occasionally produce repetitive or slightly stiff prose compared to the "creative" freedom of vanilla GPT-2. Future work will focus on refining the balance between the language modeling objective and the coherence objective. As the framework is architecture-agnostic, it could potentially be applied to larger models like GPT-3 to improve long-form storytelling.

Summary updated

Abstract

Conditional story generation and contextual text continuation have become increasingly popular topics in NLP community. Existing models are often prone to output paragraphs of texts that gradually diverge from the given prompt. Although the generated text may have a reasonable perplexity and diversity, it could easily be identified by human as gibberish. The goal of our project is to improve the coherence and consistency across sentences in a language-generation model. We aim to solve this issue by first training a sentence pair coherence classifier with GPT-2 pretrained model, and then co-train the GPT-2 language model with this new coherence objective using a method analogous to the REINFORCE algorithm. This fine-tuned language model is able to generate lengthy paragraph conditioned on a given topic without diverging too much. The simplicity of this model allows it to be applicable to a variety of underlying language model architecture since it only modifies the final layer of the pre-trained model.
BibTeX
@misc{sun2020improving,
  author = {Ruixiao Sun and Jie Yang and M. Yousefzadeh},
  title = {{Improving Language Generation with Sentence Coherence Objective}},
  year = {2020},
  eprint = {2009.06358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.06358},
}
Figure