TL;DR
This paper tackles the challenge of generating long, coherent, and cohesive text by introducing two novel neural discriminators that specifically evaluate paragraph-level coherence and sentence-level cohesion. They then leverage these discriminators within a reinforcement learning framework, "negative-critical sequence training," to fine-tune a pre-trained language model, significantly improving the quality of generated text.
What Problem Does This Paper Address?
Generating natural-sounding, long-form text is a notoriously difficult task for neural language models. While models can produce grammatically correct sentences, they often struggle to maintain a consistent thread of meaning (coherence) and logical flow between adjacent sentences (cohesion) over extended passages. This leads to texts that can feel disjointed, repetitive, or even contradictory, making them easily distinguishable from human-written content. Existing approaches often rely on large datasets of human text but lack explicit mechanisms to enforce these crucial linguistic properties during generation.
Methods
The core innovation lies in the coherence discriminator ($D_{coherence}$) and the cohesion discriminator ($D_{cohesion}$).
- $D_{coherence}$: This discriminator takes a source text chunk and a target text chunk and outputs a score indicating how likely the target is a coherent continuation of the source. It uses a dual-encoder architecture where each sentence is first represented by a bag-of-words embedding, and then these embeddings are fed into separate encoders (CNN or RNN) to produce sentence representations. The final coherence score is the cosine similarity between the encoded source and target representations.
- $D_{cohesion}$: This discriminator focuses on the relationship between two consecutive sentences. It also uses a dual-encoder architecture but, crucially, encodes individual sentences using word embeddings that preserve word order (CNN or RNN), as word order is vital for local sentence flow. The output is a cosine similarity score between the encoded consecutive sentences.
Both discriminators are trained using a pairwise ranking loss. They are presented with a positive (coherent/cohesive) pair and several artificially constructed negative (incoherent/incohesive) pairs. The discriminators learn to assign higher scores to the positive pairs.
To train the generator, the authors propose negative-critical sequence training. This is a reinforcement learning approach where the scores from the trained discriminators are used as reward signals. Instead of relying on a separate critic network, they use the difference between the discriminator's score for a generated text and the average score of negative samples as the reward. This effectively guides the generator to produce text that the discriminators deem more coherent and cohesive.
Datasets
The paper uses two publicly available datasets for training and evaluation:
- TripAdvisor hotel reviews: Collected by Wang et al. (2010).
- Yelp English reviews dataset: A well-known benchmark for review data.
For training, they filtered these datasets to include reviews with at least 10 sentences, where each sentence has between 5 and 30 words. For generation tasks, the first five sentences of a review are used as the source, and the next five as the target.
Key Insights and Ideas
- Explicitly modeling coherence and cohesion is key: The paper argues that the primary reason for poor long-form text generation is the lack of effective mechanisms to measure and control for local and global consistency.
- Dual discriminators for different levels of linguistic structure: The separation of coherence (paragraph-level) and cohesion (sentence-level) into distinct discriminators allows for a more nuanced evaluation and guidance of the generation process.
- Negative sampling for discriminator training: Artificially constructing negative examples is crucial for training discriminators to distinguish between good and bad text, especially when ground truth for "incoherent" or "incohesive" is not readily available.
- Policy gradient with discriminator feedback: Using discriminators as reward functions in a reinforcement learning setup provides a powerful way to steer a pre-trained language model towards desired linguistic properties beyond simple next-word prediction.
- "Negative-critical" reward formulation: By using the difference between a generated text's score and the average score of negative samples as a reward, the method avoids the need for a separate critic and provides a more stable learning signal.
Key Results
- Improved text quality: Human evaluation shows that the proposed model, GMLE+RL(coherence, cohesion), is preferred over the baseline MLE-trained model (GMLE) in terms of both coherence and cohesion. For instance, human judges preferred the model's output 39.24% of the time for cohesion and 39.72% for coherence compared to the baseline.
- Better automated metrics: The model also shows improvements in standard automated metrics like NLL, PPL, and BLEU scores, indicating better overall text quality.
- Qualitative improvements: Sample generations demonstrate that the discriminator-guided model can produce more interesting and sentiment-consistent continuations, avoiding the repetition and inconsistent opinions often seen in the baseline model.
Limitations and Future Work
- No perfect automated metric: The authors acknowledge that there is no single perfect automated metric for evaluating text generators, which is why human evaluation is crucial.
- "Scratched the surface": Despite promising results, the authors state that the proposed method has not yet reached the ultimate goal of generating truly meaningful and logical long-form texts.
- Future work: The paper suggests that further improvements are needed to meet the ultimate goal of generating high-quality, meaningful, and logical long-form texts. This implies continued research into more sophisticated discriminator designs, reward functions, and training strategies.
Summary updated