Abstract
Generating a long, coherent text such as a paragraph requires a high-level control of different levels of relations between sentences (e.g., tense, coreference). We call such a logical connection between sentences as a (paragraph) flow. In order to produce a coherent flow of text, we explore two forms of intersentential relations in a paragraph: one is a human-created linguistical relation that forms a structure (e.g., discourse tree) and the other is a relation from latent representation learned from the sentences themselves. Our two proposed models incorporate each form of relations into document-level language models: the former is a supervised model that jointly learns a language model as well as discourse relation prediction, and the latter is an unsupervised model that is hierarchically conditioned by a recurrent neural network (RNN) over the latent information. Our proposed models with both forms of relations outperform the baselines in partially conditioned paragraph generation task. Our codes and data are publicly available.
Introduction
While modern language models excel at local coherence—ensuring that word sequences are grammatically sound—they often struggle with global coherence. Maintaining a logical "flow" across an entire paragraph involves managing complex intersentential connections such as tense consistency, coreference, and rhetorical intent. This paper investigates two distinct methods for mathematically representing this flow: explicit linguistic relations and implicit latent relations.
Methodology: The FlowNet Architectures
The authors propose FlowNet, a framework designed to bridge the gap between the beginning and end of a paragraph. Two variants are explored:
1. Discourse-driven FlowNet (Linguistic)
This supervised model utilizes Rhetorical Structure Theory (RST) to define relations between phrases. It is trained via multi-task learning, where the model simultaneously predicts the next word and the specific discourse label (e.g., Contrast, Elaboration) that connects the current sentence to the previous one. A Conditional Random Field (CRF) is used to optimize these label sequences.
2. Delta-driven FlowNet (Latent)
This unsupervised variant calculates a "delta"—the vector subtraction between the hidden representations of consecutive sentences ($d_i = s_i - s_{i-1}$). This delta effectively captures the semantic "step" taken in the narrative and is fed into a hierarchical RNN to condition the generation of the next sentence.
Experimental Setup
The models were evaluated on a "bridging" task: given the first and last sentences of a paragraph, the model must generate the missing middle sentences. The researchers tested these models across three distinct domains to ensure generalizability:
- Academic Papers: PeerRead dataset (~16k paragraphs).
- Science Fiction: BookCorpus (~157k paragraphs).
- Fantasy: BookCorpus (~317k paragraphs).
The study focused on paragraphs between 4 and 7 sentences long to measure how performance scales as the narrative flow becomes more complex.
Results and Analysis
Quantitative evaluation using the METEOR metric demonstrated that FlowNet (Discourse) consistently achieved the highest scores across all datasets. For example, on the Papers dataset, it scored 4.0 compared to the baseline's 3.7.
Key Insights
- The Power of Subtraction: In the latent model, vector subtraction (the "delta") outperformed addition or multi-layer perceptron combinations. This suggests that the difference between two sentence states is a highly effective way to capture narrative progression.
- Discourse as a Roadmap: Explicit discourse relations act as a high-level plan. By forcing the model to predict these relations, it learns a more robust internal representation of logical progression, which also leads to higher linguistic diversity in generated text.
- Scalability with Length: As shown in the performance metrics, the gap between FlowNet and standard models widens as paragraphs grow longer (6–7 sentences), indicating that explicit flow modeling is crucial for preventing "drift" in long-form generation.
Conclusion
Both linguistic and latent relations offer significant improvements over standard hierarchical models for paragraph generation. While discourse-driven models provide superior precision and diversity, they currently rely on external parsers. Future research may focus on hybrid models that combine explicit labels with latent deltas, potentially integrated into large-scale pre-trained Transformers to further bridge the gap between AI and human-level coherence.
Summary updated