Abstract
Generating a long, coherent text such as a paragraph requires a high-level control of different levels of relations between sentences (e.g., tense, coreference). We call such a logical connection between sentences as a (paragraph) flow. In order to produce a coherent flow of text, we explore two forms of intersentential relations in a paragraph: one is a human-created linguistical relation that forms a structure (e.g., discourse tree) and the other is a relation from latent representation learned from the sentences themselves. Our two proposed models incorporate each form of relations into document-level language models: the former is a supervised model that jointly learns a language model as well as discourse relation prediction, and the latter is an unsupervised model that is hierarchically conditioned by a recurrent neural network (RNN) over the latent information. Our proposed models with both forms of relations outperform the baselines in partially conditioned paragraph generation task.
Introduction
While modern language models excel at local syntax and grammar, maintaining global coherence across an entire paragraph remains a significant challenge. Standard models often treat paragraphs as flat sequences of words, which can lead to issues like thematic drift, loss of logical consistency, or repetitive loops. This research introduces the concept of "paragraph flow"—the logical connective tissue between sentences—and investigates whether explicit human-defined linguistic structures or learned latent representations are more effective for modeling it.
Methodology: The FlowNet Framework
The authors propose FlowNet, a framework designed to improve paragraph coherence by explicitly modeling inter-sentential transitions through two distinct approaches:
- Discourse-driven FlowNet (Supervised): This model leverages Rhetorical Structure Theory (RST) to identify explicit relations (e.g., Cause, Elaboration, Contrast) between text segments. It uses a multi-task learning objective to jointly predict the next word and the corresponding RST discourse label, grounding the model's internal representations in human-defined linguistic logic.
- Delta-driven FlowNet (Unsupervised): This variant calculates a "delta" vector—the mathematical difference between adjacent sentence embeddings ($d_t = s_t - s_{t-1}$). This delta serves as a latent representation of the transition, capturing "what changed" between ideas. A hierarchical RNN then uses these deltas to condition the generation of the subsequent sentence.
Experimental Evaluation
The models were evaluated on a "Bridging" task, where the system must generate missing middle sentences given the start and end of a paragraph. This task is a rigorous test of a model's ability to maintain logical flow. Testing spanned three domains: academic papers (PeerRead), science fiction, and fantasy (BookCorpus), with paragraph lengths ranging from 4 to 7 sentences.
Results and Discussion
Quantitative analysis using METEOR scores shows that both FlowNet variants consistently outperform standard sequence-to-sequence baselines. Notably, the Discourse-driven model generally achieved the highest scores across most datasets and paragraph lengths, suggesting that explicit linguistic blueprints provide superior guidance for maintaining coherence.
Qualitatively, the Discourse-driven model was found to be more robust against "repetition loops." By requiring the model to satisfy specific discourse relations, it is forced to progress the narrative or argument logically rather than stalling on generic phrases. In the Delta-driven model, the simple subtraction of embeddings proved surprisingly effective, capturing the semantic "shift" between sentences better than more complex concatenation methods.
Conclusion
This work demonstrates that explicitly modeling the "flow" between sentences significantly enhances paragraph coherence. While both linguistic and latent approaches show promise, human-defined discourse structures currently offer the strongest performance blueprint. Future work may explore hybrid models that combine the strengths of supervised linguistic labels with the flexibility of unsupervised latent deltas, potentially integrating these objectives into large-scale pre-trained transformers.
Summary updated