Standard machine learning pipelines often treat language as a collection of independent sentences. To maximize efficiency and data variety, researchers frequently shuffle sentences across a corpus or truncate long texts into fixed-length fragments. While this "bag of sentences" approach works for local syntax, it induces what Hao Xiong and Zhongjun He call "discourse blindness." When a model only sees isolated snippets, it loses the connective tissue—the pronouns, logical transitions, and thematic consistency—that makes a document coherent.
A growing body of research suggests that the "how" of data organization is just as critical as the "how much." By preserving the original document structure during training, models can move beyond sentence-level fluency to true discourse-level understanding.
The Cost of Shuffling and Truncation
The intuition behind document-level training is simple: a model cannot learn to track a character across a chapter if that chapter has been sliced into unrelated pieces. Hantian Ding and Zijian Wang demonstrated in Fewer Truncations Improve Language Modeling that training on intact documents rather than truncated fragments leads to substantial gains in tasks like reading comprehension. When a model is trained on a continuous, meaningful context, it learns to follow premises and utilize defined variables more effectively.
Conversely, traditional auxiliary tasks like Next Sentence Prediction (NSP) can actually be counterproductive. Stéphane Aroca-Ouellette and Frank Rudzicz found that training on unrelated sentence pairs reduces the model's exposure to long-range coherent context, hindering the acquisition of complex language structures. In On Losses for Modern Language Models, they argue that maintaining a continuous stream of text is far more beneficial than forcing the model to solve artificial binary classification tasks.
Capturing Context via Batching and Embeddings
If the goal is to provide document-level awareness without the massive computational overhead of processing entire books at once, how should we structure the training?
One approach is to rethink the mini-batch. In Context-aware Neural Machine Translation with Mini-batch Embedding, Jun Suzuki and Makoto Morishita proposed grouping sentences from the same document into a single mini-batch. They introduced a Mini-batch Embedding (MBE) that represents the collective features of these sentences. This allows the model to "see" the shared thematic context of the document without requiring a complex hierarchical architecture.
Similarly, in the field of Grammatical Error Correction (GEC), Andrey Bout and Irina Piontkovskaya showed that avoiding sentence-level shuffling preserves the syntactic flow and intent of an essay. Their work in Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule proved that a smaller model (400M parameters) could outperform much larger ones simply by leveraging the document-level coherence inherent in the training data.
Scaling Discourse to Large Corpora
The shift toward document-level awareness is also transforming how we build foundational datasets. For instance, Amir Hossein Kargaran and Hinrich Schuetze developed GlotCC, a massive corpus covering over 1,000 languages. Unlike previous web-crawled datasets that often prioritized individual snippets, GlotCC preserves document-level structure to ensure that generative models can learn long-range dependencies and global context.
This structural preservation is particularly vital for "high-difficulty" tasks like literary translation. Longyue Wang and Zhaopeng Tu have highlighted this in their work on the WMT Shared Task on Discourse-Level Literary Translation. By using the GuoFeng Webnovel Corpus, which maintains chapter and book-level hierarchies, they demonstrated that specialized document-level models are better at maintaining character consistency and narrative threads than general-purpose LLMs that might only see a few sentences at a time.
Foundational Reading Behaviors
Beyond just pretraining, document-level context is being used to "educate" smaller models. Zhi Zhou and Xiangyu Duan introduced Basic Reading Distillation (BRD), which teaches models fundamental reading behaviors—like identifying entities and tracking information—across entire passages. Their findings suggest that preserving the natural flow of language during distillation helps models learn to maintain coherence across sentence boundaries, allowing a 564M parameter model to punch far above its weight class.
By treating a document as a single, coherent unit of information rather than a sequence of independent tokens, these researchers are building models that don't just translate or generate text, but actually follow the "story" of the data.