Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2

He Bai, Peng Shi, Jimmy Lin, Luchen Tan, Kun Xiong, Wen Gao, Jie Liu, Ming Li

Introduction

Large language models (LLMs) like GPT-2 have demonstrated remarkable proficiency in generating fluent, local text. However, these models often struggle with macro-structure and narrative coherence in long-form generation. A common failure mode involves models "rambling" indefinitely or producing abrupt, unsatisfying endings. This research explores the "semantics of the unwritten," arguing that structural markers like paragraph breaks are not merely formatting choices but carry significant semantic weight that guides a model's understanding of narrative progression.

Methodology

The study investigates how explicit training on paragraph boundaries affects the prediction of the End-of-Sequence (EOS) token. The researchers compared three primary configurations for marking paragraph boundaries during GPT-2 fine-tuning:

  • None: Paragraphs are concatenated without explicit markers.
  • SEP (Separator): Standard line breaks (\n) or special [SEP] tokens are used between paragraphs.
  • EOP (End of Paragraph): A specific [EOP] token is appended to the conclusion of every paragraph.

The experiments utilized a 1.5-billion parameter Chinese-GPT2 and a 117-million parameter English GPT2, employing top-p sampling ($p=0.95$) for text generation.

Datasets

To ensure cross-linguistic validity, the researchers evaluated their approach on two distinct datasets:

  1. WritingPrompts (English): Approximately 300,000 human-written stories from Reddit, filtered for length to fit the GPT-2 context window.
  2. ChineseEssay (Chinese): A collection of 1,615 descriptive essays by primary school students, characterized by clear thematic focus and structured endings.

Structural Semantics and the "Countdown" Effect

A key finding of this research is that EOP tokens function as a "countdown" signal for the model. As the number of paragraphs increases, models trained with explicit EOP markers show a significantly higher probability of predicting the final EOS token.

Figure 1: EOS Probability Ranking across relative positions, showing the 'countdown' effect in EOP-trained models.

As shown in Figure 1, the EOS probability ranking (where lower ranking indicates higher probability) drops much more sharply for EOP-based models as the relative position in the text increases. This suggests that the model learns to associate paragraph completion with the approaching end of the narrative arc. In contrast, models without these markers (the "None" approach) exhibit much higher perplexity for the EOS token, effectively failing to recognize when the story should conclude.

Human Evaluation and Qualitative Results

The quantitative improvements were mirrored in human evaluations. In the Chinese essay task, the inclusion of EOP tokens caused the EOS Perplexity to plunge from 22.15 to 2.74. Furthermore, the EOP DIY (custom token) approach achieved the highest BLEU-1 and BLEU-2 scores across the board.

Figure 4: Human evaluation comparison between 'None' and 'EOP' models across four quality metrics.

Human evaluators showed a clear preference for EOP-enabled models. For "Ending Quality," the EOP model outperformed the "None" model in 66% of cases. As illustrated in Figure 4, while fluency remained comparable between models, the EOP approach significantly outperformed the baseline in providing satisfying conclusions and maintaining topic relevance. By respecting paragraph boundaries, the model seemed less likely to drift off-topic during long generations.

Conclusion and Future Work

This paper demonstrates that treating paragraph breaks as semantic milestones significantly enhances the structural coherence of LLM-generated text. While the results are compelling, the authors note that individual paragraphs in EOP models can sometimes feel too independent. Future research is needed to determine if these benefits scale to even larger models and how to better balance paragraph-level independence with overall narrative flow.

Summary updated

BibTeX
@inproceedings{bai2021semantics,
  author = {He Bai and Peng Shi and Jimmy Lin and Luchen Tan and Kun Xiong and Wen Gao and Jie Liu and Ming Li},
  title = {{Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-srw.16},
}
Figure