Automatic Generation of German Drama Texts Using Fine Tuned GPT-2 Models

Yifan Wang, Mariam Bangura, Kristina Barabashova, Anna Karnysheva, Sarah Semczuk, Yifan Wang 0010arXiv 2023 · DOI · arXiv

Abstract

Automated story generation typically focuses on narrative prose, leaving the structurally complex genre of drama relatively unexplored. This paper presents a hierarchical, two-stage approach to generating German drama texts using fine-tuned GPT-2 models. By decomposing the task into outline generation and iterative dialogue expansion—supported by dynamic context summarization—the researchers aim to maintain global coherence and adhere to the strict formal requirements of theatrical scripts.

Introduction: The Challenge of Automated Drama

While large language models have shown proficiency in generating fluid prose, drama introduces unique structural and narrative hurdles. Unlike standard narratives, plays rely on a rigid interplay of dialogue, stage directions, and character-specific tags. Furthermore, the length of a typical play often exceeds the context window of standard transformer models, leading to a loss of character consistency and plot direction over time.

Structural Constraints and Representation

A primary challenge in drama generation is the requirement for specific structural elements. Plays are not merely blocks of text; they include metadata such as drama IDs, scene boundaries, and speaker identifiers. To handle this, the researchers utilized an XML-like tagging system to delineate different parts of the script.

Example of the XML-like tagging structure used to represent drama elements such as speaker IDs and scene boundaries.

As shown in Figure 1, these tags (e.g., $sp_#dalton) serve as anchors for the model, helping it distinguish between dialogue and metadata. However, maintaining this structure throughout a long generation process remains a significant technical difficulty.

Methodology: A Two-Stage Hierarchical Pipeline

To address long-range dependencies and global consistency, the authors proposed a two-stage generation pipeline:

  1. The Outline Model: A German GPT-2 model fine-tuned to generate a narrative scene outline based on ten extracted keywords.
  2. The Generation Model: A second fine-tuned GPT-2 that expands this outline into full dialogue.

To overcome the 1024-token limit of GPT-2, the system employs Iterative Generation. The model produces text in 100-token increments. For each step, a Dynamic Prompt is constructed to provide the model with both global and local context.

The iterative generation pipeline using a dynamic prompt that combines the scene outline, a summary of remote context, and the immediate local context.

As illustrated in Figure 2, the prompt includes the static scene outline (global anchor), a summary of the "remote context" (generated via TextRank), and the "local context" (the most recent 250 tokens). This structure ensures the model "remembers" the overall goal of the scene while responding to the immediate preceding dialogue.

Datasets and Orthographic Normalization

The models were trained on two major corpora: the German Drama Corpus (GerDraCor), containing 545 dramas from the 17th to 20th centuries, and the German Text Archive (DTA).

A critical component of the preprocessing was orthographic normalization. Historical German features archaic spellings (e.g., Cigarenttentasche vs. modern Zigarettentasche) that can confuse modern language models. The researchers developed a custom mapping system to normalize these historical variations, ensuring the model could learn from centuries of theatrical tradition without being hindered by inconsistent spelling.

Results and Qualitative Analysis

The study revealed a notable divergence between quantitative metrics and human evaluation.

  • Quantitative Performance: Fine-tuning significantly reduced perplexity (from ~20.5 to ~18.8). Models using the hierarchical "extracted outline" approach achieved the highest n-gram overlap with ground-truth texts, suggesting that the outline effectively guided the generation.
  • Qualitative Performance: Despite high scores on automatic metrics, human reviewers noted several persistent issues. Models often struggled with speaker-switching, sometimes producing long monologues or introducing "hallucinated" characters not present in the initial outline. Additionally, the models occasionally over-learned punctuation patterns, such as the frequent use of hyphens for pauses in historical texts, leading to repetitive and "stuttering" dialogue.

Conclusion

This research demonstrates that while GPT-2 can be fine-tuned to mimic the formal structure of German drama, capturing the "soul" of a play—its character depth and narrative tension—remains a challenge. The hierarchical approach provides a promising framework for maintaining global coherence, but future work is needed to improve character consistency and manage the transition between scenes to generate full-length theatrical works.

Summary updated

BibTeX
@misc{wang2023automatic,
  author = {Yifan Wang and Mariam Bangura and Kristina Barabashova and Anna Karnysheva and Sarah Semczuk and Yifan Wang},
  title = {{Automatic Generation of German Drama Texts Using Fine Tuned GPT-2 Models}},
  year = {2023},
  eprint = {2301.03119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.03119},
}
Figure