Introduction
Story completion—the task of generating a missing sentence at any arbitrary point within a narrative—represents a significant challenge in natural language processing. Unlike story ending generation, which focuses on a logical conclusion, story completion requires a model to infer underlying causal and temporal logic from both preceding and succeeding context.
Standard neural architectures, such as vanilla Transformers or Recurrent Neural Networks (RNNs), frequently fall into the "generic response trap." Because these models often optimize for the most statistically likely word sequence, they tend to produce dull, repetitive sentences like "He was happy" or "It was fun." To address these issues, this paper introduces T-CVAE, a model that integrates the Transformer architecture with a Conditional Variational Autoencoder (CVAE) to generate coherent, diverse, and logically sound narrative segments.
Methodology
The T-CVAE architecture is built upon two primary innovations designed to capture the nuances of story structure and plot diversity.
1. Shared Attention Stack
In a traditional Transformer setup for tasks like machine translation, separate encoder and decoder stacks are used. T-CVAE departs from this by employing shared self-attention layers. In this configuration, the decoder attends to the encoder's contextual states and its own previous states simultaneously within the same stack. The rationale is that the missing plot point exists within the same "story space" as the surrounding context; therefore, they should be processed within a unified latent representation. This is particularly effective for "discontinuous" context, where the model must bridge the gap between sentences appearing both before and after the missing segment.
2. Conditional Variational Autoencoder (CVAE)
To combat the production of generic responses, the authors integrate a CVAE mechanism.
- Posterior Network: During training, this network observes both the context and the ground-truth missing sentence to learn a distribution of coherent plots.
- Prior Network: This network attempts to approximate the posterior distribution using only the available context.
At inference time, the model samples a latent variable $z$ from the prior distribution. This variable acts as a "creativity" seed, allowing the model to sample from various plausible plot directions rather than defaulting to the most generic statistical average. The model is trained by optimizing the Evidence Lower Bound (ELBO), balancing reconstruction accuracy with the divergence between the prior and posterior distributions.
Experimental Setup
The model was evaluated using the ROCStories corpus, which contains approximately 100,000 five-sentence stories grounded in everyday commonsense logic. The dataset was split 8:1:1 for training, validation, and testing. The task required the model to generate a single missing sentence randomly removed from any of the five positions in a story.
Key Results and Discussion
T-CVAE demonstrated superior performance over several strong baselines, including standard Seq2Seq models, Hierarchical LSTMs (HLSTM), and the vanilla Transformer.
Quantitative and Qualitative Performance
As shown in the results table below, T-CVAE achieved a BLEU score of 4.25%, a significant improvement over the vanilla Transformer's 3.05%. More importantly, the model excelled in diversity metrics; its Distinct-2 (D2) score reached 23.46%, compared to 16.75% for the Transformer, indicating a much broader vocabulary and fewer repetitive phrases.
Human evaluations further validated these findings. Judges rated T-CVAE higher in both Grammar (2.71/3) and Logic (2.13/3). Additionally, the model achieved an 87.54% success rate in adversarial testing, successfully "fooling" a pre-trained BERT-based coherence evaluator more often than its competitors.
Insights on Story Position
An interesting finding from the study was the varying difficulty of generation based on position. The authors noted that generating the first sentence of a story often yielded higher BLEU scores because story beginnings tend to be more generic. As a narrative progresses, the required sentences become more specific and informative, increasing the difficulty for the model to predict the exact ground truth, even as the CVAE helps maintain logical coherence.
Conclusion
T-CVAE represents a significant step forward in automated story completion by combining the contextual power of Transformers with the stochastic diversity of CVAEs. While the model occasionally struggles with highly complex causal chains or matching the full informativeness of human-written text, it successfully avoids the generic response trap that plagues simpler models. Future work will focus on extending this architecture to open-domain story completion with more complex vocabularies and themes.
Summary updated