T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion

Tianming Wang, Xiaojun Wan

Introduction

Story completion—the task of generating a missing sentence at any arbitrary point within a narrative—represents a significant challenge in natural language processing. Unlike story ending generation, which focuses on a logical conclusion, story completion requires a model to infer underlying causal and temporal logic from both preceding and succeeding context.

Standard neural architectures, such as vanilla Transformers or Recurrent Neural Networks (RNNs), frequently fall into the "generic response trap." Because these models often optimize for the most statistically likely word sequence, they tend to produce dull, repetitive sentences like "He was happy" or "It was fun." To address these issues, this paper introduces T-CVAE, a model that integrates the Transformer architecture with a Conditional Variational Autoencoder (CVAE) to generate coherent, diverse, and logically sound narrative segments.

Methodology

The T-CVAE architecture is built upon two primary innovations designed to capture the nuances of story structure and plot diversity.

1. Shared Attention Stack

In a traditional Transformer setup for tasks like machine translation, separate encoder and decoder stacks are used. T-CVAE departs from this by employing shared self-attention layers. In this configuration, the decoder attends to the encoder's contextual states and its own previous states simultaneously within the same stack. The rationale is that the missing plot point exists within the same "story space" as the surrounding context; therefore, they should be processed within a unified latent representation. This is particularly effective for "discontinuous" context, where the model must bridge the gap between sentences appearing both before and after the missing segment.

2. Conditional Variational Autoencoder (CVAE)

To combat the production of generic responses, the authors integrate a CVAE mechanism.

  • Posterior Network: During training, this network observes both the context and the ground-truth missing sentence to learn a distribution of coherent plots.
  • Prior Network: This network attempts to approximate the posterior distribution using only the available context.
The T-CVAE architecture featuring a shared Transformer stack and integrated Prior/Posterior networks for latent variable sampling.

At inference time, the model samples a latent variable $z$ from the prior distribution. This variable acts as a "creativity" seed, allowing the model to sample from various plausible plot directions rather than defaulting to the most generic statistical average. The model is trained by optimizing the Evidence Lower Bound (ELBO), balancing reconstruction accuracy with the divergence between the prior and posterior distributions.

Experimental Setup

The model was evaluated using the ROCStories corpus, which contains approximately 100,000 five-sentence stories grounded in everyday commonsense logic. The dataset was split 8:1:1 for training, validation, and testing. The task required the model to generate a single missing sentence randomly removed from any of the five positions in a story.

Key Results and Discussion

T-CVAE demonstrated superior performance over several strong baselines, including standard Seq2Seq models, Hierarchical LSTMs (HLSTM), and the vanilla Transformer.

Quantitative and Qualitative Performance

As shown in the results table below, T-CVAE achieved a BLEU score of 4.25%, a significant improvement over the vanilla Transformer's 3.05%. More importantly, the model excelled in diversity metrics; its Distinct-2 (D2) score reached 23.46%, compared to 16.75% for the Transformer, indicating a much broader vocabulary and fewer repetitive phrases.

Quantitative comparison of T-CVAE against baseline models across automatic metrics, diversity scores, and human evaluation.

Human evaluations further validated these findings. Judges rated T-CVAE higher in both Grammar (2.71/3) and Logic (2.13/3). Additionally, the model achieved an 87.54% success rate in adversarial testing, successfully "fooling" a pre-trained BERT-based coherence evaluator more often than its competitors.

Insights on Story Position

An interesting finding from the study was the varying difficulty of generation based on position. The authors noted that generating the first sentence of a story often yielded higher BLEU scores because story beginnings tend to be more generic. As a narrative progresses, the required sentences become more specific and informative, increasing the difficulty for the model to predict the exact ground truth, even as the CVAE helps maintain logical coherence.

Conclusion

T-CVAE represents a significant step forward in automated story completion by combining the contextual power of Transformers with the stochastic diversity of CVAEs. While the model occasionally struggles with highly complex causal chains or matching the full informativeness of human-written text, it successfully avoids the generic response trap that plagues simpler models. Future work will focus on extending this architecture to open-domain story completion with more complex vocabularies and themes.

Summary updated

Abstract

Story completion is a very challenging task of generating the missing plot for an incomplete story, which requires not only understanding but also inference of the given contextual clues. In this paper, we present a novel conditional variational autoencoder based on Transformer for missing plot generation. Our model uses shared attention layers for encoder and decoder, which make the most of the contextual clues, and a latent variable for learning the distribution of coherent story plots. Through drawing samples from the learned distribution, diverse reasonable plots can be generated. Both automatic and manual evaluations show that our model generates better story plots than state-of-the-art models in terms of readability, diversity and coherence.
BibTeX
@inproceedings{wang2019t,
  author = {Tianming Wang and Xiaojun Wan},
  title = {{T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/727},
}
Figure