Abstract
Scripts define knowledge about how everyday scenarios (such as going to a restaurant) are expected to unfold. One of the challenges to learning scripts is the hierarchical nature of the knowledge. For example, a suspect arrested might plead innocent or guilty, and a very different track of events is then expected to happen. To capture this type of information, we propose an autoencoder model with a latent space defined by a hierarchy of categorical variables. We utilize a recently proposed vector quantization based approach, which allows continuous embeddings to be associated with each latent variable value. This permits the decoder to softly decide what portions of the latent hierarchy to condition on by attending over the value embeddings for a given setting. Our model effectively encodes and generates scripts, outperforming a recent language modeling-based method on several standard tasks, and allowing the autoencoder model to achieve substantially lower perplexity scores compared to the previous language modeling-based method.
Introduction: The Challenge of Script Generation
In artificial intelligence, a "script" is a structured sequence of events describing a common scenario. While humans intuitively understand the logical flow of events—such as "ordering food" preceding "eating"—teaching machines to generate these sequences is difficult. Scripts are inherently hierarchical and branching. For instance, a legal proceeding can diverge significantly based on whether a suspect pleads guilty or innocent.
Standard Neural Language Models (LMs) often struggle with these structures due to two primary issues:
- Local Coherency vs. Global Logic: LMs excel at next-token prediction but often lose long-term logical consistency. They might generate a script where a suspect both "denies charges" and "pleads guilty" because both are common in legal contexts, despite being mutually exclusive in a single track.
- Track Blending: LMs tend to blend all possible outcomes into a single distribution, failing to maintain the distinct "tracks" of a narrative.
Methods: Hierarchical Quantized Autoencoder (HAQAE)
To address these limitations, we introduce the Hierarchical Quantized Autoencoder (HAQAE). Unlike traditional Variational Autoencoders (VAEs) that use continuous latent spaces, HAQAE utilizes Vector Quantization (VQ) to define a discrete latent space.
Model Architecture
The model's latent space is structured as a hierarchy of categorical variables. Each variable can take one of $K$ values, with each value associated with a learned embedding in a "codebook." * Hierarchical Structure: Variables are organized in a tree or linear chain. High-level variables capture broad thematic information (e.g., "Legal"), while lower-level variables represent specific sub-tracks or outcomes (e.g., "Conviction"). * Attention Mechanisms: The encoder uses attention to map input event sequences to specific levels of the latent hierarchy. Conversely, the decoder "softly" attends to these latent embeddings to generate a coherent, track-consistent sequence. * Commitment Loss: To avoid "posterior collapse"—a common issue where the model ignores the latent space—we employ a training objective that encourages the encoder to map inputs closely to specific discrete embeddings.
Experimental Setup
The model was trained on the New York Times Gigaword Corpus. We used the Ollie Open Information Extraction system to extract event 4-tuples (subject, verb, object, preposition). The final dataset included approximately 7.1 million event sequences, with lengths ranging from 8 to 50 events.
Key Results
HAQAE demonstrated superior performance across several metrics compared to standard RNNLMs and vanilla VAEs:
- Perplexity: HAQAE achieved a test set perplexity of 42.10, a significant improvement over the standard RNNLM's 90.92. This suggests that global latent structures are highly effective for modeling long-form event sequences.
- Inverse Narrative Cloze: In tasks requiring the model to identify the correct continuation of a story, HAQAE achieved 33.85% accuracy, outperforming the RNNLM baseline (26.30%).
- Human Evaluation: Human judges rated HAQAE significantly higher on "Branching Quality" (1.00 vs. 0.51 for the baseline), validating the model's ability to maintain distinct, non-contradictory narrative paths.
- Efficiency: The hierarchical structure facilitated faster convergence and lower error rates compared to non-hierarchical variants.
Conclusion
The HAQAE model demonstrates that hierarchical, discrete latent spaces are powerful tools for capturing the branching nature of real-world scripts. By moving beyond simple language modeling and incorporating global structural knowledge, we can generate more coherent and logically consistent event sequences. Future work will explore more complex latent structures, such as Directed Acyclic Graphs (DAGs), and explicit modeling of participant roles within scripts.
Summary updated