Hierarchical Quantized Representations for Script Generation

Noah Weber, Leena Shekhar, Niranjan Balasubramanian, Nathanael Chambers

TL;DR

This paper introduces the Hierarchical Quantized Autoencoder (HAQAE), a novel deep learning model designed to generate realistic event sequences (scripts) that capture hierarchical variations. By employing a structured latent space and vector quantization, HAQAE effectively models different "tracks" within a script, outperforming previous methods in both language modeling and script generation tasks.

What Problem Does This Paper Address?

Everyday scenarios, like going to a restaurant, can unfold in many different ways. This inherent variability, or "script tracks," is crucial for understanding and generating natural language. Traditional language models struggle with this because they tend to focus on local coherence and jumble together contradictory events, failing to explicitly represent the hierarchical nature of scripts. For instance, a model might assign high probability to both "suspect pleads guilty" and "suspect pleads innocent" within the same script, which is nonsensical. This paper aims to address this by explicitly modeling the hierarchical structure of scripts.

Methods

The core contribution is the Hierarchical Quantized Autoencoder (HAQAE). It builds upon the Variational Autoencoder (VAE) framework but incorporates two key innovations:

  • Hierarchical Latent Space: Instead of a single latent code, HAQAE uses a hierarchy of discrete latent variables. This hierarchy mirrors the structure of scripts, where higher-level variables capture general topics (e.g., "restaurant") and lower-level variables capture specific tracks (e.g., "fancy restaurant" vs. "casual restaurant"). The latent variables are structured as a tree (or DAG), with dependencies between parent and child latents.
  • Vector Quantization (VQ): To handle discrete latent variables effectively and avoid the "posterior collapse" problem common in VAEs, HAQAE employs vector quantization, similar to VQ-VAEs. Each discrete latent value is associated with a learned embedding vector. The encoder maps the input to a continuous representation, which is then "quantized" to the closest embedding. This quantized embedding is then used by the decoder.

The model is trained end-to-end using a combination of reconstruction loss and a "commit loss" to encourage the encoder's output to be close to the learned embeddings. Attention mechanisms are used in both the encoder and decoder to allow them to focus on relevant parts of the input and latent representations.

Datasets

The primary dataset used is the New York Times Gigaword Corpus. This corpus consists of approximately 1.8 million articles. The authors processed this data to extract event sequences, where each event is represented as a 4-tuple (verb, subject, object, preposition). They extracted event tuples using the Ollie system and grouped them into sequences. The final dataset contains over 700,000 training sequences, with fixed minimum and maximum sequence lengths of 8 and 50, respectively.

Key Insights and Ideas

  • Hierarchy Matters for Scripts: Explicitly modeling the hierarchical structure of scripts is crucial for generating coherent and distinct "tracks" of events, rather than a jumbled mix.
  • Vector Quantization for Discrete Latents: VQ is an effective technique for handling discrete latent variables in autoencoders, mitigating issues like posterior collapse and enabling better representation learning.
  • Attention Bridges Latents and Data: Using attention mechanisms allows the model to dynamically weigh the influence of different latent variables and input segments, leading to more nuanced generation.
  • Human Evaluation Reveals Nuance: While perplexity is a standard metric, human evaluation on tasks like "event schemas" and "inverse narrative cloze" is essential for assessing the model's ability to capture subtle script variations and branching.

Key Results

  • Perplexity: HAQAE achieved the lowest negative log-likelihood and perplexity scores on both validation and test sets compared to RNNLM, RNNLM+Role, and VAE baselines, indicating superior language modeling capabilities.
  • Inverse Narrative Cloze: HAQAE and a hierarchyless variant (NOHIER) significantly outperformed other baselines, achieving top accuracies of 34.0% and 33.85% respectively, demonstrating their ability to identify legitimate event sequences.
  • Human Evaluation (Event Schemas): HAQAE consistently outperformed the RNNLM+Role baseline across all metrics, including relevance, coherence, branching uniqueness, and branching quality. Notably, HAQAE doubled the RNNLM+Role scores for branching uniqueness and quality, highlighting its strength in generating distinct script tracks.
  • Training Efficiency: HAQAE showed faster convergence and lower cross-entropy error during training compared to the NOHIER model, suggesting that the explicit hierarchical inductive bias aids learning.

Limitations and Future Work

  • Interpreting Latent Variables: The authors acknowledge that understanding exactly what is learned by the discrete latent variables is non-trivial, especially due to the attention mechanism which can obscure direct causal links.
  • Future Work:
    • Further in-depth study into the properties of learned discrete latents.
  • Exploring more complex hierarchical structures beyond linear chains (e.g., DAGs).
  • Investigating methods for composing event components into single distributed representations.
  • Extending the model to handle more complex script structures and roles.

Summary updated

BibTeX
@inproceedings{weber2018hierarchical,
  author = {Noah Weber and Leena Shekhar and Niranjan Balasubramanian and Nathanael Chambers},
  title = {{Hierarchical Quantized Representations for Script Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=rJ4sEzzdWr},
}
Figure