Harry Potter and the Action Prediction Challenge from Natural Language

David Vilares, Carlos Gómez-Rodríguez

Abstract

Predicting subsequent actions from natural language descriptions is a core challenge in narrative understanding. This paper introduces the Harry Potter Action prediction (HPAC) corpus, leveraging the unique semantic anchors provided by "spells" in the Wizarding World to create a large-scale, self-annotated dataset. We evaluate several baseline architectures, finding that while sequential models like LSTMs benefit significantly from increased context, simpler models remain competitive on infrequent actions.

Introduction

Action prediction involves inferring the most likely next behavior from a scene description. Unlike reading comprehension or cloze tasks, action prediction suffers from labeling ambiguity; actions are often described with varied, nuanced language. To address this, we utilize the Harry Potter universe, where spells like Alohomora or Lumos serve as discrete, unambiguous labels for complex actions (e.g., unlocking a door or turning on a light).

Examples of text fragments and their corresponding action labels in the HPAC dataset.

The HPAC Corpus

The primary contribution of this work is the HPAC (Harry Potter Action prediction) Corpus. Because the original seven books contain only 432 spell occurrences—insufficient for deep learning—the authors crawled fanfiction.net for English stories. The resulting corpus contains 82,836 samples across 85 distinct actions (spells). The data follows a "long tail" distribution, where a few spells (like Avada Kedavra) appear thousands of times, while others appear only a handful of times.

Methods

We frame action prediction as a sequence-to-label classification task. Four architectures were evaluated to establish baselines:

  1. Multinomial Logistic Regression (MLR): A simple model using one-hot encoded word counts.
  2. MultiLayer Perceptron (MLP): A basic neural network with one hidden layer.
  3. Convolutional Neural Network (CNN): Designed to capture local patterns and n-gram features.
  4. Long Short-Term Memory (LSTM): A sequential model designed to handle long-range dependencies using pre-trained GloVe embeddings.

We varied the "snippet length" (32, 64, 96, or 128 tokens) provided to the models to assess the impact of narrative context on prediction accuracy.

Key Results and Insights

The LSTM model achieved the highest overall performance, particularly as context length increased. With a 128-token snippet, the LSTM reached a Recall@1 of 35.1% and a Recall@10 of 76.9%. This suggests that LSTMs successfully "read" the narrative arc rather than just identifying immediate trigger words.

Recall@k results across different model architectures and snippet lengths.

However, a "Simple Model Paradox" emerged: while LSTMs dominate on frequent actions, MLR often performs better on infrequent actions. Neural models tend to over-fit to majority classes, whereas simpler models remain more robust under data scarcity. Additionally, models frequently struggle with "counter-actions" (e.g., confusing Lumos with Nox) because the surrounding context—such as being in a dark hallway—is nearly identical for both.

Conclusion

The HPAC challenge demonstrates that fictional magic systems provide a unique, discrete vocabulary for intent that can serve as a benchmark for action prediction. While LSTMs lead the pack with sufficient context, the massive performance gap between frequent and infrequent actions remains a hurdle. Future work should explore transferring these insights to real-world domains and incorporating external knowledge bases to resolve semantic confusion between related actions.

Summary updated

Abstract

We explore the challenge of action prediction from textual descriptions of scenes, a testbed to approximate whether text inference can be used to predict upcoming actions. As a case of study, we consider the world of the Harry Potter fantasy novels and inferring what spell will be cast next given a fragment of a story. Spells act as keywords that abstract actions (e.g. 'Alohomora' to open a door) and denote a response to the environment. This idea is used to automatically build HPAC, a corpus containing 82,836 samples and 85 actions. We then evaluate different baselines. Among the tested models, an LSTM-based approach obtains the best performance for frequent actions and large scene descriptions, but approaches such as logistic regression behave well on infrequent actions.
BibTeX
@inproceedings{vilares2019harry,
  author = {David Vilares and Carlos Gómez-Rodríguez},
  title = {{Harry Potter and the Action Prediction Challenge from Natural Language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1218},
}
Figure