Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind

Mo Yu, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Jing Li, Yue Yu, Jie Zhou

Abstract

Understanding fictional characters quickly is a hallmark of human social intelligence, rooted in Theory-of-Mind (ToM). We introduce TOM-IN-AMC, a novel benchmark that evaluates whether AI can mimic this "few-shot" social learning by identifying anonymized characters in movie scripts. We also propose ToMPro (ToM Prompting), a two-stage framework for iterative character modeling. Our findings reveal that even advanced models like GPT-4 lag significantly behind human performance, particularly when memorization shortcuts are removed via name perturbation.

Introduction

Humans are adept at "reading the room." Upon starting a new movie, we do not require a full biography to understand a protagonist; within a few scenes, we infer their personality, goals, and likely actions. This ability to infer internal mental states is known as Theory-of-Mind (ToM). While ToM has been studied in synthetic environments, existing character-understanding tasks often provide models with exhaustive histories, which does not reflect the rapid, few-shot nature of human social reasoning. We argue that humans utilize meta-learning, applying prior knowledge of character archetypes to understand new individuals.

The Character Guessing Task

The core of our assessment is a character guessing task. Given a scene where character names are replaced with generic IDs (e.g., P0, P1), the model must identify the characters based on a limited number of preceding "training" scenes from the same movie.

Overview of the TOM-IN-AMC dataset and the meta-learning task setting, including script pre-processing and task splitting.

Methods: ToMPro

To address this challenge, we developed ToMPro (ToM Prompting) for Large Language Models (LLMs). This approach consists of two primary stages:

  1. Iterative Modeling: The model processes a sequence of scenes, maintaining and updating a "mental model" for each character across five dimensions: personality, emotions, beliefs, desires, and intentions.
  2. Identity Guessing: The model leverages these generated mental descriptions to identify anonymized characters in a novel test scene.
The ToMPro framework, illustrating the iterative update of character descriptions across personality and other ToM dimensions to facilitate identity guessing.

To prevent models from relying on pre-training memorization, we employ Name Perturbation, replacing iconic names with common ones while preserving gender.

The TOM-IN-AMC Dataset

We present TOM-IN-AMC, the first benchmark for meta-learning of ToM in realistic narrative settings. Derived from approximately 1,000 movie scripts from the Internet Movie Script Database (IMSDB), the dataset includes 3,837 characters and over 74,000 scenes. Unlike previous datasets limited to specific shows, TOM-IN-AMC spans diverse genres, including Drama, Thriller, and Comedy, providing a robust testbed for generalized social reasoning.

Key Insights and Results

Our experiments yielded several critical findings:

  • The Human-AI Gap: Humans achieved 88.0% accuracy, while the best-performing AI (ToMPro with GPT-4) reached 66.9%. This 20% gap underscores the difficulty of rapid social nuance for current AI.
  • Archetypal Reasoning: Human participants reported using "archetypes" or direct analogies to characters in other films, confirming that character understanding is a meta-learning process.
  • Dimensions of ToM: Ablation studies indicated that Desire and Intention are the most critical dimensions for character identification, whereas Emotion proved the least helpful.
  • Memorization vs. Reasoning: GPT-4's performance dropped significantly when name perturbation was applied, indicating a heavy reliance on memorized training data rather than genuine social reasoning.

Conclusion and Future Work

While ToMPro advances the state-of-the-art in few-shot character understanding, several challenges remain. Models still struggle with "global understanding," often failing to connect immediate reactions to long-term goals. Furthermore, the dataset reflects historical Hollywood gender biases, which models may exploit as shortcuts. Future work should focus on distilling ToM capabilities into smaller models and improving long-term contextual coherence.

Summary updated

Abstract

When reading a story, humans can rapidly understand new fictional characters with a few observations, mainly by drawing analogy to fictional and real people they met before in their lives. This reflects the few-shot and meta-learning essence of humans' inference of characters' mental states, i.e., humans' theory-of-mind (ToM), which is largely ignored in existing research. We fill this gap with a novel NLP benchmark, TOM-IN-AMC, the first assessment of models' ability of meta-learning of ToM in a realistic narrative understanding scenario. Our benchmark consists of $\sim$1,000 parsed movie scripts for this purpose, each corresponding to a few-shot character understanding task; and requires models to mimic humans' ability of fast digesting characters with a few starting scenes in a new movie. Our human study verified that humans can solve our problem by inferring characters' mental states based on their previously seen movies; while the state-of-the-art metric-learning and meta-learning approaches adapted to our task lags 30% behind.
BibTeX
@inproceedings{yu2024few,
  author = {Mo Yu and Yisi Sang and Kangsheng Pu and Zekai Wei and Han Wang and Jing Li and Yue Yu and Jie Zhou},
  title = {{Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=ZZ7UKgK4c1},
}
Figure