Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind

Mo Yu, Qiujing Wang, Shunchi Zhang, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Liyan Xu, Jing Li, Yue Yu, Jie Zhou

TL;DR

Humans can quickly grasp a new fictional character's personality and motivations after just a few scenes by drawing on past experiences. This paper introduces TOM-IN-AMC, a novel dataset of movie scripts designed to test if AI models can mimic this "meta-learning" aspect of Theory-of-Mind (ToM). While the authors' new ToM-prompting method improves LLM performance, even GPT-4 lags more than 20% behind human baselines, revealing a significant gap in current AI social reasoning capabilities.

What Problem Does This Paper Address?

Theory-of-Mind (ToM)—the cognitive ability to understand and infer the mental states, beliefs, and intentions of others—is a hot topic in AI. However, most current NLP benchmarks evaluate ToM using highly synthetic, artificial scenarios or require massive amounts of data per character.

In the real world, humans perform "few-shot" ToM. When we meet someone new or start watching a new movie, we don't wait for a massive dataset of their behaviors to understand them. We make rapid inferences based on a few initial observations, drawing analogies to people or character tropes we already know. This paper argues that this is fundamentally a meta-learning problem. The authors introduce a new task to evaluate whether machines can rapidly digest new characters using only a few starting scenes, just like a human audience.

Methods

The core task is character guessing: given a movie scene where the speakers' names are anonymized (e.g., "P0" and "P1"), the model must guess which of the movie's main characters are speaking.

Overview of the TOM-IN-AMC dataset creation and meta-learning task setup.

To prevent Large Language Models (LLMs) from "cheating" by simply recognizing the movie from its pre-training data, the authors use name perturbation—swapping real character names with random common names of the same gender.

The authors test standard meta-learning baselines (Prototypical Networks, LEOPARD) and standard In-Context Learning with LLMs. However, their primary contribution is a novel LLM-based approach called ToMPro (ToM Prompting).

The ToMPro (ToM Prompting) framework, which generates mental state descriptions to aid in character identification.

ToMPro works in two stages:

  1. Mental State Generation: It iteratively reads a stream of introductory scenes and prompts GPT-4 to build and update a "mental model" for each character across five specific ToM dimensions: personality, emotions, beliefs, desires, and intentions.
  2. Identity Guessing: When presented with a new, anonymized test scene, the model uses these generated mental descriptions to reason about who is most likely speaking.

Datasets

The authors introduce TOM-IN-AMC (ToM meta-learning Assessment with Movie Characters).

  • It consists of ~1,000 parsed movie scripts sourced from IMSDB.
  • The first 3/5 of each script is used as the "training" set (the few-shot observations to learn the characters).
  • The remaining 2/5 is used for testing (the anonymized scenes).
  • Each movie acts as its own distinct few-shot learning task, making it a highly diverse, cross-domain benchmark compared to existing homogeneous datasets.

Key Insights and Ideas

  • Humans use meta-learning via "Tropes": A human study revealed that annotators frequently solved the task by mapping new characters to established archetypes (e.g., the "Super Villain" or the "Hero") learned from previously watched movies. This proves the task is a valid proxy for meta-learning.
  • Not all ToM dimensions are created equal: Ablation studies on ToMPro showed that modeling all five dimensions helps, but understanding a character's intentions and desires is the most critical for predicting their dialogue. Emotion was the least crucial.
  • LLMs lack "Global" ToM: While GPT-4 is excellent at extracting immediate emotions from a single scene, it struggles to track long-term desires or connect disparate events across a timeline. It often fails to maintain contextual coherence between a character's past mental states and their current actions.
  • Memorization is a massive shortcut: Without name perturbation, GPT-4 could guess characters with nearly 70% accuracy zero-shot (without reading any scenes), proving it had memorized the scripts. Forcing the model to rely purely on behavioral descriptions provides a much more rigorous test of actual reasoning.

Key Results

  • Human Performance: Human annotators achieved an impressive 88.0% accuracy on the task.
  • Best AI Performance: The proposed GPT-4 ToMPro method achieved 66.9% accuracy on the test set, significantly outperforming standard meta-learning baselines like LEOPARD (58.6%) and Prototypical Networks (53.2%).
  • Open Source struggles: Open-source models like Llama-2 and Mistral performed poorly, hovering between 18% and 55%, showing that advanced ToM reasoning is currently limited to frontier models.
  • The Gap: Despite the sophisticated prompting strategy, the best AI system still lags >20% behind human performance.

Limitations and Future Work

The authors acknowledge a few limitations. First, movies are inherently multimodal; audiences use visual and audio cues (facial expressions, tone of voice) to infer mental states, whereas this dataset is strictly text-based. Second, the dataset relies on historical movie scripts, which contain inherent biases (e.g., gender imbalances or outdated tropes).

For future work, the authors suggest focusing on improving LLMs' ability to abstract historical mental states and maintain global coherence over long contexts, which is currently the biggest bottleneck preventing machines from achieving human-level Theory-of-Mind.

Summary updated

Abstract

When reading a story, humans can quickly understand new fictional characters with a few observations, mainly by drawing analogies to fictional and real people they already know. This reflects the few-shot and meta-learning essence of humans' inference of characters' mental states, *i.e.*, theory-of-mind (ToM), which is largely ignored in existing research. We fill this gap with a novel NLP dataset in a realistic narrative understanding scenario, ToM-in-AMC. Our dataset consists of $\sim$1,000 parsed movie scripts, each corresponding to a few-shot character understanding task that requires models to mimic humans' ability of fast digesting characters with a few starting scenes in a new movie. We further propose a novel ToM prompting approach designed to explicitly assess the influence of multiple ToM dimensions. It surpasses existing baseline models, underscoring the significance of modeling multiple ToM dimensions for our task. Our extensive human study verifies that humans are capable of solving our problem by inferring characters' mental states based on their previously seen movies. In comparison, all the AI systems lag $>20\%$ behind humans, highlighting a notable limitation in existing approaches' ToM capabilities. Code and data are available at https://github.com/ShunchiZhang/ToM-in-AMC
BibTeX
@inproceedings{yu2024few,
  author = {Mo Yu and Qiujing Wang and Shunchi Zhang and Yisi Sang and Kangsheng Pu and Zekai Wei and Han Wang and Liyan Xu and Jing Li and Yue Yu and Jie Zhou},
  title = {{Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=ZZ7UKgK4c1},
}
Figure