Abstract
Understanding fictional characters quickly is a hallmark of human social intelligence, rooted in Theory-of-Mind (ToM). We introduce TOM-IN-AMC, a novel benchmark that evaluates whether AI can mimic this "few-shot" social learning by identifying anonymized characters in movie scripts. We also propose ToMPro (ToM Prompting), a two-stage framework for iterative character modeling. Our findings reveal that even advanced models like GPT-4 lag significantly behind human performance, particularly when memorization shortcuts are removed via name perturbation.
Introduction
Humans are adept at "reading the room." Upon starting a new movie, we do not require a full biography to understand a protagonist; within a few scenes, we infer their personality, goals, and likely actions. This ability to infer internal mental states is known as Theory-of-Mind (ToM). While ToM has been studied in synthetic environments, existing character-understanding tasks often provide models with exhaustive histories, which does not reflect the rapid, few-shot nature of human social reasoning. We argue that humans utilize meta-learning, applying prior knowledge of character archetypes to understand new individuals.
The Character Guessing Task
The core of our assessment is a character guessing task. Given a scene where character names are replaced with generic IDs (e.g., P0, P1), the model must identify the characters based on a limited number of preceding "training" scenes from the same movie.
Methods: ToMPro
To address this challenge, we developed ToMPro (ToM Prompting) for Large Language Models (LLMs). This approach consists of two primary stages:
- Iterative Modeling: The model processes a sequence of scenes, maintaining and updating a "mental model" for each character across five dimensions: personality, emotions, beliefs, desires, and intentions.
- Identity Guessing: The model leverages these generated mental descriptions to identify anonymized characters in a novel test scene.
To prevent models from relying on pre-training memorization, we employ Name Perturbation, replacing iconic names with common ones while preserving gender.
The TOM-IN-AMC Dataset
We present TOM-IN-AMC, the first benchmark for meta-learning of ToM in realistic narrative settings. Derived from approximately 1,000 movie scripts from the Internet Movie Script Database (IMSDB), the dataset includes 3,837 characters and over 74,000 scenes. Unlike previous datasets limited to specific shows, TOM-IN-AMC spans diverse genres, including Drama, Thriller, and Comedy, providing a robust testbed for generalized social reasoning.
Key Insights and Results
Our experiments yielded several critical findings:
- The Human-AI Gap: Humans achieved 88.0% accuracy, while the best-performing AI (ToMPro with GPT-4) reached 66.9%. This 20% gap underscores the difficulty of rapid social nuance for current AI.
- Archetypal Reasoning: Human participants reported using "archetypes" or direct analogies to characters in other films, confirming that character understanding is a meta-learning process.
- Dimensions of ToM: Ablation studies indicated that Desire and Intention are the most critical dimensions for character identification, whereas Emotion proved the least helpful.
- Memorization vs. Reasoning: GPT-4's performance dropped significantly when name perturbation was applied, indicating a heavy reliance on memorized training data rather than genuine social reasoning.
Conclusion and Future Work
While ToMPro advances the state-of-the-art in few-shot character understanding, several challenges remain. Models still struggle with "global understanding," often failing to connect immediate reactions to long-term goals. Furthermore, the dataset reflects historical Hollywood gender biases, which models may exploit as shortcuts. Future work should focus on distilling ToM capabilities into smaller models and improving long-term contextual coherence.
Summary updated