TL;DR
While Large Language Models (LLMs) have mastered the logic of math and the facts of science, they remain surprisingly poor at understanding the "soul" of a story. This paper demonstrates that state-of-the-art models like GPT-4 struggle with narrative reasoning—identifying abstract tropes in movie synopses—and reveals that common techniques like Chain-of-Thought (CoT) can actually trigger hallucinations and make models more vulnerable to deception.
What Problem Does This Paper Address?
We know LLMs are excellent at factual reasoning (e.g., "If A > B and B > C, then A > C"). However, human storytelling relies on narrative reasoning, which is far more abstract. Consider the trope "Heroic Sacrifice." It isn't just the act of a character dying; it requires understanding the character's motivation, the stakes for the "greater good," and the emotional weight of the choice.
Current benchmarks focus heavily on objective data and logical deduction. The authors argue that narrative reasoning represents a distinct frontier because it requires a deep understanding of human behavior, beliefs, and world knowledge that isn't always explicitly stated in the text. As shown in Figure 1, there is a massive performance gap between standard Natural Language Understanding (NLU) tasks and trope identification.
Methods
The researchers explored several strategies to see if they could bridge this "narrative gap":
- Trope-wise Querying: Instead of asking a model to identify all relevant tropes from a list of 95 (a multi-label classification task), they reframed the problem. They queried the model for one specific trope at a time ("Is 'Heroic Sacrifice' present in this plot?"). This allows the model to focus its attention on a single concept.
- Chain-of-Thought (CoT) Prompting: They asked models to provide a definition of the trope, explain their reasoning, and cite specific paragraphs as evidence before giving a final "yes" or "no."
- Adversarial Injection: To test if models were truly reasoning or just matching keywords, the authors injected trope-related keywords into synopses where the trope did not actually occur. For example, they might add a sentence about a character "lying" to see if the model would falsely flag the "Blatant Lies" trope, even if the narrative context didn't support it.
- Fine-Tuning: They fine-tuned LLaMa-2-7B using Low-Rank Adaptation (LoRA) to see if specialized training could overcome the limitations of general-purpose prompting.
Datasets
The study utilizes the TiMoS (Trope in Movie Synopses) dataset. This dataset contains movie synopses labeled with 95 different tropes. Unlike simple sentiment analysis, TiMoS requires connecting disparate plot points. For instance, a character might mention a love for freedom in the first act, which only becomes relevant to their "Heroic Sacrifice" in the final scene.
Key Insights and Ideas
- The CoT Paradox: In most domains, "thinking step-by-step" improves performance. Here, CoT actually decreased GPT-4's precision. The model would often "reason" its way into a hallucination, fabricating connections between unrelated plot points to justify a trope's presence.
- Decomposition is Key: The most significant performance boost came not from better reasoning, but from simplifying the task. Trope-wise querying improved the F1 score by 11.8 points, suggesting that LLMs struggle to hold multiple complex narrative concepts in "working memory" simultaneously.
- Keyword Over-reliance: The Adversarial Injection experiments revealed a major vulnerability. LLMs, especially when using CoT, were easily misled by surface-level keywords. If a synopsis contained the word "obvious," the model was far more likely to hallucinate the "Blatant Lies" trope, even if no lie occurred.
- Blind Spots: Certain tropes are "easy" for LLMs because they rely on explicit actions (like "Driven to Suicide"), while others are "hard" because they require understanding irony or subtext (like "Stealth Pun" or "Jerkass Has a Point").
Key Results
- State-of-the-Art Surpassed: By using trope-wise querying, GPT-4 achieved an F1 score that surpassed previous supervised learning models (reaching ~27.1 F1 compared to the previous SOTA of 25.0). However, this is still far below human performance (~64.9 F1).
- CoT Performance: For ChatGPT, CoT helped (boosting F1 from 20.57 to 24.99), but for the more advanced GPT-4, it caused a decline (from 29.59 down to 27.91).
- Adversarial Success: The adversarial attacks were devastatingly effective. For the "Blatant Lies" trope, the attack success rate jumped from 63.33% (Base) to a staggering 96.67% when CoT was enabled.
Limitations and Future Work
The authors acknowledge that because GPT-4 and ChatGPT are closed-source, their results might not be perfectly reproducible as the models are updated. Furthermore, movie synopses can contain sensitive or offensive content, which might trigger safety filters or bias the results.
Future research needs to focus on helping LLMs grasp holistic context. Current models tend to over-interpret fragments of text rather than understanding how those fragments fit into a larger narrative arc. Developing models that can resist keyword-based adversarial attacks will be crucial for building trustworthy narrative AI.
Summary updated