Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses

Hung-Ting Su, Ya-Ching Hsu, Xudong Lin, Xiang-Qian Shi, Yulei Niu, Han-Yuan Hsu, Hung-yi Lee, Winston H. Hsu

TL;DR

While Large Language Models (LLMs) have mastered the logic of math and the facts of science, they remain surprisingly poor at understanding the "soul" of a story. This paper demonstrates that state-of-the-art models like GPT-4 struggle with narrative reasoning—identifying abstract tropes in movie synopses—and reveals that common techniques like Chain-of-Thought (CoT) can actually trigger hallucinations and make models more vulnerable to deception.

What Problem Does This Paper Address?

We know LLMs are excellent at factual reasoning (e.g., "If A > B and B > C, then A > C"). However, human storytelling relies on narrative reasoning, which is far more abstract. Consider the trope "Heroic Sacrifice." It isn't just the act of a character dying; it requires understanding the character's motivation, the stakes for the "greater good," and the emotional weight of the choice.

Current benchmarks focus heavily on objective data and logical deduction. The authors argue that narrative reasoning represents a distinct frontier because it requires a deep understanding of human behavior, beliefs, and world knowledge that isn't always explicitly stated in the text. As shown in Figure 1, there is a massive performance gap between standard Natural Language Understanding (NLU) tasks and trope identification.

Figure 1: While LLMs have revolutionized NLP rea- soning, surpassing previous supervised learning (SL) methods and even reaching human-level performance on some tasks, their limitations become apparent when tested against the Trope dataset. NLU: Natural Lan- guage Understanding, CS: Commonsense. Check Sec- tion 1 and 2.2 for details.

Methods

The researchers explored several strategies to see if they could bridge this "narrative gap":

  1. Trope-wise Querying: Instead of asking a model to identify all relevant tropes from a list of 95 (a multi-label classification task), they reframed the problem. They queried the model for one specific trope at a time ("Is 'Heroic Sacrifice' present in this plot?"). This allows the model to focus its attention on a single concept.
  2. Chain-of-Thought (CoT) Prompting: They asked models to provide a definition of the trope, explain their reasoning, and cite specific paragraphs as evidence before giving a final "yes" or "no."
  3. Adversarial Injection: To test if models were truly reasoning or just matching keywords, the authors injected trope-related keywords into synopses where the trope did not actually occur. For example, they might add a sentence about a character "lying" to see if the model would falsely flag the "Blatant Lies" trope, even if the narrative context didn't support it.
  4. Fine-Tuning: They fine-tuned LLaMa-2-7B using Low-Rank Adaptation (LoRA) to see if specialized training could overcome the limitations of general-purpose prompting.

Datasets

The study utilizes the TiMoS (Trope in Movie Synopses) dataset. This dataset contains movie synopses labeled with 95 different tropes. Unlike simple sentiment analysis, TiMoS requires connecting disparate plot points. For instance, a character might mention a love for freedom in the first act, which only becomes relevant to their "Heroic Sacrifice" in the final scene.

Figure 2: Trope in Movie Synopses (TiMoS) requires the abstraction of narrative reasoning beyond physical presentation. For example, themes of justice (red block) or sacrifice (blue block) extend beyond death. TiMoS also explores connections between seemingly unrelated ideas, such as Batman’s departure and his efforts to save the city (blue block).

Key Insights and Ideas

  • The CoT Paradox: In most domains, "thinking step-by-step" improves performance. Here, CoT actually decreased GPT-4's precision. The model would often "reason" its way into a hallucination, fabricating connections between unrelated plot points to justify a trope's presence.
  • Decomposition is Key: The most significant performance boost came not from better reasoning, but from simplifying the task. Trope-wise querying improved the F1 score by 11.8 points, suggesting that LLMs struggle to hold multiple complex narrative concepts in "working memory" simultaneously.
  • Keyword Over-reliance: The Adversarial Injection experiments revealed a major vulnerability. LLMs, especially when using CoT, were easily misled by surface-level keywords. If a synopsis contained the word "obvious," the model was far more likely to hallucinate the "Blatant Lies" trope, even if no lie occurred.
  • Blind Spots: Certain tropes are "easy" for LLMs because they rely on explicit actions (like "Driven to Suicide"), while others are "hard" because they require understanding irony or subtext (like "Stealth Pun" or "Jerkass Has a Point").

Key Results

  • State-of-the-Art Surpassed: By using trope-wise querying, GPT-4 achieved an F1 score that surpassed previous supervised learning models (reaching ~27.1 F1 compared to the previous SOTA of 25.0). However, this is still far below human performance (~64.9 F1).
  • CoT Performance: For ChatGPT, CoT helped (boosting F1 from 20.57 to 24.99), but for the more advanced GPT-4, it caused a decline (from 29.59 down to 27.91).
  • Adversarial Success: The adversarial attacks were devastatingly effective. For the "Blatant Lies" trope, the attack success rate jumped from 63.33% (Base) to a staggering 96.67% when CoT was enabled.

Limitations and Future Work

The authors acknowledge that because GPT-4 and ChatGPT are closed-source, their results might not be perfectly reproducible as the models are updated. Furthermore, movie synopses can contain sensitive or offensive content, which might trigger safety filters or bias the results.

Future research needs to focus on helping LLMs grasp holistic context. Current models tend to over-interpret fragments of text rather than understanding how those fragments fit into a larger narrative arc. Developing models that can resist keyword-based adversarial attacks will be crucial for building trustworthy narrative AI.

Summary updated

Abstract

Large language models (LLMs) equipped with chain-of-thoughts (CoT) prompting have shown significant multi-step reasoning capabilities in factual content like mathematics, commonsense, and logic. However, their performance in narrative reasoning, which demands greater abstraction capabilities, remains unexplored. This study utilizes tropes in movie synopses to assess the abstract reasoning abilities of state-of-the-art LLMs and uncovers their low performance. We introduce a trope-wise querying approach to address these challenges and boost the F1 score by 11.8 points. Moreover, while prior studies suggest that CoT enhances multi-step reasoning, this study shows CoT can cause hallucinations in narrative content, reducing GPT-4's performance. We also introduce an Adversarial Injection method to embed trope-related text tokens into movie synopses without explicit tropes, revealing CoT's heightened sensitivity to such injections. Our comprehensive analysis provides insights for future research directions.
BibTeX
@inproceedings{su2024unveiling,
  author = {Hung-Ting Su and Ya-Ching Hsu and Xudong Lin and Xiang-Qian Shi and Yulei Niu and Han-Yuan Hsu and Hung-yi Lee and Winston H. Hsu},
  title = {{Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=ksRQH9nQcD},
}
Figure