TL;DR
This paper introduces "True Detective," a new benchmark for evaluating deep abductive reasoning in Large Language Models (LLMs) using long-form detective mysteries. Even state-of-the-art models like GPT-4 struggle significantly, highlighting a substantial gap between current LLM capabilities and human reasoning in complex, multi-step inference tasks.
What Problem Does This Paper Address?
LLMs have shown impressive performance on many NLP tasks, including reasoning. However, existing benchmarks often rely on short, common-sense observations, which LLMs can solve with relative ease. This paper argues for the need for more challenging benchmarks that require deeper, more complex reasoning. Specifically, they focus on abductive reasoning – the process of finding the most plausible explanation for a set of observations. While abductive reasoning is crucial for tasks like detective work, it hasn't been adequately explored as a benchmark for LLMs. The authors aim to fill this gap by creating a benchmark that mimics the complexity of solving a crime, requiring LLMs to sift through lengthy narratives and make intricate inferences.
Methods
The researchers evaluated LLMs using three prompting strategies:
- Vanilla: A direct prompt asking for the final answer after presenting the mystery and options.
- CoT (Chain-of-Thought): The model is prompted to generate a step-by-step reasoning process before providing the final answer. This is a standard technique to improve LLM reasoning.
- Golden CoT: This is an ablation study where the LLM is provided with a pre-written, human-authored "golden" explanation (Chain-of-Thought) along with the mystery. This simplifies the task by removing the need for the LLM to generate its own reasoning steps, allowing researchers to assess its ability to perform the final inference given a correct reasoning path.
The models evaluated were GPT-3.5 (specifically, InstructGPT-3.5 models like FeedME and PPO) and GPT-4.
Datasets
The primary dataset used is True Detective, a newly constructed benchmark consisting of 191 long-form detective mysteries. These mysteries are sourced from the "5 Minute Mystery" platform and are characterized by:
- Length: Averaging around 1200 words per mystery, with some extending to 2000 words.
- Structure: Each mystery includes multiple suspects and a multiple-choice question to identify the guilty party or solve a related puzzle.
- Human Performance: The dataset is grounded in human performance data from the platform, where the average human solve rate is 47%, and top human solvers achieve over 80% accuracy.
This dataset is significantly longer and more complex than previous abductive reasoning benchmarks like ROCStories or SWAG.
Key Insights and Ideas
- LLMs Struggle with Deep Abductive Reasoning: The core finding is that even advanced LLMs like GPT-4 are far from human-level performance on this complex reasoning task. This suggests that current LLM architectures and training methods, while powerful, may not fully capture the nuances of deep, multi-step abductive inference.
- Length and Complexity Matter: The long-form nature of the mysteries, requiring the integration of information spread across many sentences and characters, poses a significant challenge. This is unlike simpler tasks that rely on local context.
- Chain-of-Thought Helps, But Not Enough for GPT-3.5: While CoT prompting improves performance slightly for GPT-3.5, it doesn't bridge the gap to human performance. This indicates that GPT-3.5 might lack the underlying reasoning capacity to effectively utilize CoT for such complex problems. GPT-4 shows more improvement with CoT.
- Golden CoT Reveals Specific Weaknesses: The Golden CoT experiment is particularly insightful. Even when provided with the correct reasoning path, GPT-3.5 models still struggle to make the final inference correctly. GPT-4 performs much better with Golden CoT, suggesting its difficulty lies more in generating the reasoning than in the final inference step when guided.
Key Results
- Vanilla & CoT Performance:
- GPT-3.5 models (FeedME and PPO) achieve accuracies around 26-29%, which is barely above random guessing (24%).
- GPT-4 achieves around 38% accuracy, which is halfway between random guessing and the average human baseline.
- Golden CoT Performance:
- GPT-3.5 models improve to 46-63% accuracy, showing that providing the reasoning path helps, but they still fall short of top human performance.
- GPT-4 achieves 83% accuracy with Golden CoT, nearing the performance of top human solvers.
- Human Baseline: Average human solve rate is 47%, with top human solvers reaching 80-90%.
Limitations and Future Work
- Model Accessibility: The evaluation focused on leading-edge GPT models whose internal details are not publicly available.
- Dataset Subjectivity: The authors acknowledge that with an average human solve rate of 47%, some mysteries might be ill-defined or overly complicated.
- Future Research: The authors suggest extending this study to other LLMs like PaLM and LLAMA. They also plan to explore how to improve LLM performance on this benchmark, potentially through more advanced training techniques or by investigating the specific reasoning steps that models find most challenging. The benchmark itself is intended to spur further research into LLM abductive reasoning capabilities.
Summary updated