Abstract
Large language models (LLMs) often struggle with "situated" reasoning tasks that require an understanding of social context, ethics, and human emotions. We present DREAM (Dynamically wRitten ElAborations to iMprove question Manswering), a framework that enhances situational question-answering by first generating a structured "scene elaboration." By explicitly modeling motivations, emotions, social norms, and consequences, DREAM provides off-the-shelf QA models with the necessary context to reason more effectively. Our results demonstrate significant accuracy improvements across multiple benchmarks, including ETHICS-CS and Social IQA, without requiring retraining of the underlying reasoning models.
Introduction
While large language models have become remarkably proficient at general question-answering, they frequently stumble on scenarios involving complex social interactions or everyday common sense. This "situated" reasoning requires more than just pattern matching; it requires a coherent mental picture of the scenario. Cognitive science suggests that humans answer such questions by forming internal mental models. Current LLMs, however, often lack this "imagination," leading to errors in judgment.
Previous research into improving reasoning, such as "Chain of Thought" prompting, primarily focuses on the logical path to an answer. In contrast, the DREAM framework focuses on elaborating the situation itself, providing the model with the social and emotional context required for effective reasoning.
The DREAM Framework
DREAM is a T5-11B model specifically trained to act as a "scene imagination" module. It generates a Scene Elaboration (SE) that represents a situation across four key social dimensions:
- Motivation (M): The character's intent prior to the situation.
- Emotion (E): The character's feelings following the situation.
- Rule of Thumb (ROT): The social norm or moral rule governing the action.
- Consequence (Con): The likely outcome of the action.
The workflow is modular: DREAM takes a situation $S$ and question $Q$ as input to produce the SE components. A separate, off-the-shelf QA model (e.g., Macaw or UnifiedQA) then receives the original input plus the DREAM-generated SE to produce the final answer. This dataset-neutral approach allows DREAM to be plugged into various models without task-specific fine-tuning.
Evaluation and Results
DREAM was trained on general commonsense resources (Story Commonsense, Social Chemistry, and Moral Stories) and evaluated on three distinct QA tasks: ETHICS-CS, CODAH, and Social IQA.
Quality of Elaboration
Human evaluations indicate that DREAM's scene elaborations are significantly superior to those generated by zero-shot models like Macaw-11B. As shown in the table below, DREAM achieves higher accuracy, usefulness, and consistency across all tested datasets.
Performance Gains
Integrating DREAM's elaborations led to consistent accuracy improvements for the Macaw-11B model:
- ETHICS-CS: +2.8% overall improvement (and +2.1% on the "hard" subset).
- Social IQA: +4.2% improvement.
Notably, the structured SE context proved more effective than simply fine-tuning the QA model on the raw training data used for DREAM. This highlights the value of the four-dimensional representation (M, E, ROT, Con) in providing a uniquely valuable signal for social reasoning.
Discussion and Limitations
The DREAM framework demonstrates that "soft" social constructs are often more effective for situational QA than formal logic. Furthermore, the modular nature of the system provides a significant advantage for smaller models, helping them bridge the performance gap with much larger counterparts.
However, challenges remain. In approximately 12% of cases, DREAM may "hallucinate" incorrect details, which can occasionally mislead the downstream QA model. Additionally, the current four dimensions are optimized for social scenarios; expanding DREAM to other domains like science or mathematics would require identifying new relevant dimensions, such as spatial or temporal relations.
Conclusion
DREAM offers a portable, effective method for improving situational reasoning in LMs by mimicking human-like scene imagination. By separating the "imagination" of a scene from the "reasoning" about an answer, the framework provides a robust path toward more socially aware artificial intelligence.
Summary updated