DREAM: Improving Situational QA by First Elaborating the Situation

Yuling Gu, Bhavana Dalvi, Peter Clark

Abstract

Large language models (LLMs) often struggle with "situated" reasoning tasks that require an understanding of social context, ethics, and human emotions. We present DREAM (Dynamically wRitten ElAborations to iMprove question Manswering), a framework that enhances situational question-answering by first generating a structured "scene elaboration." By explicitly modeling motivations, emotions, social norms, and consequences, DREAM provides off-the-shelf QA models with the necessary context to reason more effectively. Our results demonstrate significant accuracy improvements across multiple benchmarks, including ETHICS-CS and Social IQA, without requiring retraining of the underlying reasoning models.

Introduction

While large language models have become remarkably proficient at general question-answering, they frequently stumble on scenarios involving complex social interactions or everyday common sense. This "situated" reasoning requires more than just pattern matching; it requires a coherent mental picture of the scenario. Cognitive science suggests that humans answer such questions by forming internal mental models. Current LLMs, however, often lack this "imagination," leading to errors in judgment.

Previous research into improving reasoning, such as "Chain of Thought" prompting, primarily focuses on the logical path to an answer. In contrast, the DREAM framework focuses on elaborating the situation itself, providing the model with the social and emotional context required for effective reasoning.

The DREAM Framework

DREAM is a T5-11B model specifically trained to act as a "scene imagination" module. It generates a Scene Elaboration (SE) that represents a situation across four key social dimensions:

  1. Motivation (M): The character's intent prior to the situation.
  2. Emotion (E): The character's feelings following the situation.
  3. Rule of Thumb (ROT): The social norm or moral rule governing the action.
  4. Consequence (Con): The likely outcome of the action.
The DREAM framework workflow: Situation S and Question Q are processed by DREAM to generate a Scene Elaboration (SE), which then informs the QA model.

The workflow is modular: DREAM takes a situation $S$ and question $Q$ as input to produce the SE components. A separate, off-the-shelf QA model (e.g., Macaw or UnifiedQA) then receives the original input plus the DREAM-generated SE to produce the final answer. This dataset-neutral approach allows DREAM to be plugged into various models without task-specific fine-tuning.

Evaluation and Results

DREAM was trained on general commonsense resources (Story Commonsense, Social Chemistry, and Moral Stories) and evaluated on three distinct QA tasks: ETHICS-CS, CODAH, and Social IQA.

Quality of Elaboration

Human evaluations indicate that DREAM's scene elaborations are significantly superior to those generated by zero-shot models like Macaw-11B. As shown in the table below, DREAM achieves higher accuracy, usefulness, and consistency across all tested datasets.

Comparison of Scene Elaboration quality between Macaw (zero-shot) and DREAM across different datasets.

Performance Gains

Integrating DREAM's elaborations led to consistent accuracy improvements for the Macaw-11B model:

  • ETHICS-CS: +2.8% overall improvement (and +2.1% on the "hard" subset).
  • Social IQA: +4.2% improvement.

Notably, the structured SE context proved more effective than simply fine-tuning the QA model on the raw training data used for DREAM. This highlights the value of the four-dimensional representation (M, E, ROT, Con) in providing a uniquely valuable signal for social reasoning.

Discussion and Limitations

The DREAM framework demonstrates that "soft" social constructs are often more effective for situational QA than formal logic. Furthermore, the modular nature of the system provides a significant advantage for smaller models, helping them bridge the performance gap with much larger counterparts.

However, challenges remain. In approximately 12% of cases, DREAM may "hallucinate" incorrect details, which can occasionally mislead the downstream QA model. Additionally, the current four dimensions are optimized for social scenarios; expanding DREAM to other domains like science or mathematics would require identifying new relevant dimensions, such as spatial or temporal relations.

Conclusion

DREAM offers a portable, effective method for improving situational reasoning in LMs by mimicking human-like scene imagination. By separating the "imagination" of a scene from the "reasoning" about an answer, the framework provides a robust path toward more socially aware artificial intelligence.

Summary updated

Abstract

When people answer questions about a specific situation, e.g., "I cheated on my mid-term exam last week. Was that wrong?", cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the "scene". To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further finetuning the QA system on DREAM's training data. These results suggest that adding focused elaborations about a situation can improve a system's reasoning about it, and may serve as an effective way of injecting new scenario based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters. We make our dataset and model publicly available at https://github.com/allenai/dream.
BibTeX
@inproceedings{gu2022dream,
  author = {Yuling Gu and Bhavana Dalvi and Peter Clark},
  title = {{DREAM: Improving Situational QA by First Elaborating the Situation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=hnLiNvG95s},
}
Figure