The RIP Corpus of Collaborative Hypothesis-Making

Ella Schad, Jacky Visser, Chris ReedCOLING 2024

TL;DR

This paper introduces the "Resolving Investigative hyPotheses" (RIP) corpus, a novel dataset for studying collaborative hypothesis-making in a simulated murder mystery game. By analyzing group dialogues annotated with Inference Anchoring Theory, the authors shed light on emergent group dynamics, individual roles, and common pitfalls like confirmation bias in problem-solving scenarios.

What Problem Does This Paper Address?

The paper identifies a significant gap in research at the intersection of hypothesis generation and collaborative problem-solving. While there's work on how individuals form hypotheses and how groups solve problems, there's a lack of understanding of how hypothesis-making specifically unfolds within collaborative settings, especially in investigative contexts. Real-world investigations are complex, making it difficult to isolate and study the hypothesis-generation process. To address this, the authors create a controlled, yet engaging, environment – a murder mystery game – to facilitate the study of collaborative hypothesis-making.

Methods

The core methodology involves creating and annotating a corpus of group dialogues. The researchers used a commercially available murder mystery game, "Death at the Dive Bar," as the simulated investigative environment. Three groups of three participants played the game, and their interactions were recorded and transcribed. The transcripts were then meticulously annotated using the Inference Anchoring Theory (IAT) framework. IAT models arguments and their relationships within dialogues, capturing both propositional content and discourse structure. This detailed annotation allows for the analysis of individual contributions, group strategies, and the specific types of arguments and discourse moves made during hypothesis generation and evaluation.

Datasets

The paper introduces the Resolving Investigative hyPotheses (RIP) corpus. This corpus comprises 49,000 words from the transcribed dialogues of three groups playing a murder mystery game. The corpus is annotated according to the Inference Anchoring Theory (IAT) framework, detailing argumentative and discourse structures. This dataset is made publicly accessible.

Key Insights and Ideas

  • Controlled Environment for Hypothesis Study: Using a murder mystery game provides a finite and structured environment that limits the number of possible hypotheses, making it easier to compare group strategies and analyze hypothesis generation in a controlled manner.
  • Emergent Roles in Collaboration: The analysis reveals that participants naturally adopt different roles within the group, such as "Captain" (active and socially responsible) and "Free-rider" (contributes when prompted), even without pre-assigned roles.
  • Confirmation Bias is a Major Pitfall: Two of the three groups failed to solve the mystery, with confirmation bias identified as a significant factor. This involved participants focusing on evidence that supported their initial suspicions and ignoring or downplaying contradictory information.
  • Strategic Hypothesis Rejection is Key: The successful group demonstrated a strategy of systematically considering and rejecting suspects before settling on the culprit, which helped them avoid the confirmation bias that plagued the other groups.
  • Argumentation Patterns Differ in Success: The successful group exhibited higher rates of arguing and question-asking/answering, suggesting a more robust and critical engagement with the evidence and hypotheses.

Key Results

  • Three Groups, One Success: Out of three groups, only one successfully solved the murder mystery game.
  • Varied Dialogue Lengths: The groups spent differing amounts of time on the task, with the successful group spending the longest (2 hours and 40 minutes) and contributing the most words (26,279).
  • Confirmation Bias in Action: Group 1 jumped to conclusions about Chris and Cherie, while Group 2 was influenced by a desire for a "twist" and focused on Carmen, a non-suspect. Both groups exhibited confirmation bias.
  • Successful Group's Strategy: Group 3 systematically considered and rejected suspects (Joan, Donna, Chris) before identifying Cherie as the culprit, demonstrating a more thorough and less biased approach.
  • Inter-Annotator Agreement: The corpus achieved a substantial inter-annotator agreement score of κ = 0.68, indicating high reliability in the complex IAT annotation process.

Limitations and Future Work

  • Synthetic Domain Limitations: The authors acknowledge that a synthetic game environment, while useful, cannot perfectly replicate the complexities and nuances of real-world investigations. Real cases lack neat conclusions and the element of enjoyment.
  • Future Work - Prescribed Roles: Future research could explore prescribing specific roles to participants to see how this influences hypothesis-making dynamics.
  • Future Work - Critical Questioning: Investigating the impact of specific types of critical questions on hypothesis formation and evaluation is another avenue for future work.
  • Future Work - LLM Applications: The corpus is intended to be a resource for training Argument Mining systems and Large Language Models (LLMs), which currently struggle with inference detection.

Summary updated

BibTeX
@inproceedings{schad2024rip,
  author = {Ella Schad and Jacky Visser and Chris Reed},
  title = {{The RIP Corpus of Collaborative Hypothesis-Making}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure