Query-focused Scenario Construction

Su Wang, Greg Durrett, Katrin Erk

Abstract

In the wake of major news events, reports often present a chaotic mixture of conflicting accounts and irrelevant details. This paper introduces a query-based system designed to reconstruct a single, coherent "scenario" from this noise by iteratively selecting compatible events. By integrating an insertion-sort mechanism with word-level relation networks, the proposed model significantly outperforms traditional clustering baselines on a new, human-curated dataset of real-world news controversies.

Introduction: The Challenge of Information Chaos

During the early stages of a high-profile news event—such as the disappearance of journalist Jamal Khashoggi—the information landscape is notoriously fragmented. Conflicting reports may emerge simultaneously: one source might claim a subject left a building safely, while another alleges a state-sponsored crime. For both humans and AI, synthesizing a consistent narrative from these contradictory documents is a significant challenge.

Traditional research has largely focused on "narrative clustering," which attempts to identify all possible stories within a dataset. This paper instead tackles query-focused scenario construction. Given a specific query (e.g., "Jamal Khashoggi was murdered"), the goal is to extract only the sentences that form a compatible, chronological story supporting that specific claim. The authors frame this as a "one-class clustering" problem, where the system must distinguish a target scenario from a sea of distractors and irrelevant information.

Overview of the query-focused scenario construction task, illustrating how a target scenario is extracted from a mixture of sentences based on a specific query.

Methodology

The authors propose an iterative model that builds a scenario sentence by sentence, exploring three architectural variations:

  1. Compatibility-Attention (COMP): A neural baseline using bilinear attention to score the fit between a candidate sentence and the existing "scenario-in-construction."
  2. Insertion-sort Selection (COMP-INS): A key innovation where the model determines not just if a sentence is compatible, but exactly where it fits in the timeline. By forcing the model to find a specific insertion point, it learns to leverage the logical and chronological flow of a narrative to improve selection accuracy.
  3. Relation Networks (COMP-INS-RN): This version adds a "Relation Network" to capture word-to-word connections. It identifies subtle cues like anaphoric references (e.g., "he" linking to a previously mentioned name) or semantic links between events (e.g., "died" relating to "murdered").

These models are trained using marginal log-likelihood, allowing the system to learn the most effective way to build a scenario without being restricted to a single rigid path during training.

Datasets and Evaluation

To evaluate the system, the authors developed two distinct types of data:

  • Human100: A high-quality, human-curated set of 100 real-world news topics. Each topic includes two mutually exclusive scenarios (e.g., conflicting theories on the downing of flight MH-17). This dataset is particularly difficult because competing scenarios often share high topical similarity, rendering simple keyword matching ineffective.
  • Synthetic Data: To provide sufficient training volume, the authors created "story salads" using the New York Times (NYT) and ROCStories corpora, mixing target article sentences with related distractors and random noise.
The data collection and annotation pipeline for the Human100 dataset, showing the transition from web search to human-sorted scenarios.

Key Results and Insights

The experimental results highlight several critical findings:

  • Superior Performance: The best-performing model, COMP-INS-RN, achieved an F1 score of 0.70 on the Human100 dataset, significantly surpassing a strong pairwise baseline (0.56) and simple embedding models (0.49).
  • Ordering as a Proxy for Compatibility: A major insight is that training a model to order events improves its ability to select them. The structural constraints of narrative flow provide a powerful signal for identifying which events belong together.
  • Generalization: Models trained on NYT news data generalized well to ROCStories (fictional narratives), suggesting the system learns general principles of event compatibility rather than just memorizing specific news entities.
  • The Importance of Nuance: The addition of the Relation Network was crucial for moving beyond bag-of-words similarity, allowing the model to recognize logical bridges created by specific verbs or pronouns.

Conclusion

While the model represents a significant step forward, it still trails the human benchmark of 0.82 F1. Future work is needed to address "dynamic stopping"—the ability for a model to decide on its own when a story is complete—and to handle more nuanced conflicts where different scenarios represent varying perspectives rather than strict contradictions. Nonetheless, the use of insertion-sort mechanisms provides a robust framework for extracting clarity from the noise of modern news cycles.

Summary updated

BibTeX
@inproceedings{wang2019query,
  author = {Su Wang and Greg Durrett and Katrin Erk},
  title = {{Query-focused Scenario Construction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=M_Z8XPPjTD1},
}

Loading related content...

Figure