Abstract
The news coverage of events often contains not one but multiple incompatible accounts of what happened. We develop a query-based system that extracts compatible sets of events (scenarios) from such data, formulated as one-class clustering. Our system incrementally evaluates each event's compatibility with already selected events, taking order into account. We use synthetic data consisting of article mixtures for scalable training and evaluate our model on a new human-curated dataset of scenarios about real-world news topics. Stronger neural network models and harder synthetic training settings are both important to achieve high performance, and our final scenario construction system substantially outperforms baselines based on prior work.
Introduction
In the wake of major news events, media coverage often presents a "story salad"—a chaotic mixture of contradictory accounts and irrelevant details. For instance, early reports on the death of Jamal Khashoggi contained multiple incompatible narratives: one scenario posited he was murdered within the consulate, while another suggested he departed the building unharmed.
The challenge of Query-Focused Scenario Construction involves automatically disentangling these threads. Given a specific query (e.g., "Khashoggi was murdered"), the system must iteratively select sentences that form a consistent, readable narrative while filtering out distractors and contradictory claims. This task is particularly difficult because competing scenarios often share significant topical overlap, mentioning the same entities and locations.
Methodology
The authors frame scenario construction as a one-class clustering problem, identifying members of a target scenario within a noisy corpus. They propose an iterative framework that builds a narrative sentence by sentence.
Model Architectures
Three increasingly sophisticated neural architectures were evaluated:
- Compatibility-Attention (COMP): Utilizes a global attention mechanism to assess the fit of a candidate sentence against the existing "scenario-in-construction."
- Insertion-Sort Selection (COMP-INS): This model treats selection as an ordering task. It evaluates a candidate based on its optimal insertion point within the current narrative, leveraging the idea that logical flow is a strong indicator of scenario compatibility.
- Relation Networks (COMP-INS-RN): To capture fine-grained semantic connections, a Relation Network module compares word-level tokens between the candidate and the current scenario. This allows the model to identify specific event-to-event or entity-to-entity links.
Data and Training
To address the scarcity of human-curated data, the researchers employed two primary datasets:
- Synthetic Proxy Data: Models were trained at scale using mixtures of articles from the New York Times (NYT) and ROCStories. Sentences from a single article formed the "target scenario," mixed with distractors from other articles.
- Human100: A new evaluation dataset consisting of 100 real-world news topics, each with two mutually exclusive scenarios. This dataset is designed to be challenging, with high topical similarity between target and distractor scenarios.
Results and Discussion
The final COMP-INS-RN model significantly outperformed previous baselines. On synthetic NYT data, it achieved an F1 score of 0.84, compared to 0.53 for the PAIRWISE baseline. On the more difficult Human100 dataset, the model reached an F1 of 0.70.
Key Insights
- Ordering as a Proxy for Selection: Forcing the model to determine a logical narrative position for a sentence improves its ability to select the correct content. If a sentence lacks a logical place in the story, it likely does not belong.
- Training Robustness: Training on "hybrid" mixtures (target story + coherent distractor + random noise) proved more effective than simpler training regimes.
- Cross-Domain Generalization: Models trained on news data generalized well to fictional stories, indicating the learning of fundamental narrative principles rather than just keyword memorization.
Conclusion
While the COMP-INS-RN model represents a significant advancement, challenges remain. The system can still be misled by superficial word overlap in contradictory events. Future work will explore deeper logical reasoning and dynamic termination strategies to handle scenarios of varying lengths.
Summary updated