Variation in Situational Awareness Information due to Selection of Data Source, Summarization Method, and Method Implementation

Maria Janina Sarol, Ly Dinh, Jana DiesnerIcwsm 2021

Introduction

During a large-scale disaster, such as the 2010 Haiti earthquake, the sheer volume of digital information can overwhelm first responders and decision-makers. To manage this influx, researchers and practitioners increasingly rely on automated text mining tools—including topic modeling, extractive summarization, and needs detection—to extract "Situational Awareness Information" (SII). However, the reliability of these tools is often taken for granted. This study investigates how the choice of data source, the specific summarization algorithm, and even minor implementation details can significantly skew the resulting "truth" presented to responders.

Data and Methodology

The researchers employed a rigorous "one-factor-at-a-time" experimental design to isolate the variables that cause variation in SII. They focused on three primary computational tasks: Latent Dirichlet Allocation (LDA) for topic modeling, extractive summarization (comparing the disaster-specific COWTS method against the general-purpose SumBasic), and needs detection using word2vec embeddings to identify urgent resource requirements.

Table of datasets used in the study, including the primary Haiti earthquake corpus and validation sets.

The study primarily analyzed data from the 2010 Haiti Earthquake across four distinct sources: blogs, news articles, Twitter, and First Responder (FR) accounts. To ensure the findings were not unique to a single event, the methods were validated against Twitter data from 11 other major disasters, including the 2015 Nepal earthquake and the 2014 Ebola crisis.

Thematic Divergence Across Sources

A central finding of the research is that data sources are not interchangeable; they exist in "silos" with distinct thematic priorities. Twitter is predominantly action-oriented, with a heavy focus on fundraising and donation collection. In contrast, news articles provide a more situational overview, focusing on international aid, infrastructure, and logistics. Blogs tend to be more narrative and emotional, often discussing religious perspectives or the personal affect of the tragedy.

Thematic distribution across data sources, showing how different platforms prioritize different aspects of the disaster.

As shown in the thematic analysis, news data contained the highest proportion of core situational topics (43.7%), significantly outperforming Twitter (15.1%) in providing unique, actionable information—such as specific infrastructure needs—that was entirely absent in other summaries. Furthermore, news summaries were found to be the most closely aligned with actual First Responder accounts, suggesting that news remains a critical, though sometimes overlooked, source for high-quality SII.

The Impact of Implementation Sensitivity

Perhaps the most surprising finding was the impact of minor implementation tweaks. The researchers tested the COWTS summarization algorithm using two different methods for selecting "content words": TF-IDF scores versus Keyword-in-Context (KWIC) frequency. Despite using the exact same algorithm on the same data, the two implementations produced summaries with almost zero overlap in selected sentences.

This sensitivity extends to the facts themselves. Depending on the source or the specific implementation used, summaries reported conflicting data points, such as varying earthquake magnitudes (7.0 vs. 7.3) or differing death tolls. When summarizing the Haiti earthquake across all three main sources, only one single sentence—regarding the earthquake's magnitude—was common to all generated summaries. This suggests that the "default" settings in many AI tools can significantly and arbitrarily skew a user's understanding of a crisis.

Conclusion and Practical Implications

The study concludes that there is no single "objective" summary of a disaster. The variation in SII is a product of the data source, the method, and the implementation. For first responders, this means that relying on a single data stream (like Twitter) or a single tool may lead to a biased or incomplete picture of the situation on the ground.

Future work in disaster informatics must move beyond simple accuracy metrics. The authors advocate for "data triangulation"—developing tools that intentionally synthesize multiple sources to mitigate individual biases. Furthermore, researchers are encouraged to report the sensitivity of their results to preprocessing and algorithmic parameters, ensuring that the tools used in life-and-death situations are both robust and transparent.

Summary updated

BibTeX
@inproceedings{sarol2021variation,
  author = {Maria Janina Sarol and Ly Dinh and Jana Diesner},
  title = {{Variation in Situational Awareness Information due to Selection of Data Source, Summarization Method, and Method Implementation}},
  booktitle = {Icwsm},
  year = {2021},
}
Figure