What to Make Sense of in the Era of LLM? A Perspective from the Structure and Efforts in Sensemaking

Tianyi Li, Satya Samhita Bonepalli, Vikram Mohanty

TL;DR

This paper explores how Large Language Models (LLMs) like GPT-4 compare to human "crowds" in complex sensemaking tasks, such as solving fictional terrorist plots. The study reveals a fascinating trade-off: while humans excel at the rigorous, detailed extraction of facts, LLMs are significantly more "aggressive" and effective at generating a broad range of high-level hypotheses.

What Problem Does This Paper Address?

Sensemaking is the messy, cognitive process of turning a mountain of ambiguous data into a coherent story or insight. Traditionally, this has been the domain of expert analysts or "crowds" of humans who distribute the cognitive load. However, as LLMs become more capable, researchers are asking: Where do these models fit into the classic sensemaking loop?

The authors address the gap between human-led analysis—which is often slow but precise—and AI-led analysis, which is fast but prone to "hallucinations" or over-simplification. They specifically investigate whether LLMs should be used for "holistic" (one-shot) analysis or a "step-by-step" guided process to best assist human experts.

Methods

The researchers conducted a preliminary investigation comparing human performance (using a baseline of novice crowd workers from previous studies) against GPT-4. They tested two primary AI strategies:

  1. Holistic Approach: The LLM is asked to develop alternative hypotheses directly from the dataset.
  2. Step-by-Step Approach: The LLM follows a predefined pipeline of intermediate steps (e.g., extracting facts first, then synthesizing connections).

To evaluate performance, they used two specific metrics:

  • Coverage: Did the analysis find the relevant facts or hypotheses at all?
  • Comprehensiveness: Did the analysis capture the specific, rigorous details (like bank account numbers or specific locations) required by the "gold standard" answer key?

Datasets

The study utilized a dataset consisting of 10 documents describing a fictional terrorist plot. This type of dataset is a standard benchmark in sensemaking research (often derived from the VAST Challenge or similar intelligence analysis simulations), designed to be complex, interconnected, and filled with "noise."

Key Insights and Ideas

  • The "Abstraction Gap": LLMs tend to summarize too much. While a human might record a specific financial transaction between two people, an LLM might simply state they have "financial ties." This lack of "analytical rigor" can be a bottleneck in high-stakes investigations where the devil is in the details.
  • Process Blurring: In human sensemaking, there is usually a clear "foraging" phase (gathering facts) followed by a "synthesis" phase (forming hypotheses). LLMs tend to merge these steps, forming hypotheses while they are still supposed to be extracting facts.
  • Complementary Strengths: The paper identifies a clear "division of labor." Humans are better at "fact-heavy" work—capturing the factual content of documents with high comprehension. LLMs are "hypothesis-heavy"—they are better at maintaining an overarching perspective and proposing a wider variety of potential explanations.

Key Results

The quantitative results highlight the distinct profiles of human vs. AI performance:

  • Fact Extraction (Evidence Gathering):
    • Crowds: Successfully identified 17 key facts, all of which were "comprehensive" (detailed).
  • GPT-4 (Step-by-Step): Identified 9 facts, but only 5 were comprehensive.
  • GPT-4 (Holistic): Identified 6 facts, none of which were comprehensive.
  • Hypothesis Development (Insight Generation):
    • Crowds: Only identified 4 out of 15 predefined hypotheses.
  • GPT-4 (Holistic): Identified 10 out of 15 hypotheses, with 6 being comprehensive.
  • GPT-4 (Step-by-Step): Only identified 3 comprehensive hypotheses.

Interestingly, the "Holistic" approach for LLMs was better for generating broad ideas, while the "Step-by-Step" approach was slightly better for capturing factual details.

Limitations and Future Work

The authors acknowledge that this is a preliminary study with a small dataset (10 documents). The primary limitation is the trade-off between the LLM's ability to synthesize and its tendency to omit the "rigor" required for expert-level analysis.

Future Work: The authors propose a hybrid collaborative model. In this workflow, humans would lead the initial "data gathering" phase to ensure accuracy and detail. LLMs would then take that high-quality factual foundation and use their superior pattern recognition to synthesize and interpret the data, proposing broad hypotheses that humans can then verify.

Summary updated

Abstract

Sensemaking tasks often entail navigating through complex, ambiguous data to construct coherent insights. Prior work has shown that crowds can effectively distribute cognitive load, pooling diverse perspectives to enhance analytical depth. Recent advancements in LLMs have further expanded the toolkit for sensemaking, offering scalable data processing, complex pattern recognition, and the ability to infer and propose meaningful hypotheses. In this study, we explore how LLMs (i.e., GPT-4) can assist in a complex sensemaking task of deciphering fictional terrorist plots. We explore two different approaches for leveraging GPT-4's capabilities: a holistic sensemaking process and a step-by-step approach. Our preliminary investigations open the doors for future research into optimizing human-AI collaborative workflows, aiming to harness the complementary strengths of both for more effective sensemaking in complex scenarios.
BibTeX
@misc{li2026what,
  author = {Tianyi Li and Satya Samhita Bonepalli and Vikram Mohanty},
  title = {{What to Make Sense of in the Era of LLM? A Perspective from the Structure and Efforts in Sensemaking}},
  year = {2026},
  eprint = {2603.08604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08604},
}

Loading related content...

Figure