Introduction
Summarization research has traditionally focused on "clean" data—neatly structured news articles where sentences follow a logical progression. However, real-world data from sources like Reddit threads, customer feedback, or medical records is often a jumble of different voices, rapid topic shifts, and fragmented information. To address this gap, researchers from Dartmouth College introduce Disordered-DABS, a benchmark designed to evaluate "Dynamic Aspect-Based Summarization" (DABS) in intentionally scrambled source texts.
The Challenge: Overcoming Coherence Bias
Most summarization models benefit from the inherent coherence of their training data. In a standard news article, transitions provide a roadmap for the model. In contrast, real-world applications often involve abruptly changing topics and scattered information. Existing benchmarks for Aspect-Based Summarization (ABS) typically rely on predefined categories, while Dynamic ABS (DABS) identifies aspects on the fly but still assumes organized source text.
The authors argue that this "coherence bias" inflates model performance. By shuffling sentences in the source text, they create a "disordered" environment that forces models to understand the relationships between ideas rather than relying on their sequence.
The Disordered-DABS Benchmark
The researchers created Disordered-DABS by adapting two high-quality datasets:
- D-CnnDM: Derived from the CNN/DailyMail news dataset, aggregating up to 11 different stories into one disordered document.
- D-WikiHow: Derived from WikiHow, where each aspect corresponds to a different step or method in a "how-to" guide.
The final benchmark contains 255,286 aspect-oriented summaries, with source sentences randomly shuffled to disrupt natural flow.
Methodological Approaches
The study compared three distinct strategies:
- Clustering-Based (Cluster): A two-step pipeline using BERTopic for unsupervised grouping of related sentences, followed by a Longformer model to generate summaries for each cluster.
- Keyword-Guided (Keyword): Extracts the top 10 keywords for each aspect to serve as "hints" for the summarization model.
- Prompt-Guided (Prompting): Uses GPT-3.5 with specific instructions to identify and summarize aspects in a zero-shot or few-shot setting.
Key Results and Insights
The evaluation revealed significant performance gaps between specialized pipelines and general-purpose LLMs.
Specialized Clustering vs. Raw Prompting
On the D-CnnDM dataset, the Cluster method achieved a ROUGE-L score of 17.1, nearly double that of zero-shot Prompting (9.1). Furthermore, the "Absolute Aspect Difference" (#AbsAspDiff) was much lower for the Cluster and Keyword methods (~1.3) compared to GPT-3.5 (~6.2). This indicates that LLMs struggle to accurately count and identify distinct topics when sentences are not contiguous, often "hallucinating" extra aspects or creating redundant summaries.
Human Evaluation and Subjectivity
Human evaluators noted that identifying aspects in WikiHow was significantly harder than in News, suggesting that the "granularity" of a topic remains a major hurdle. While GPT-3.5 excelled in Fluency, it lagged behind the Cluster method in Consistency and Relevance on the news dataset. However, on the more complex WikiHow dataset, GPT-3.5’s general reasoning capabilities allowed it to take a slight lead in overall ranking.
Conclusion
The Disordered-DABS benchmark serves as a new "stress test" for the NLP community. The findings suggest that for "messy" data, dedicated information organization steps (like clustering) remain superior to one-shot prompting of current LLMs. Future work will look to expand this benchmark to even more challenging domains like social media and medical records.
Summary updated