Abstract
Researchers from Dartmouth College have introduced Disordered-DABS, a new benchmark designed to evaluate the robustness of summarization models against "disordered" text—environments where topics shift rapidly and sentence order is non-linear. By intentionally shuffling sentences from standard datasets, the authors reveal that both traditional models and Large Language Models (LLMs) like GPT-3.5 rely heavily on a "coherence crutch." When this structural context is removed, models struggle to identify the correct number of topics and group related information, often leading to fragmented or hallucinated summaries.
Introduction: The Challenge of Disordered Text
Most automated summarization research focuses on well-structured documents, such as news articles, where narrative flow is logical and sequential. However, real-world data—including social media threads, customer feedback, and medical records—is frequently "disordered." In these contexts, multiple authors may discuss various "aspects" (topics) simultaneously, and the conversation rarely follows a linear path.
Existing Aspect-Based Summarization (ABS) models typically use predefined categories. Dynamic Aspect-Based Summarization (DABS) is more flexible, identifying topics on the fly. However, even DABS models often depend on sentence transitions to detect topic shifts. This paper introduces a challenge where models must rely purely on content, rather than context, to generate coherent summaries.
The Disordered-DABS Benchmark
The benchmark consists of 255,286 aspect-oriented summaries across two primary domains:
- D-CnnDM: Based on CNN/DailyMail news, featuring long articles (~4,143 words) with coarse-grained, distinct news stories.
- D-WikiHow: Based on WikiHow, featuring shorter articles (~452 words) with fine-grained, overlapping steps in a process.
To create this benchmark, the authors took "clean" source articles and intentionally shuffled their sentences. This disruption forces models to "cluster" related information from across the entire document to form a coherent summary.
Methodology and Baselines
The authors evaluated three distinct strategies to handle disordered text:
- Clustering-Based: This approach uses BERTopic to group sentences by topic before passing them to a Longformer model for final summarization.
- Keyword-Guided: This method extracts the top 10 keywords for each aspect to serve as "hints" for the summarization model.
- Prompt-Guided (LLM): This uses GPT-3.5 with zero-shot, few-shot, and fine-tuned prompting to test if massive scale can compensate for the lack of structure.
Key Results and Insights
The experimental results highlight a significant performance gap when narrative coherence is removed.
The "Coherence Crutch"
Performance across metrics like ROUGE and BERTScore dropped significantly when text was shuffled. This confirms that current models rely heavily on sentence transitions to identify topics.
Clustering vs. Zero-Shot Prompting
On the D-CnnDM dataset, the Cluster baseline achieved a Rouge-L of 17.1, significantly outperforming zero-shot Prompting (9.1). GPT-3.5 tended to be "too sensitive" in zero-shot settings, often splitting a single topic into multiple tiny segments (high "Aspect Difference" score). This suggests that explicit topic modeling remains more effective than the internal attention mechanisms of current LLMs for disordered text.
The Importance of Topic Identification
When models were provided with the exact number of aspects (the "perfect-match" scenario), performance improved drastically. For instance, the Rouge-1 for the Cluster method on D-WikiHow rose from 19.1 to 38.1. This indicates that identifying the number of latent topics is one of the primary bottlenecks in summarizing disordered text.
Limitations and Future Work
While Disordered-DABS provides a rigorous testbed, the authors note several limitations. The study primarily utilized GPT-3.5-turbo, and results may vary with more advanced models like GPT-4 or Llama-3. Furthermore, the benchmark currently focuses on news and encyclopedic text, leaving specialized domains like legal or medical records for future exploration. Future research should focus on more robust unsupervised clustering and addressing the "fragmentation" issues observed in LLM outputs.
Summary updated