TL;DR
LaMSUM is a novel multi-level framework that leverages Large Language Models (LLMs) and voting algorithms from Social Choice Theory to perform extractive summarization on large volumes of sensitive incident reports. By combining shuffling to combat positional bias with robust voting methods, the framework outperforms state-of-the-art models, providing authorities with diverse, authentic, and descriptive snapshots of harassment trends.
What Problem Does This Paper Address?
Citizen reporting platforms for sexual harassment and abuse generate massive amounts of data. For authorities and the public to make sense of this, summarization is essential. However, two major hurdles exist:
- The Extractive vs. Abstractive Gap: LLMs are naturally "abstractive"—they like to paraphrase. In sensitive contexts like harassment reporting, maintaining the user’s exact words (extractive summarization) is crucial for authenticity, legal context, and trust. As shown in Figure 1, even when prompted for extractive results, LLMs often default to paraphrasing.
- Technical Constraints: LLMs have finite context windows, making it impossible to process thousands of reports at once. Furthermore, they suffer from positional bias, where they tend to favor information at the beginning or end of a prompt.
Methods
The authors introduce LaMSUM (Large Language Model based Extractive SUMmarization), which operates through a multi-level, "election-style" process:
- Multi-Level Framework: To handle datasets exceeding context windows, the input is divided into chunks. Summaries are generated for each chunk and then merged iteratively until a final summary of size k is reached.
- Tackling Positional Bias: Within each chunk, the framework creates $m$ different variations by shuffling the order of posts. Each variation is sent to the LLM to select the best representative posts.
- Summarization as an Election: The framework treats the $m$ different LLM outputs as "ballots." It uses voting algorithms—specifically Plurality Voting, Ranked Choice (Borda Count), and Proportional Approval Voting (PAV)—to aggregate these outputs into a final, robust chunk summary.
- Output Calibration: Since LLMs might still slightly alter words, the authors implement a calibration step using edit distance and keyword matching to ensure the final output is strictly a subset of the original text.
Datasets
The primary evaluation was conducted on incident reports from a major Asian platform across five cities (Cities A, B, C, D, and E), totaling thousands of posts. These reports are often "code-mixed" (e.g., Hinglish), adding a layer of linguistic complexity.
To test generalizability, the authors also evaluated LaMSUM on three public datasets:
- Claritin: Tweets about drug side effects.
- US-Election: Tweets from the 2016 presidential election.
- Me-Too: Tweets recounting harassment experiences.
Key Insights and Ideas
- Voting for Diversity: The authors found that Proportional Approval Voting (PAV) performed best. Unlike simple plurality, PAV applies "diminishing returns" to similar posts, acting like an editor that balances popular themes with distinct, less common insights.
- Shuffling is Mandatory: The experiments confirmed that LLMs are highly sensitive to the order of input. Shuffling ($m$ variations) and then aggregating is a necessary step to ensure every post has a fair chance of being selected.
- LLM Choice Matters: Interestingly, Claude-3-Haiku consistently outperformed larger or more famous models like GPT-4o-mini and Llama-3.1-8B in this specific extractive task.
- Descriptive Power: LaMSUM doesn't just pick the "average" post; it tends to select posts that are more detailed and descriptive (higher word count) compared to traditional algorithms, as seen in Figure 6.
Key Results
LaMSUM significantly outperformed state-of-the-art extractive summarization models like BERTSUM, LexRank, and LSA across all datasets.
- Quantitative: In City A, LaMSUM achieved a ROUGE-LSum score of 59.996, compared to BERTSUM’s 54.226.
- Qualitative: The framework achieved higher Entropy scores (Table 7), indicating that the summaries were more diverse and covered a broader range of harassment categories (e.g., stalking, physical assault, ogling) rather than repeating the same type of incident.
- Significance: Paired t-tests showed that the improvements over "Vanilla" LLM approaches (no shuffling/voting) were statistically significant ($p < 0.05$).
Limitations and Future Work
- Final Summary Size: While the framework can process infinite input, the final summary must currently fit within a single LLM context window.
- Data Leakage: The authors acknowledge that newer LLMs might have seen some public datasets during pre-training, though the superior performance of LaMSUM over Vanilla LLM setups suggests the framework's logic is the primary driver of success.
- Ethical Oversight: The authors warn against "black box" reliance. Summarization might overlook infrequent but critical issues, meaning human oversight remains necessary for policy-making.
- Future Directions: The framework could be extended to handle social media "threads" by incorporating hierarchical clustering before the summarization steps.
Summary updated