Fair Summarization: Bridging Quality and Diversity in Extractive Summaries

Sina Bagheri Nezhad, Sayan Bandyapadhyay, Ameeta AgrawalNAACL 2025

TL;DR

This paper tackles the critical challenge of bias in multi-document summarization, particularly for user-generated content. They introduce two novel methods, FairExtract and FairGPT, that achieve perfect fairness in representation while maintaining competitive summarization quality, offering a more balanced and representative view of diverse social groups.

What Problem Does This Paper Address?

Summarizing multiple documents, especially from diverse sources like social media, is a complex task. Existing methods often struggle to represent different social groups equitably, leading to biased summaries that can underrepresent certain voices. This is particularly problematic in social media where public opinion is shaped. The authors highlight that while improving fairness can sometimes decrease quality, there's a need to understand this trade-off and develop methods that excel in both. They specifically focus on dialectal variations among different social groups (e.g., White-aligned, Hispanic, African-American) as a proxy for social group representation.

Methods

The paper introduces two novel methods for fair extractive summarization:

  • FairExtract: This method uses a clustering-based approach. It begins by embedding documents using BERT, then decomposes them into "fairlets" that maintain proportional representation of social groups. It then finds the center of these fairlets and applies k-median clustering to these centers. Finally, it constructs the summary by selecting documents from the fairlets corresponding to the cluster centers, ensuring both fairness and quality.
  • FairGPT: This method leverages large language models (LLMs) like GPT-3.5-turbo. It prepares input by splitting documents into groups and then prompts the LLM to extract an equal number of sentences from each group. To ensure accuracy, it uses the Longest Common Subsequence (LCS) to match generated sentences with original tweets and includes full sentences. It also includes an output check to verify fairness and similarity, repeating the process if conditions aren't met.

The authors also introduce composite metrics (e.g., SUPERT+F, BLANC+F) that combine normalized quality scores with a fairness score (F), providing a more holistic evaluation of summarization models.

Datasets

The primary dataset used is DivSumm (Olabisi et al., 2022), which consists of tweets from three dialect groups (White-aligned, Hispanic, and African-American) across 25 topics, with 30 tweets per group per topic, totaling 2,250 tweets. The experiments explore three pairings of these groups: White-Hispanic, Hispanic-African American, and White-African American.

Key Insights and Ideas

  • Perfect fairness is achievable without sacrificing significant quality: The core finding is that their proposed methods, FairExtract and FairGPT, can achieve perfect fairness (F=1) while still performing competitively on various quality metrics.
  • Composite metrics offer a nuanced view: Introducing metrics that combine quality and fairness (e.g., SUPERT+F) provides a better understanding of the trade-offs and allows for a more balanced evaluation than looking at quality and fairness in isolation.
  • LLMs can be guided towards fairness: FairGPT demonstrates that with careful prompting and post-processing (like LCS matching), LLMs can be steered to produce fair summaries, not just high-quality ones.
  • Clustering can be adapted for fairness: FairExtract shows that by incorporating fairlet decomposition into clustering, it's possible to build fair summaries from diverse text sources.

Key Results

  • Fairness: Both FairExtract and FairGPT achieved a perfect fairness score (F=1) across all tested pairings, outperforming all baseline methods in this regard.
  • Quality:
    • ChatGPT-EXT was the top performer in terms of raw quality metrics (SUPERT, BLANC, BARTScore, UniEval), but its fairness score was moderate (F=0.698).
  • FairGPT achieved high quality scores, particularly in SUPERT (0.644) and BARTScore (-0.821), while also achieving perfect fairness.
  • FairExtract performed comparably to TextRank in quality metrics, excelling in BLANC (0.140) and achieving respectable scores in SUPERT and UniEval, alongside perfect fairness.
  • Composite Metrics (equal weighting):
    • FairGPT achieved the highest composite scores across most metrics, demonstrating its superior balance of quality and fairness.
  • FairExtract consistently outperformed other clustering-based models on composite metrics, showing its effectiveness in balancing quality and fairness.
  • Reduced Fairness Weighting: Even with a significantly reduced fairness weight (16%), FairExtract continued to outperform other clustering-based methods, highlighting its robustness.

Limitations and Future Work

  • Extractive vs. Abstractive: The current methods focus on extractive summarization. Extending them to abstractive summarization presents challenges, particularly in balancing fairness with coherence and fluency.
  • Dataset Domain: The study used social media content. Generalizing to other domains like news or scientific articles might require adjustments due to differences in language and formality.
  • Monolingual Focus: The work is limited to English. Future research could explore multilingual summarization, considering language diversity and cross-lingual transfer.
  • Subjective Factors: Standard metrics don't fully capture subjective factors like readability or user trust. Human evaluation could provide deeper insights.
  • Computational Complexity: The computational complexity of fair clustering and LLMs might limit scalability in real-time or resource-constrained environments.

Summary updated

BibTeX
@inproceedings{nezhad2025fair,
  author = {Sina Bagheri Nezhad and Sayan Bandyapadhyay and Ameeta Agrawal},
  title = {{Fair Summarization: Bridging Quality and Diversity in Extractive Summaries}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure