BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization

Ahmed Rafid, Rumman Adib, Fariya Ahmed, Ajwad Abrar, Mohammed Saidul IslamarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces BanglaSummEval, the first reference-free framework for evaluating the factual consistency of Bangla text summaries. It leverages a unified, multilingual LLM to generate questions and answers from source documents and summaries, enabling interpretable diagnostics and strong correlation with human judgments, crucial for low-resource languages.

What Problem Does This Paper Address?

Accurate summarization is vital, especially in critical fields like healthcare and news. However, evaluating the factual consistency of these summaries is challenging. Existing methods often rely on human-written reference summaries, which are expensive and time-consuming to create, especially for languages with fewer resources. Furthermore, many metrics struggle with morphologically rich languages like Bangla, where word order is flexible and paraphrasing is common, making surface-level comparisons insufficient. This gap hinders the development and reliable evaluation of Bangla summarization models.

Methods

BanglaSummEval adapts the QuestEval framework, employing a reference-free, question-answering (QA) based approach. The core innovation lies in using a single, unified, 4-bit quantized multilingual LLM (Qwen3-14B-bnb-4bit) to handle multiple components:

  • Question Generation (QG): Generates questions from both source documents and summaries.
  • Question Answering (QA): Answers these generated questions using the respective text.
  • Named Entity and Noun Extraction (NER): Identifies candidate answers (entities and nouns) from the text.
  • Question Weighting: Assigns an importance score to each question, prioritizing those that cover core facts.

To assess factual consistency, the framework compares answers derived from the source document with those derived from the summary. Instead of simple lexical overlap, it uses BERTScore-Recall to measure semantic similarity between answers, capturing meaning even with different phrasing. The final score is a harmonic mean of precision (summary claims supported by source) and recall (source information covered by summary), weighted by question importance. This unified, step-wise approach offers transparency and diagnostic insights into LLM failures.

Datasets

The authors evaluated their framework on a dataset of 300 human-written summaries. These summaries were sampled from two sources:

  • NCTB dataset: Summaries written by professional writers and edited for curriculum relevance.
  • BanglaCHQ-Summ: Summaries created by medical informatics experts for medical queries.

The models were benchmarked on the BanglaRQA corpus for evaluating QA and QG capabilities.

Key Insights and Ideas

  • Unified LLM for Efficiency: Using a single LLM for all components (QA, QG, NER, weighting) significantly reduces system complexity and computational overhead, making it practical for low-resource settings.
  • BERTScore-Recall for Semantic Nuance: Moving beyond lexical similarity (like ROUGE or BLEU) to BERTScore-Recall is crucial for morphologically rich languages like Bangla, where paraphrasing is common and surface-level matches can be misleading.
  • Interpretable Diagnostics: The step-wise nature of the evaluation (generating questions, answers, and then comparing) provides valuable insights into why a summary is flagged as inconsistent, moving beyond a single score.
  • Importance Weighting Matters: Not all facts are equally important. Weighting questions based on their centrality to the document's meaning ensures that the evaluation focuses on critical information rather than minor details.

Key Results

  • BanglaSummEval demonstrates strong correlation with human judgments: Pearson's $r = 0.694$ and Spearman's $ρ = 0.763$.
  • The framework is computationally efficient, processing an average document-summary pair in approximately 25 minutes on a single NVIDIA T4 GPU, with peak memory usage under 13GB.
  • The chosen LLM, Qwen3-14B-bnb-4bit, outperformed other models in QA and QG tasks on the BanglaRQA dataset.
  • Semantic metrics like BERTScore-Recall showed significantly higher correlation with human judgments than lexical metrics.

Limitations and Future Work

  • LLM Biases: The framework might inherit biases from the high-resource training data of the underlying LLM. Future work could explore bias mitigation strategies and adversarial evaluation.
  • Semantic Nuance Limitations: While BERTScore-Recall is an improvement, it might not fully capture all subtle semantic nuances in Bangla compared to larger encoders.
  • Relational Error Detection: The QA-based approach is optimized for atomic facts. Capturing complex relational errors (e.g., causal chains, implicit role reversals) remains a challenge, suggesting future work on explicit relation extraction or knowledge graph integration.
  • Scalability: While efficient, the framework's performance might not be substantially stronger than directly prompting a general-purpose LLM. Future work could explore mixed-precision techniques or multi-model approaches.
  • Domain Adaptation: The authors suggest exploring domain-specific adaptations and extensions to other underrepresented languages.

Summary updated

Abstract

Evaluating factual consistency is essential for reliable text summarization, particularly in high-stakes domains such as healthcare and news. However, most existing evaluation metrics overlook Bangla, a widely spoken yet under-resourced language, and often depend on reference summaries. We introduce BanglaSummEval, a reference-free, question-answering-based framework for evaluating factual consistency in Bangla summarization. The proposed method assesses both factual accuracy and content coverage through automatically generated questions and answers derived from the source document and the summary. A single multilingual instruction-tuned language model handles question generation, question answering, candidate answer extraction, and question importance weighting. This unified design reduces system complexity and computational cost. To capture semantic consistency beyond surface-level overlap, we use BERTScore-Recall for answer comparison. We validate BanglaSummEval on 300 human-written summaries from educational and medical domains, demonstrating strong correlation with expert human judgments (Pearson's $r = 0.694$, Spearman's $ρ= 0.763$). By providing interpretable, step-wise diagnostics alongside reliable evaluation scores, BanglaSummEval offers a practical and transparent solution for factual consistency evaluation in low-resource language settings.
BibTeX
@misc{rafid2026banglasummeval,
  author = {Ahmed Rafid and Rumman Adib and Fariya Ahmed and Ajwad Abrar and Mohammed Saidul Islam},
  title = {{BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization}},
  year = {2026},
  eprint = {2602.16843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.16843},
}
Figure