ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

Alireza Salemi, Emad Kebriaei, Ghazal Neisi Minaei, Azadeh Shakery

Abstract

ARMAN is a novel Transformer-based pre-training framework designed to improve abstractive summarization by focusing on semantic meaning rather than simple word overlaps. By introducing a "Top Sentence Selection" (TSS) algorithm based on semantic scores and a sentence reordering objective, ARMAN achieves state-of-the-art results across six Persian summarization tasks. The model demonstrates significant improvements in low-resource and zero-shot settings, effectively addressing the "copying trap" prevalent in syntactic-based pre-training methods.

Introduction

Current state-of-the-art summarization models often rely on pre-training objectives like "Gap Sentence Generation," which select sentences to mask based on syntactic similarity (e.g., ROUGE scores). This approach tends to reward models for extractive behavior—copying and pasting—rather than true abstractive rephrasing. In the context of Persian, a language with limited research in abstractive summarization, this limitation is particularly pronounced. ARMAN (Abstractive Reordering and Masking Network) addresses this by prioritizing semantic similarity and logical flow during pre-training.

Methodology

ARMAN utilizes a standard Transformer encoder-decoder architecture with a specialized pre-training pipeline designed to capture semantic importance and narrative structure.

Top Sentence Selection (TSS)

Instead of syntactic metrics, ARMAN uses a modified BERTScore to identify semantically central sentences. To maintain computational efficiency during large-scale data preparation, the authors employ FastText embeddings. This allows the model to pinpoint sentences that carry the core meaning of a document, regardless of exact word matches.

Pre-training Objectives

The framework introduces three primary objectives:

  1. TSS-Semantic Similarity (SS): The model masks the most semantically important sentences. Using a single mask token for multiple consecutive sentences forces the model to infer both the length and the content of the missing information.
  2. TSS-Shuffling (SH): Spans within sentences are shuffled to discourage simple copying and encourage deeper linguistic understanding.
  3. TSS-Modified Sentence Reordering (MSR): This objective requires the model to restore the original logical order of the top 30% most important sentences from a shuffled document.
The ARMAN pre-training framework, illustrating the Top Sentence Selection (TSS) and Modified Sentence Reordering (MSR) objectives.

Datasets and Benchmarks

The model was pre-trained on a massive 64.7GB cleaned Persian corpus (irBlogs, MirasText, CC100, and YJC News). Evaluation was conducted across several downstream tasks, including PN-Summary, Wiki-Summary, VOA, and Perkey. Additionally, the authors introduced the Tebyan Dataset, a new benchmark for Persian abstractive summarization containing over 92,000 document-summary pairs.

Results and Discussion

ARMAN(MSR) established a new state-of-the-art for Persian NLP, outperforming existing models across all six summarization tasks.

Performance and Low-Resource Capabilities

The model showed exceptional performance in low-resource scenarios, surpassing previous SOTA models with as few as 1,000 examples. In zero-shot settings, it significantly outperformed PEGASUS. Quantitative metrics for density and coverage indicated that ARMAN's summaries are more abstractive and factually consistent than those produced by baseline models.

Comparison of ROUGE scores (R1, R2, RL) for ARMAN variants and PEGASUS across six Persian datasets.

Human Evaluation

Blind human evaluations confirmed the quantitative findings. Evaluators preferred ARMAN(MSR) summaries over 50% of the time, noting superior fluency and informativeness compared to PEGASUS and other variants.

Conclusion

ARMAN demonstrates that focusing on semantic importance and logical reordering during pre-training significantly enhances abstractive summarization quality. While currently optimized for Persian using FastText for efficiency, the methodology is language-independent and offers a robust framework for improving summarization in other languages and potentially extractive tasks.

Summary updated

Abstract

ive text summarization is one of the areas influenced by the emergence of pre-trained language models. Current pre-training works in abstractive summarization give more points to the summaries with more words in common with the main text and pay less attention to the semantic similarity between generated sentences and the original document. We propose ARMAN, a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. To summarize more accurately and similar to human writing patterns, we applied modified sentence reordering. We evaluated our proposed models on six downstream Persian summarization tasks. Experimental results show that our proposed model achieves state-of-the-art performance on all six summarization tasks measured by ROUGE and BERTScore. Our models also outperform prior works in textual entailment, question paraphrasing, and multiple choice question answering. Finally, we established a human evaluation and show that using the semantic score significantly improves summarization results.
BibTeX
@inproceedings{salemi2021arman,
  author = {Alireza Salemi and Emad Kebriaei and Ghazal Neisi Minaei and Azadeh Shakery},
  title = {{ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.741},
}
Figure