Abstract
ARMAN is a novel Transformer-based pre-training framework designed to improve abstractive summarization by focusing on semantic meaning rather than simple word overlaps. By introducing a "Top Sentence Selection" (TSS) algorithm based on semantic scores and a sentence reordering objective, ARMAN achieves state-of-the-art results across six Persian summarization tasks. The model demonstrates significant improvements in low-resource and zero-shot settings, effectively addressing the "copying trap" prevalent in syntactic-based pre-training methods.
Introduction
Current state-of-the-art summarization models often rely on pre-training objectives like "Gap Sentence Generation," which select sentences to mask based on syntactic similarity (e.g., ROUGE scores). This approach tends to reward models for extractive behavior—copying and pasting—rather than true abstractive rephrasing. In the context of Persian, a language with limited research in abstractive summarization, this limitation is particularly pronounced. ARMAN (Abstractive Reordering and Masking Network) addresses this by prioritizing semantic similarity and logical flow during pre-training.
Methodology
ARMAN utilizes a standard Transformer encoder-decoder architecture with a specialized pre-training pipeline designed to capture semantic importance and narrative structure.
Top Sentence Selection (TSS)
Instead of syntactic metrics, ARMAN uses a modified BERTScore to identify semantically central sentences. To maintain computational efficiency during large-scale data preparation, the authors employ FastText embeddings. This allows the model to pinpoint sentences that carry the core meaning of a document, regardless of exact word matches.
Pre-training Objectives
The framework introduces three primary objectives:
- TSS-Semantic Similarity (SS): The model masks the most semantically important sentences. Using a single mask token for multiple consecutive sentences forces the model to infer both the length and the content of the missing information.
- TSS-Shuffling (SH): Spans within sentences are shuffled to discourage simple copying and encourage deeper linguistic understanding.
- TSS-Modified Sentence Reordering (MSR): This objective requires the model to restore the original logical order of the top 30% most important sentences from a shuffled document.
Datasets and Benchmarks
The model was pre-trained on a massive 64.7GB cleaned Persian corpus (irBlogs, MirasText, CC100, and YJC News). Evaluation was conducted across several downstream tasks, including PN-Summary, Wiki-Summary, VOA, and Perkey. Additionally, the authors introduced the Tebyan Dataset, a new benchmark for Persian abstractive summarization containing over 92,000 document-summary pairs.
Results and Discussion
ARMAN(MSR) established a new state-of-the-art for Persian NLP, outperforming existing models across all six summarization tasks.
Performance and Low-Resource Capabilities
The model showed exceptional performance in low-resource scenarios, surpassing previous SOTA models with as few as 1,000 examples. In zero-shot settings, it significantly outperformed PEGASUS. Quantitative metrics for density and coverage indicated that ARMAN's summaries are more abstractive and factually consistent than those produced by baseline models.
Human Evaluation
Blind human evaluations confirmed the quantitative findings. Evaluators preferred ARMAN(MSR) summaries over 50% of the time, noting superior fluency and informativeness compared to PEGASUS and other variants.
Conclusion
ARMAN demonstrates that focusing on semantic importance and logical reordering during pre-training significantly enhances abstractive summarization quality. While currently optimized for Persian using FastText for efficiency, the methodology is language-independent and offers a robust framework for improving summarization in other languages and potentially extractive tasks.
Summary updated