Abstract
In this paper, we introduce a large-scale Indonesian summarization dataset. We harvest articles from Liputan6.com, an online news portal, and obtain 215,827 document-summary pairs. We leverage pre-trained language models to develop benchmark extractive and abstractive summarization methods over the dataset with multilingual and monolingual BERT-based models. We include a thorough error analysis by examining machine-generated summaries that have low ROUGE scores, and expose both issues with ROUGE it-self, as well as with extractive and abstractive summarization models.
Introduction
Indonesian is the fourth most spoken language globally, yet it remains significantly under-represented in Natural Language Processing (NLP) research. A primary obstacle has been the "data desert"—a lack of massive, high-quality datasets required to train modern neural models. While English-centric research benefits from large-scale datasets like CNN/Daily Mail (320k pairs), Indonesian researchers have historically relied on smaller collections like IndoSum (~19k pairs), which are predominantly extractive. This work introduces Liputan6, a massive, diverse, and more abstractive corpus designed to push the boundaries of Indonesian automated summarization.
Methodology
To establish a rigorous benchmark, we developed several neural architectures built upon pre-trained language models. A key component of our study is IndoBERT, a monolingual BERT-Base model trained on 220 million words of Indonesian text. We evaluated two primary classes of models:
- Extractive Models (BERTEXT): These models identify and select the most salient sentences directly from the source text.
- Abstractive Models (BERTABS & BERTEXTABS): These models generate novel sentences. BERTEXTABS utilizes a hybrid approach where the encoder is pre-trained on an extractive task before fine-tuning for abstractive generation.
The Liputan6 Dataset
The dataset consists of 215,827 document-summary pairs scraped from Liputan6.com, spanning the years 2000 to 2010. We provide two main variants:
- Canonical: A standard 90/5/5 split for training and evaluation.
- Xtreme: A specialized test set containing highly abstractive summaries (where at least 90% of 4-grams are novel), serving as a "stress test" for generative models.
Analysis revealed a significant "lead bias" in the dataset, where the most critical information typically appears in the first few sentences. Furthermore, we observed a temporal shift where summaries became less abstractive over the decade, leading us to use older, more challenging articles for our test sets to ensure model robustness.
Results and Discussion
Our experiments yielded several key insights:
- Monolingual Superiority: IndoBERT consistently outperformed Multilingual BERT (mBERT), highlighting the value of language-specific pre-training for Indonesian.
- Model Performance: The IndoBERT-based BERTEXTABS model was the top performer, achieving a ROUGE-1 score of 41.08 on the Canonical set. However, performance dropped significantly on the Xtreme set (34.84), indicating the difficulty of true abstraction.
- The ROUGE Mirage: A critical finding was the limitation of the ROUGE metric. Manual evaluation of 100 "low-scoring" summaries revealed that 60% were actually high-quality. ROUGE's reliance on exact word overlap fails to account for synonyms, morphological variations, and paraphrasing.
Conclusion and Future Work
While Liputan6 provides a robust foundation for Indonesian summarization, challenges remain. Future research should focus on adopting semantic metrics like BERTScore to better capture summary quality. Additionally, addressing "hallucinations"—where models generate information not present in the source document—remains a priority. Finally, expanding large-scale Indonesian datasets beyond the news domain into social media and technical texts will be crucial for broader NLP advancement.
Summary updated