Fine-tuning GPT-3 for Russian Text Summarization

Alexandr Nikolich, Arina PuchkovaarXiv 2021 · DOI · arXiv

TL;DR

This paper fine-tunes the Russian GPT-3 (ruGPT3) model for abstractive text summarization of news articles. While achieving competitive results with a BERT-based metric, the model struggles with factual accuracy and named entity preservation, highlighting the ongoing challenges in robust abstractive summarization.

What Problem Does This Paper Address?

Automatic text summarization aims to condense lengthy texts into shorter, informative summaries while retaining the core message. While abstractive summarization models, particularly those based on Transformer architectures like GPT-3, have shown great promise in English, there's a notable gap in applying and evaluating these advanced models for the Russian language. Existing Russian summarization solutions often lag behind their English counterparts, and there's a need to explore how powerful pre-trained models like GPT-3 can be adapted for this specific linguistic context.

Methods

The researchers fine-tuned a pre-trained ruGPT3Small model (125M parameters) on a Russian news dataset. To improve the quality and coherence of the generated summaries, they employed several hyperparameter tuning techniques:

  • Temperature: Set to 0 to restrict the model's output to more probable tokens, reducing randomness.
  • Beam Search: Tuned top-k and top-p values (0.95 and 3 respectively) and increased the number of beams to 20 to encourage more coherent and sensible text generation.
  • Early Stopping: Enabled to ensure the best output is selected only when the model signals the end of the sequence.
  • Repetition Penalties: Implemented no_repeat_ngram_size (set to 3) and repetition_penalty (set to 2) to discourage the model from repeating phrases or information within the summary.

Datasets

  • Gazeta Dataset: This dataset, scraped from gazeta.ru, contains 63,435 Russian news articles with their corresponding human-generated summaries. The authors preprocessed the data by removing summaries that were too short or too long, or had an n-gram overlap with the text that was too narrow or too wide. The dataset was split into a training set (57,665 examples) and a testing set (5,770 examples).

Key Insights and Ideas

  • BERTscore as a Better Metric for Abstractive Summarization: The paper strongly advocates for using metrics like BERTscore, which leverage contextual embeddings, over traditional n-gram overlap metrics (like ROUGE and BLEU) for evaluating abstractive summarization. This is because abstractive models can rephrase, use synonyms, and rearrange sentences, making direct word overlap a less reliable indicator of quality.
  • Fine-tuning GPT-3 for Russian is Viable: The study demonstrates that ruGPT3, when fine-tuned on a relevant dataset, can produce summaries that are competitive, especially when evaluated with more semantically aware metrics.
  • Named Entity Preservation is a Major Hurdle: A significant finding is the model's tendency to alter or invent Named Entities (names, places, dates). This is a critical flaw for factual summarization, as it can distort essential details even if the overall narrative is coherent.

Key Results

  • ROUGE/BLEU Scores: The fine-tuned ruGPT3 model performed poorly on ROUGE and BLEU metrics, scoring approximately three times worse than the state-of-the-art mbart_ru_sum_gazeta model.
  • BERTscore: In contrast, ruGPT3 achieved significantly better BERTscore results (precision: 0.87, recall: 0.90, F1: 0.89) compared to mbart_ru_sum_gazeta (precision: 0.73, recall: 0.73, F1: 0.73), indicating its strength in capturing the semantic meaning.
  • Qualitative Findings: Manual validation revealed that while the model can produce sensible summaries, it frequently alters Named Entities, sometimes misses the key idea, or repeats information.

Limitations and Future Work

The authors acknowledge several limitations:

  • Named Entity Alteration: The model's tendency to change or invent Named Entities is a significant issue that severely degrades summarization quality.
  • Factual Inaccuracies: The model can deviate from facts stated in the original document.
  • Repetition: Despite penalties, the model sometimes reiterates information.

Future work proposed includes:

  • Targeted Penalties: Developing methods to specifically penalize the alteration of Named Entities during training or by creating a list of forbidden words not present in the source text.
  • Multiple Sampling: Allowing the model to generate multiple summary candidates and then selecting the best one, potentially using dynamic modeling or algorithms like Bellman-Ford for optimization.
  • Larger Model Training: Training a larger version of the model (ruGPT3medium) to better handle complex text structures and contextual details.

Summary updated

Abstract

Automatic summarization techniques aim to shorten and generalize information given in the text while preserving its core message and the most relevant ideas. This task can be approached and treated with a variety of methods, however, not many attempts have been made to produce solutions specifically for the Russian language despite existing localizations of the state-of-the-art models. In this paper, we aim to showcase ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. Additionally, we employ hyperparameter tuning so that the model's output becomes less random and more tied to the original text. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function. Despite being able to produce sensible summaries, our model still suffers from a number of flaws, namely, it is prone to altering Named Entities present in the original text (such as surnames, places, dates), deviating from facts stated in the given document, and repeating the information in the summary.
BibTeX
@misc{nikolich2021fine,
  author = {Alexandr Nikolich and Arina Puchkova},
  title = {{Fine-tuning GPT-3 for Russian Text Summarization}},
  year = {2021},
  eprint = {2108.03502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.03502},
}
Figure