Towards Automatic Finnish Text Simplification

Anna Dmitrieva, Jörg TiedemannCOLING 2024

TL;DR

This paper tackles the challenge of automatic text simplification for Finnish, a language with limited resources. The authors significantly expand the Finnish-Easy Finnish parallel corpus and develop the first baseline simplification models for the language using mBART and Finnish GPT, making their data and models publicly available.

What Problem Does This Paper Address?

Automatic Text Simplification (ATS) aims to make complex texts easier to understand. While ATS has seen significant progress in languages like English, it lags behind in low-resource languages. Finnish, despite having a well-established "Easy Finnish" variant used in practice, lacked substantial parallel data and dedicated simplification models. This paper addresses this gap by creating a larger, sentence-aligned dataset and training initial simplification models, paving the way for future research and applications in Finnish text accessibility.

Methods

The core of this work involves two main stages: dataset augmentation and model development.

Dataset Augmentation: * Document and Sentence Alignment: The authors employed sophisticated techniques to align existing Finnish and Easy Finnish documents and then further refined this to create sentence-level alignments. They experimented with various embedding models (LASER, LaBSE, MPNet, DistilUSE) and alignment algorithms (Vecalign, MASSAlign, Bertalign) to find the optimal methods for this task. * Alignment Strategies: For document alignment, they used cosine similarity between averaged sentence embeddings and a candidate generation/re-scoring method from Thompson and Koehn (2020). For sentence alignment, they sought methods that could handle one-to-one, many-to-one, and one-to-many alignments, allow crossing links, and ensure exclusivity. They evaluated Vecalign, Bertalign, and a custom cosine similarity matrix approach.

Model Development: * Model Architectures: They fine-tuned two powerful pre-trained language models: * mBART: A multilingual sequence-to-sequence denoising autoencoder known for its strong performance in text generation tasks. * Finnish GPT: A large language model specifically trained for Finnish, adapted using instruction fine-tuning. * Training: Both models were fine-tuned for 10 epochs using the newly created sentence-aligned dataset.

Datasets

  • Parallel Corpus of Finnish and Easy-to-read Finnish (Dmitrieva et al., 2022): This served as the initial dataset.
  • Yle Finnish News Archive 2011-2018 ("general" archive): A large collection of standard Finnish news articles.
  • Yle News Archive Easy-to-read Finnish 2011-2018: A collection of Easy Finnish news articles.

The paper significantly augmented the original parallel corpus, increasing document-aligned data by 6.5 times and creating a new sentence-aligned version with over 12,000 sentence pairs. They also created a small, manually aligned golden test set for sentence alignment evaluation.

Key Insights and Ideas

  • Alignment is Tricky: Sentence alignment, even with advanced tools, proved to be a genuinely difficult task for their dataset, suggesting that human annotators also struggle with certain sentence pairs.
  • Embedding Choice Matters: LaBSE and LASER embeddings consistently yielded the best results for document alignment, highlighting their effectiveness for Finnish.
  • Model Performance Varies: While Vecalign performed well for document alignment, sentence alignment results were not as high across the board, indicating that different methods excel on different linguistic phenomena or dataset characteristics.
  • Finnish GPT Shows Promise: The Finnish GPT model achieved a higher SARI score than mBART, suggesting its potential for generating simpler Finnish text.
  • Data Availability is Crucial: The authors emphasize the importance of making their expanded dataset and trained models publicly available to foster further research in Finnish text simplification.

Key Results

  • Dataset Expansion: The dataset was expanded to include over 8,000 document pairs and 12,000 sentence pairs, covering nearly 6.5 years of news content.
  • Document Alignment: Vecalign with LASER embeddings achieved the best performance in document alignment.
  • Sentence Simplification Models:
    • Finnish GPT achieved a SARI score of 44.63, outperforming mBART (37.612).
  • Quality estimation reports (Table 6) show that both models achieved reasonable compression ratios and sentence splitting, with mBART being more conservative in its edits.
  • Example Simplifications: Qualitative examples (Table 7) show that mBART tends to make fewer changes, while Finnish GPT sometimes produces shorter, simpler sentences but may not always reach the simplification level of human-generated "Easy Finnish."

Limitations and Future Work

  • Sentence Alignment Accuracy: The authors acknowledge that sentence alignment is not perfect, and some erroneous alignments are expected. They provide cost scores for transparency.
  • Automatic Metric Limitations: They note that automatic metrics like SARI don't fully capture grammaticality or coherence, and human evaluation is still crucial.
  • Computational Resources: Training required significant GPU resources, with Finnish GPT fine-tuning needing multiple nodes.
  • Future Work:
    • Improving sentence alignment accuracy.
  • Developing document-level simplification models for Finnish.
  • Exploring more sophisticated data-driven simplification evaluation approaches.

Summary updated

BibTeX
@inproceedings{dmitrieva2024towards,
  author = {Anna Dmitrieva and Jörg Tiedemann},
  title = {{Towards Automatic Finnish Text Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure