Transfer Learning for Russian Legal Text Simplification

Mark Athugodage, Olga Mitrofanova, Vadim GudkovCOLING 2024

TL;DR

This paper introduces the first dataset and models for simplifying Russian legal texts, a crucial step towards making legal information more accessible. They fine-tuned T5 and GPT models on this new dataset, evaluating their performance using automatic metrics and human judgment, finding that GPT models, particularly when trained on reduced text, show promising results.

What Problem Does This Paper Address?

Legal documents are notoriously complex, filled with specialized jargon, intricate sentence structures, and references to other laws, making them difficult for the average citizen to understand. This complexity creates a barrier to accessing justice, as the principle "ignorance of the law does not excuse anyone" places the burden of understanding on the individual. While governments acknowledge this issue and advocate for simpler language, the complexity of Russian legal texts has even been observed to increase. This paper tackles the challenge of automatically simplifying Russian legal texts to improve their readability and accessibility for a broader audience, including those with disabilities or who are not native Russian speakers.

Methods

The authors explored three approaches to text simplification using T5 and GPT model architectures. They fine-tuned these models on a newly created parallel corpus of Russian legal documents and their simplified commentaries.

  • T5-RLS2000: A larger T5 model fine-tuned on approximately 2,000 pairs of original legal texts and their simplified versions. This model is designed to process single sentences.
  • GPT-simplifier-large-text: A larger GPT model fine-tuned on the same 2,000 pairs of texts, using the full text without compression.
  • GPT-simplifier25: A smaller GPT model, also fine-tuned on the 2,000 pairs, but with a pre-processing step that involved filtering documents to remove introductory phrases, long citations, and, if the document remained too long, keeping only the last 35 sentences. This approach aimed to test the hypothesis that data economy could positively impact simplification performance.

The pre-processing for the T5 model involved text alignment using a Natural Language Inference (NLI) model to ensure logical similarity between original and simplified texts.

Datasets

The core contribution in terms of data is the creation of a novel parallel corpus for Russian legal text simplification.

  • "Rossiyskaya Gazeta Legal Papers" Corpus: This dataset was extracted from the "Rossiyskaya Gazeta" newspaper, which publishes official legal documents and their commentaries. The corpus consists of 2,963 pairs of original legal texts and their simplified versions. The data spans from December 2008 to November 2022. The authors also performed dataset filtration, removing documents with more than 40,000 tokens, original texts less than 400 tokens, and pairs where the original text was smaller than its simplified version. The final corpus used for fine-tuning contained 2,017 document pairs.

Key Insights and Ideas

  • The need for domain-specific simplification: Existing general-purpose language models are not inherently suited for the complex domain of legal text simplification, necessitating fine-tuning on specialized data.
  • Data economy can improve performance: The GPT-simplifier25 model, trained on a filtered and reduced subset of the data, achieved better results on readability indices, suggesting that focusing on the most informative parts of legal documents can be beneficial.
  • Different models excel at different metrics: While T5-RLS2000 performed best on the SARI metric, GPT models showed stronger results on readability indices. This highlights the importance of using a diverse set of evaluation metrics.
  • Human evaluation is crucial: Automatic metrics alone are insufficient. Human evaluation, especially by legal experts, provided valuable insights into the practical utility and quality of the simplified texts.
  • Simplified texts can outperform human commentaries: In human evaluations, the models' simplified texts were rated higher than the commentaries provided by "Rossiyskaya Gazeta."

Key Results

  • Automatic Evaluation:
    • SARI: T5-RLS2000 achieved the highest SARI score of 54.96, which is considered state-of-the-art compared to previous work.
  • BERTScore: The T5 model also showed strong performance on BERTScore.
  • ROUGE: All models performed poorly on ROUGE metrics, which are more suited for summarization.
  • Readability Indices:
    • Gunning Fog Index & Flesch-Kincaid Grade Level: The GPT-simplifier25 model demonstrated significant improvements in readability, reducing the Gunning Fog Index from 59.5 to 41.8 and FKGL from 26.58 to 18.23. The GPT-simplifier-large-text also showed improvements.
  • Human Evaluation:
    • Respondents rated the simplified texts on a scale of 0-10.
  • Legal experts rated GPT-simplifier25 slightly higher than other models.
  • Overall, the proposed models received higher ratings than the "Rossiyskaya Gazeta" commentaries.

Limitations and Future Work

  • Limitations:
    • Simplified texts may still have shortcomings and require verification by experts or editors.
  • The T5 model cannot process multi-sentence texts directly.
  • The filtering process for GPT-simplifier25, while beneficial, might lead to the omission of some context, though the authors argue that the structure of legal documents makes this less impactful.
  • Future Work:
    • Improve existing simplification technology to allow for immediate user consumption of generated texts.
  • Fine-tune Longformer Encoder-Decoder and LongT5 models for simplification.
  • Further reduce defects in generated texts.

Summary updated

BibTeX
@inproceedings{athugodage2024transfer,
  author = {Mark Athugodage and Olga Mitrofanova and Vadim Gudkov},
  title = {{Transfer Learning for Russian Legal Text Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure