Is It Really That Simple? Prompting Large Language Models for Automatic Text Simplification in Italian

Debora Nozza, Giuseppe AttanasioCLiC-it 2023

Introduction

Italian administrative language, often referred to as burocratese, is characterized by its artificial and obscure nature. This complexity creates significant accessibility barriers for citizens, non-native speakers, and individuals with cognitive impairments. While Automatic Text Simplification (ATS) has advanced significantly for the English language, it remains an underdeveloped field for Italian. This study investigates whether the recent surge in instruction-tuned Large Language Models (LLMs) can bridge this gap through zero-shot prompting.

Methodology

The researchers conducted a comprehensive zero-shot evaluation of six state-of-the-art LLMs, ranging in size from 7 billion to 170 billion parameters. The models included both proprietary (GPT-3.5) and open-source (LLaMA-based variants, Vicuna, Camoscio, Guanaco, and Flan-T5-XXL) architectures.

Experimental Setup

Two distinct prompting strategies were employed to test the models' ability to follow instructions and maintain language consistency:

  1. Template-EN (Explicit): An English prompt specifically requesting the output in Italian.
  2. Template-IT (Implicit): A prompt written entirely in Italian to implicitly guide the model toward the target language.

The evaluation utilized the Admin-It benchmark, a parallel corpus of 736 Italian administrative sentence pairs. The workflow for this evaluation is illustrated below:

Figure 3: Overview of the zero-shot evaluation workflow using the Admin-It dataset and various evaluation metrics.

Evaluation Metrics

To move beyond traditional readability scores, which can be misleading, the authors employed a multi-faceted evaluation approach:

  • Traditional Readability: Gulpease index and a localized version of Flesch-Kincaid.
  • Specialized Italian Metrics: READ-IT, a tool specifically designed for Italian text.
  • Semantic Similarity: BERTScore was used to ensure the simplified text preserved the original meaning.
  • Adjusted Metric: A novel metric, $READ-IT_{all\_adjusted}$, was introduced. By calculating the product of the readability score and the BERTScore, the researchers could penalize models that produced simple but irrelevant or "hallucinated" text.

Key Results and Analysis

The study revealed a stark contrast between proprietary and open-source performance, as well as significant issues with language consistency in several models.

The Language Consistency Gap

A primary finding was the struggle of many "multilingual" open-source models to remain in the target language. As shown in the table below, Llama 2 Chat (70B) produced Italian responses only 14-15% of the time, frequently reverting to English despite explicit instructions.

Figure 6: Performance comparison across models showing BERTScore and the percentage of responses correctly detected as Italian.

Performance Highlights

  • GPT-3.5: The clear leader, GPT-3.5 demonstrated superior editing capabilities, sometimes even outperforming human-written simplifications in readability while maintaining high semantic fidelity.
  • Flan-T5-XXL: This 11B parameter model emerged as the most robust open-source alternative. It followed instructions reliably, staying in Italian 99% of the time and achieving a high BERTScore (0.88).
  • The Hallucination Problem: Models like Guanaco-65B highlighted the limitations of raw readability metrics. While it achieved high readability scores, its low semantic similarity indicated it was often generating fluent but factually unrelated Italian sentences.

Conclusion

While proprietary models like GPT-3.5 currently offer the most effective solution for Italian ATS, the study underscores the potential and the pitfalls of open-source alternatives. The "hallucination" of simple text and the failure to maintain language consistency are critical hurdles. Future research must focus on improving factual consistency and exploring the benefits of fine-tuning on domain-specific Italian administrative data to ensure that automated simplification remains both accessible and accurate.

Summary updated

BibTeX
@inproceedings{nozza2023is,
  author = {Debora Nozza and Giuseppe Attanasio},
  title = {{Is It Really That Simple? Prompting Large Language Models for Automatic Text Simplification in Italian}},
  booktitle = {CLiC-it},
  year = {2023},
}
Figure