Introduction
Italian administrative language, often referred to as burocratese, is characterized by its artificial and obscure nature. This complexity creates significant accessibility barriers for citizens, non-native speakers, and individuals with cognitive impairments. While Automatic Text Simplification (ATS) has advanced significantly for the English language, it remains an underdeveloped field for Italian. This study investigates whether the recent surge in instruction-tuned Large Language Models (LLMs) can bridge this gap through zero-shot prompting.
Methodology
The researchers conducted a comprehensive zero-shot evaluation of six state-of-the-art LLMs, ranging in size from 7 billion to 170 billion parameters. The models included both proprietary (GPT-3.5) and open-source (LLaMA-based variants, Vicuna, Camoscio, Guanaco, and Flan-T5-XXL) architectures.
Experimental Setup
Two distinct prompting strategies were employed to test the models' ability to follow instructions and maintain language consistency:
- Template-EN (Explicit): An English prompt specifically requesting the output in Italian.
- Template-IT (Implicit): A prompt written entirely in Italian to implicitly guide the model toward the target language.
The evaluation utilized the Admin-It benchmark, a parallel corpus of 736 Italian administrative sentence pairs. The workflow for this evaluation is illustrated below:
Evaluation Metrics
To move beyond traditional readability scores, which can be misleading, the authors employed a multi-faceted evaluation approach:
- Traditional Readability: Gulpease index and a localized version of Flesch-Kincaid.
- Specialized Italian Metrics: READ-IT, a tool specifically designed for Italian text.
- Semantic Similarity: BERTScore was used to ensure the simplified text preserved the original meaning.
- Adjusted Metric: A novel metric, $READ-IT_{all\_adjusted}$, was introduced. By calculating the product of the readability score and the BERTScore, the researchers could penalize models that produced simple but irrelevant or "hallucinated" text.
Key Results and Analysis
The study revealed a stark contrast between proprietary and open-source performance, as well as significant issues with language consistency in several models.
The Language Consistency Gap
A primary finding was the struggle of many "multilingual" open-source models to remain in the target language. As shown in the table below, Llama 2 Chat (70B) produced Italian responses only 14-15% of the time, frequently reverting to English despite explicit instructions.
Performance Highlights
- GPT-3.5: The clear leader, GPT-3.5 demonstrated superior editing capabilities, sometimes even outperforming human-written simplifications in readability while maintaining high semantic fidelity.
- Flan-T5-XXL: This 11B parameter model emerged as the most robust open-source alternative. It followed instructions reliably, staying in Italian 99% of the time and achieving a high BERTScore (0.88).
- The Hallucination Problem: Models like Guanaco-65B highlighted the limitations of raw readability metrics. While it achieved high readability scores, its low semantic similarity indicated it was often generating fluent but factually unrelated Italian sentences.
Conclusion
While proprietary models like GPT-3.5 currently offer the most effective solution for Italian ATS, the study underscores the potential and the pitfalls of open-source alternatives. The "hallucination" of simple text and the failure to maintain language consistency are critical hurdles. Future research must focus on improving factual consistency and exploring the benefits of fine-tuning on domain-specific Italian administrative data to ensure that automated simplification remains both accessible and accurate.
Summary updated