Abstract
WebNLG-IT is the first high-quality, aligned RDF-to-text corpus for the Italian language. Developed by researchers at the University of Turin, the dataset was created using a multi-stage pipeline that combines neural machine translation (NMT) with rigorous semi-automatic post-editing. This work addresses the scarcity of structured data-to-text resources for Italian, providing a benchmark that enables Large Language Models (LLMs) to generate fluent Italian text from RDF triples with performance levels approaching those of English-language models.
Introduction
In Natural Language Generation (NLG), the "RDF-to-text" task involves converting structured data triples—such as (Abilene_Texas, isPartOf, Texas)—into natural language sentences like "Abilene is part of Texas." While the English WebNLG corpus serves as a robust gold standard for this task, non-English languages often lack comparable resources. Previous "silver-standard" datasets for languages like German or Russian frequently suffer from translation artifacts. WebNLG-IT fills this gap for the Italian language, offering a validated resource for training and evaluating data-to-text systems.
Methodology
The creation of WebNLG-IT followed a rigorous multi-stage pipeline designed to minimize translation errors and ensure grammatical fluency.
NMT Selection and Initial Translation
The researchers benchmarked four major Neural Machine Translation (NMT) systems: DeepL, OpusMT, M2M-100, and NLLB-200. DeepL was selected as the primary translation engine due to its superior performance, achieving a 0.93 BERTScore and 0.67 BLEU score on the initial translation of the English WebNLG 3.0 corpus.
Semi-Automatic Post-Editing
To move beyond a simple "silver" translation, the team implemented a hybrid error-correction workflow to ensure the corpus felt "natively" Italian rather than translated.
As shown in the pipeline above, the process involved:
- Automatic Error Detection: Using the Language Tool Python library and SpaCy to flag potential grammatical and syntactic errors.
- Manual Review: Human annotators filtered out false positives, which were frequently triggered by misclassified named entities.
- Rule-Based Refinement: Hand-written rules were applied to resolve 3,325 specific stylistic and grammatical issues, ensuring consistency across the 45,040 verbalizations.
The WebNLG-IT Dataset
The resulting WebNLG-IT corpus is a fully aligned Italian version of WebNLG 3.0. It contains over 45,000 verbalizations across various data unit complexities.
The dataset is partitioned into Train, Dev, and Test sets, maintaining the structural integrity of the original English source while providing high-quality Italian references. The post-editing phase identified 198 unique error types, with over 91% of corrections involving stylistic changes to noun phrases.
Experimental Results and Discussion
To evaluate the utility of WebNLG-IT, the authors fine-tuned several LLMs, including Llama 3.1, Mistral-Nemo, and Italian-specialized models like LLaMAntino, using QLoRA.
Model Performance
Mistral-Nemo emerged as a top performer, achieving a BLEU score of 0.54 and a METEOR score of 0.74 on the WebNLG-IT test set. Interestingly, this multilingual model often outperformed Italian-specific models, suggesting that broad multilingual pre-training provides a strong foundation for specialized data-to-text tasks.
Handling Complexity with SGA
For complex inputs containing multiple triples, the researchers implemented a "Split, Generate, Aggregate" (SGA) pipeline. This modular approach successfully reduced factual hallucinations and omissions. However, it introduced a "verbosity" trade-off; the zero-shot aggregation step often produced more repetitive or "chatty" text compared to human references.
Conclusion
WebNLG-IT represents a significant step forward for Italian NLG. By combining high-quality NMT with a structured post-editing process, the researchers have created a resource that significantly outperforms previous silver-standard datasets. Future work will focus on refining the SGA pipeline's aggregation step to produce more concise and natural-sounding summaries, further closing the gap between machine-generated and human-written Italian text.
Summary updated