WebNLG-IT: Construction of an aligned RDF-Italian corpus through Machine Translation techniques

Michael Oliverio, Pier Felice Balestrucci, Alessandro Mazzei, Valerio BasileACL 2025

Abstract

WebNLG-IT is the first high-quality, aligned RDF-to-text corpus for the Italian language. Developed by researchers at the University of Turin, the dataset was created using a multi-stage pipeline that combines neural machine translation (NMT) with rigorous semi-automatic post-editing. This work addresses the scarcity of structured data-to-text resources for Italian, providing a benchmark that enables Large Language Models (LLMs) to generate fluent Italian text from RDF triples with performance levels approaching those of English-language models.

Introduction

In Natural Language Generation (NLG), the "RDF-to-text" task involves converting structured data triples—such as (Abilene_Texas, isPartOf, Texas)—into natural language sentences like "Abilene is part of Texas." While the English WebNLG corpus serves as a robust gold standard for this task, non-English languages often lack comparable resources. Previous "silver-standard" datasets for languages like German or Russian frequently suffer from translation artifacts. WebNLG-IT fills this gap for the Italian language, offering a validated resource for training and evaluating data-to-text systems.

Methodology

The creation of WebNLG-IT followed a rigorous multi-stage pipeline designed to minimize translation errors and ensure grammatical fluency.

NMT Selection and Initial Translation

The researchers benchmarked four major Neural Machine Translation (NMT) systems: DeepL, OpusMT, M2M-100, and NLLB-200. DeepL was selected as the primary translation engine due to its superior performance, achieving a 0.93 BERTScore and 0.67 BLEU score on the initial translation of the English WebNLG 3.0 corpus.

Semi-Automatic Post-Editing

To move beyond a simple "silver" translation, the team implemented a hybrid error-correction workflow to ensure the corpus felt "natively" Italian rather than translated.

The semi-automatic post-editing pipeline for error detection and removal.

As shown in the pipeline above, the process involved:

  1. Automatic Error Detection: Using the Language Tool Python library and SpaCy to flag potential grammatical and syntactic errors.
  2. Manual Review: Human annotators filtered out false positives, which were frequently triggered by misclassified named entities.
  3. Rule-Based Refinement: Hand-written rules were applied to resolve 3,325 specific stylistic and grammatical issues, ensuring consistency across the 45,040 verbalizations.

The WebNLG-IT Dataset

The resulting WebNLG-IT corpus is a fully aligned Italian version of WebNLG 3.0. It contains over 45,000 verbalizations across various data unit complexities.

Distribution of data units and verbalizations in the WebNLG-IT corpus.

The dataset is partitioned into Train, Dev, and Test sets, maintaining the structural integrity of the original English source while providing high-quality Italian references. The post-editing phase identified 198 unique error types, with over 91% of corrections involving stylistic changes to noun phrases.

Experimental Results and Discussion

To evaluate the utility of WebNLG-IT, the authors fine-tuned several LLMs, including Llama 3.1, Mistral-Nemo, and Italian-specialized models like LLaMAntino, using QLoRA.

Model Performance

Mistral-Nemo emerged as a top performer, achieving a BLEU score of 0.54 and a METEOR score of 0.74 on the WebNLG-IT test set. Interestingly, this multilingual model often outperformed Italian-specific models, suggesting that broad multilingual pre-training provides a strong foundation for specialized data-to-text tasks.

Handling Complexity with SGA

For complex inputs containing multiple triples, the researchers implemented a "Split, Generate, Aggregate" (SGA) pipeline. This modular approach successfully reduced factual hallucinations and omissions. However, it introduced a "verbosity" trade-off; the zero-shot aggregation step often produced more repetitive or "chatty" text compared to human references.

Conclusion

WebNLG-IT represents a significant step forward for Italian NLG. By combining high-quality NMT with a structured post-editing process, the researchers have created a resource that significantly outperforms previous silver-standard datasets. Future work will focus on refining the SGA pipeline's aggregation step to produce more concise and natural-sounding summaries, further closing the gap between machine-generated and human-written Italian text.

Summary updated

BibTeX
@inproceedings{oliverio2025webnlg,
  author = {Michael Oliverio and Pier Felice Balestrucci and Alessandro Mazzei and Valerio Basile},
  title = {{WebNLG-IT: Construction of an aligned RDF-Italian corpus through Machine Translation techniques}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure