XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languages

Shivprasad Sagare, Tushar Abhishek, Bhavyajeet Singh, Anubhav Sharma, Manish Gupta, Vasudeva Varma

Abstract

Multiple business scenarios require an automated generation of descriptive human-readable text from structured input data. Hence, fact-to-text generation systems have been developed for various downstream tasks like generating soccer reports, weather and financial reports, medical reports, person biographies, etc. Unfortunately, previous work on fact-to-text (F2T) generation has focused primarily on English mainly due to the high availability of relevant datasets. Only recently, the problem of cross-lingual fact-to-text (XF2T) was proposed for generation across multiple languages alongwith a dataset, XALIGN for eight languages. However, there has been no rigorous work on the actual XF2T generation problem. We extend XALIGN dataset with annotated data for four more languages: Punjabi, Malayalam, Assamese and Oriya. We conduct an extensive study using popular Transformer-based text generation models on our extended multi-lingual dataset, which we call XALIGNV2. Further, we investigate the performance of different text generation strategies: multiple variations of pretraining, fact-aware embeddings and structure-aware input encoding. Our extensive experiments show that a multi-lingual mT5 model which uses fact-aware embeddings with structure-aware input encoding leads to best results on average across the twelve languages. We make our code, dataset and model publicly available, and hope that this will help advance further research in this critical area.

Introduction

Fact-to-Text (F2T) generation is the process of converting structured data—such as a list of attributes about a person—into a coherent, readable sentence. While these systems are relatively mature for English, supporting automated sports summaries or weather reports, the technology is severely lacking for low-resource (LR) languages.

The challenge in cross-lingual fact-to-text (XF2T) generation is twofold. First, there is a massive data sparsity gap; English Wikidata contains roughly 20 facts per person entity, while many Indian languages have fewer than two. Second, there is an alignment gap, as training these systems requires expensive pairs of facts and corresponding natural language sentences across dozens of languages.

The XF2T framework transforms structured facts into natural language sentences across multiple target languages.

Methodology

To address these challenges, the XF2T framework moves beyond standard "off-the-shelf" models by introducing structural priors into the Transformer architecture:

  • Structure-aware Input Encoding: Instead of treating input as a simple string, the system uses special tokens like <S> (Subject), <R> (Relation), and <O> (Object) to delineate the boundaries of fact triples.
  • Fact-aware Embeddings: This is a key architectural innovation. A new embedding layer is added to the mT5 model to explicitly inform the encoder of the "role" of each token (e.g., whether a token belongs to a relation or an object), as shown in Figure 2.
  • Training Strategies: The researchers evaluated several configurations, including bi-lingual models, translate-pivot strategies using English, and unified multi-lingual models. They also explored multi-stage pretraining where models first learn translation before specializing in the F2T task.
The proposed architecture incorporates role-specific embeddings to explicitly model the structure of input facts within the Transformer encoder.

The XALIGNV2 Dataset

The study introduces XALIGNV2, an extension of the original XALIGN dataset. It covers 12 languages: English, Hindi, Telugu, Bengali, Gujarati, Marathi, Kannada, Tamil, and four new additions: Punjabi, Malayalam, Assamese, and Oriya. The dataset contains approximately 0.55 million instances. To ensure high quality, the authors implemented a two-phase manual annotation process with expert translators to create a rigorous gold-standard test set.

Key Results and Insights

The experiments demonstrate that a multi-lingual mT5 model utilizing both fact-aware embeddings and structure-aware encoding achieves the best overall performance. Across all 12 languages, this configuration reached an average BLEU score of 29.27, a METEOR of 53.64, and a chrF++ of 57.30.

Key takeaways include:

  1. Cross-lingual Transfer: Multi-lingual models generally outperform individual bi-lingual models, suggesting that the model effectively "borrows" linguistic patterns from high-resource languages to assist in low-resource generation.
  2. Role Clarity: Explicit role-specific embeddings significantly improve the model's ability to map specific facts to the correct sentence components, reducing confusion in complex descriptions.
  3. Scaling Benefits: Increasing model scale (e.g., moving to 12 encoder/decoder layers) boosted the BLEU score to 30.90, indicating that larger parameters continue to provide gains in this domain.

Limitations and Future Work

Despite these advances, the models still face hurdles. Hallucinations remain a concern, where the model may inject information (like a birthplace) not present in the input facts, relying instead on internal pre-trained knowledge. Furthermore, performance varies by language family; the model struggles more with the complex agglutinative morphology of South Indian (Dravidian) languages compared to Indo-Aryan languages like Hindi. Future research will focus on expanding the domain beyond person biographies to areas like sports and medical reporting.

Summary updated

Abstract

Multiple business scenarios require an automated generation of descriptive human-readable text from structured input data. Hence, fact-to-text generation systems have been developed for various downstream tasks like generating soccer reports, weather and financial reports, medical reports, person biographies, etc. Unfortunately, previous work on fact-to-text (F2T) generation has focused primarily on English mainly due to the high availability of relevant datasets. Only recently, the problem of cross-lingual fact-to-text (XF2T) was proposed for generation across multiple languages alongwith a dataset, XALIGN for eight languages. However, there has been no rigorous work on the actual XF2T generation problem. We extend XALIGN dataset with annotated data for four more languages: Punjabi, Malayalam, Assamese and Oriya. We conduct an extensive study using popular Transformer-based text generation models on our extended multi-lingual dataset, which we call XALIGNV2. Further, we investigate the performance of different text generation strategies: multiple variations of pretraining, fact-aware embeddings and structure-aware input encoding. Our extensive experiments show that a multi-lingual mT5 model which uses fact-aware embeddings with structure-aware input encoding leads to best results on average across the twelve languages. We make our code, dataset and model publicly available, and hope that this will help advance further research in this critical area.
BibTeX
@inproceedings{sagare2023xf,
  author = {Shivprasad Sagare and Tushar Abhishek and Bhavyajeet Singh and Anubhav Sharma and Manish Gupta and Vasudeva Varma},
  title = {{XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languages}},
  booktitle = {INLG},
  year = {2023},
  url = {https://openreview.net/forum?id=HbxLnqXBRf},
}
Figure