TL;DR
This paper introduces T2X, a new dataset for generating text from structured data in isiXhosa, an agglutinative language. They propose SSPG, a novel subword-aware model that outperforms existing methods for this challenging low-resource task, demonstrating the critical need for subword modeling in agglutinative languages.
What Problem Does This Paper Address?
Data-to-text generation, the task of converting structured data (like tables or triples) into human-readable text, has seen significant progress, primarily in English. However, most existing datasets and models are tailored for morphologically simple languages like English. This leaves a significant gap in understanding how to perform this task for low-resource, agglutinative languages, where words are formed by stringing together many morphemes. These languages present unique challenges: simple word-level templates are insufficient, and subword units become crucial for accurate representation and generation. The authors aim to address this by creating a new dataset and developing models specifically designed for this linguistic context.
Methods
The paper introduces two main methodological directions:
- Dedicated Architectures: They propose the Subword Segmental Pointer Generator (SSPG), a novel LSTM-based encoder-decoder model. SSPG is designed to jointly learn three key tasks: mapping data triples to text, segmenting output text into subwords, and copying entities directly from the input data. This is a significant departure from traditional models that operate on whole words.
- Pretrained Language Models (PLMs): The authors explore fine-tuning existing PLMs, including multilingual models like mT5 and its African-language-adapted version, Afri-mT5. They also investigate an unconventional approach: fine-tuning machine translation (MT) models trained for English-to-isiXhosa.
A key innovation is unmixed decoding, a new decoding strategy for SSPG that separately extracts probabilities from the model's different components (character decoder, lexicon, and copy mechanism) to select the most likely next subword. This differs from standard dynamic decoding, which proved less effective for SSPG.
Datasets
- Triples-to-isiXhosa (T2X): A newly created dataset for isiXhosa data-to-text generation. It's based on the 1-triple subset of the WebNLG dataset and manually translated by native isiXhosa speakers. T2X covers 15 DBpedia categories and is designed to highlight the challenges of agglutinative languages.
- Finnish Hockey Dataset: Used to evaluate the generalizability of their findings to another agglutinative language.
Key Insights and Ideas
- Subword Modeling is Essential: For agglutinative languages like isiXhosa, subword segmentation is not just beneficial but essential for accurate data-to-text generation, unlike in English where word-level models suffice.
- Dedicated Models Can Outperform PLMs: For low-resource agglutinative languages, a carefully designed dedicated model like SSPG can outperform general-purpose PLMs, especially when those PLMs are not specifically trained or adapted for the target language's linguistic characteristics.
- MT Models as a Strong Baseline: Fine-tuning existing machine translation models (especially English-to-isiXhosa) can be a surprisingly effective strategy for data-to-text generation in low-resource settings, sometimes even surpassing large multilingual PLMs.
- Copying vs. Generating: The paper highlights the trade-off between copying entities directly and translating them. SSPG's ability to jointly learn both is crucial for handling entities that should be preserved (like names) and those that need translation.
- Decoding Matters: The choice of decoding strategy (unmixed decoding for SSPG) significantly impacts performance, underscoring the need for task-specific decoding methods.
Key Results
- SSPG Performance: SSPG significantly outperforms other dedicated models trained from scratch on T2X, achieving improvements of 2.21 chrF++ and 1.11 BLEU over the next best model. It also outperforms mT5 variants.
- MT Finetuning Success: Finetuning English-to-isiXhosa BPE MT models yielded the best overall results on T2X, outperforming even SSPG. Similar success was observed for Finnish.
- Extractive Evaluation: The authors developed an extractive evaluation framework that assesses how accurately models capture the data's content (subject/object copying vs. translation). SSPG showed strong performance in balancing these decisions, while other models struggled.
- Finnish Results: The relative performance of models on the Finnish dataset mirrored the T2X results, suggesting generalizability of the findings for agglutinative languages.
Limitations and Future Work
- Dataset Size and Scope: The experiments are limited to two datasets and two languages. The authors acknowledge that claims about SSPG's generalizability to other languages and more complex data-to-text tasks are limited.
- Training Time: SSPG's dynamic programming algorithm for subword segmentation increases training time compared to simpler models like PG.
- Bias in Data: The T2X dataset, derived from WebNLG, may contain biases and outdated information related to Western content, which could impact model performance and fairness.
- Hallucinations: Models, including SSPG, can still hallucinate or generate incorrect entities, limiting their current reliability.
- Future Work: The authors suggest further investigation into finetuning bilingual NMT models as an alternative to PLMs and exploring SSPG's architecture for other low-resource agglutinative languages. They also highlight the need for more data-to-text datasets for African languages.
Summary updated