TL;DR
This paper introduces XINFOTABS, the first multilingual dataset for tabular natural language inference (NLI), expanding the English-only INFOTABS dataset to ten diverse languages. They demonstrate that while current multilingual models show promise, significant challenges remain in cross-lingual tabular reasoning, highlighting the need for more inclusive benchmarks.
What Problem Does This Paper Address?
Natural Language Inference (NLI) on semi-structured data like tables is a crucial but challenging task for NLP systems. While progress has been made, existing benchmarks and research are largely confined to English, creating a significant barrier to developing truly multilingual NLI systems. This lack of multilingual resources hinders the ability of AI to understand and reason about tabular information across different languages, limiting its applicability in a globalized world. The authors aim to bridge this gap by creating a multilingual dataset and evaluating existing multilingual models on this new challenge.
Methods
The core of this work involves creating a multilingual dataset through machine translation. The authors leverage state-of-the-art machine translation models (mBART50, M2M100, MarianMT) to translate the English INFOTABS dataset into ten target languages. To address the unique challenges of translating tabular data (e.g., lack of sentential structure, handling named entities), they propose a pipeline that incorporates context (table category) and uses Named Entity Recognition (NER) to handle entities that should be transliterated rather than translated. They also employ rigorous translation quality verification using automatic metrics and human evaluation. For model evaluation, they test various multilingual models (mBERT, XLM-ROBERTa) using different training and testing strategies, including English-only training, language-specific fine-tuning, and multilingual fine-tuning.
Datasets
- INFOTABS: The original English dataset for tabular NLI, used as the source for translation.
- XINFOTABS: The newly introduced multilingual tabular NLI dataset, an extension of INFOTABS to ten languages: English (en), German (de), French (fr), Spanish (es), Afrikaans (af), Russian (ru), Chinese (zh), Korean (ko), Hindi (hi), and Arabic (ar). This dataset aims to foster linguistic inclusion in tabular reasoning research.
Key Insights and Ideas
- The "Tabular Translation" Challenge: Translating tabular data is significantly harder than translating natural language sentences. The lack of grammatical structure and the need to preserve specific semantic relationships between headers and cells pose unique difficulties for standard MT models.
- Context is King (for Tables): Simply translating table cells independently leads to poor results. Incorporating table category as context significantly improves translation quality, demonstrating the importance of understanding the domain.
- Multilingual Models Show Promise, But Gaps Remain: While models like XLM-ROBERTa perform comparably across languages, they still lag behind human performance, especially on adversarial or complex reasoning tasks. This highlights that current multilingual models are not yet fully equipped for robust cross-lingual tabular NLI.
- High-Resource Languages Still Lead: Models trained on or fine-tuned with high-resource languages (like English) tend to exhibit better cross-lingual transfer capabilities, a trend observed across various experiments.
- Human Evaluation is Crucial: Despite advancements in automatic metrics, human evaluation remains essential for truly assessing translation quality and model performance, especially for nuanced tasks like NLI.
Key Results
- The XINFOTABS dataset was successfully created for ten languages, expanding the scope of multilingual tabular NLI research.
- The proposed translation pipeline, incorporating category context and NER, achieved good translation quality, with German and Arabic showing the most and least exact translations, respectively.
- XLM-ROBERTa (XNLI) generally outperformed mBERT across various evaluation settings and languages.
- Language-specific fine-tuning showed benefits, particularly for low-resource languages, though English-based pre-finetuning often yielded the best results.
- Human evaluation scores consistently surpassed model performance across all languages and reasoning types, indicating significant room for improvement.
- The dataset proved to be a challenging benchmark, with models exhibiting varying degrees of cross-lingual transferability, with Spanish showing strong transfer for XLM-R(XNLI) on the α1 set.
Limitations and Future Work
- Translation Quality Variation: While the translation pipeline is effective, variations in quality across languages, particularly for Arabic, can impact downstream NLI performance.
- Model Performance Gaps: Despite improvements, current multilingual models still struggle with complex reasoning types and adversarial examples, falling short of human-level performance.
- Future Work: The authors suggest enhancing table representation beyond simple linearization, exploring more sophisticated NER handling, and investigating direct human translation or fine-grained post-translation verification for even higher quality datasets. They also propose extending the dataset's utility to other tasks like multilingual question answering.
Summary updated