TL;DR
This paper introduces TamilATIS, the first large-scale dataset for task-oriented dialogue (TOD) in Tamil, containing 4874 utterances. They demonstrate its utility by benchmarking state-of-the-art NLU models, with JointBERT achieving the best performance, paving the way for more inclusive AI in low-resource languages.
What Problem Does This Paper Address?
Task-oriented dialogue (TOD) systems, like those powering voice assistants, are becoming increasingly common. These systems allow users to accomplish specific tasks through natural language commands. However, building effective TOD systems requires vast amounts of annotated data. While significant progress has been made for high-resource languages like English, there's a considerable gap for low-resource languages, including Tamil. This lack of data hinders the development and deployment of TOD technologies for Tamil speakers, limiting inclusivity and diversity in AI. This paper aims to bridge this gap by creating a comprehensive dataset for Tamil TOD.
Methods
The core of this work is the creation of the TamilATIS dataset. The authors started by taking the well-known ATIS (Air Travel Information System) dataset, which is in English, and automatically translating it into Tamil. To ensure high-quality translations, they compared two methods: IndicTrans and Google Translate API, finding Google Translate to be superior.
Following translation, a crucial step was manual annotation. Three native Tamil speakers, with varying educational backgrounds, were employed as annotators. They were trained on the nuances of intent classification and slot filling within the context of TOD. The annotation process involved a "dry run" to calibrate annotator agreement, measured using Cohen's kappa, which showed high inter-annotator agreement. After this calibration, a final annotation phase was conducted, where annotators also had the option to reject poorly translated utterances.
For evaluating the dataset, the authors benchmarked eight state-of-the-art Natural Language Understanding (NLU) models. These models represent both generative and classification-based approaches for joint intent detection and slot filling. They also experimented with different multilingual utterance encoders for the JointBERT model.
Datasets
- ATIS (Air Travel Information System): This is the original English dataset used as the source for translation. It's a standard benchmark for intent classification and slot filling in the airline domain.
- TamilATIS: This is the newly introduced dataset for Tamil TOD. It contains 4874 translated and manually annotated utterances from the ATIS dataset, covering airline-related inquiries.
Key Insights and Ideas
- Google Translate's Superiority for Tamil: The paper highlights that for translating English TOD utterances into Tamil, Google Translate API provided significantly higher quality results compared to IndicTrans, especially in preserving domain-specific information like airline names.
- High Inter-Annotator Agreement is Achievable: Despite the complexity of slot filling and intent detection, the authors achieved high Cohen's kappa scores during their dry run, indicating that with proper training and clear guidelines, reliable manual annotation is feasible even for low-resource languages.
- JointBERT with XLM-Roberta Excels: The JointBERT architecture, which jointly models intent detection and slot filling, proved to be the most effective. When paired with the XLM-Roberta multilingual encoder, it achieved the highest performance, demonstrating the power of pre-trained multilingual models for low-resource TOD.
- The Importance of Explicit Slot-Intent Modeling: The paper suggests that models which explicitly model the relationship between slots and intents (like Stack-Propagation, SlotRefine, and GL-GIN) tend to perform better than simpler encoder-decoder architectures.
Key Results
The JointBERT model, using XLM-Roberta as its utterance encoder, achieved the best performance on the TamilATIS dataset, with an intent accuracy of 96.26% and a slot F1 score of 94.01%. This significantly outperformed other baseline models, including generative approaches like Stack-Propagation and classification-based models like SF-ID. The paper also provides a detailed breakdown of the performance of various models, showing that while generative models like Seq2seq struggled, others like SlotRefine and GL-GIN achieved competitive results.
Limitations and Future Work
- Domain Specificity: The current TamilATIS dataset is limited to the airline domain, inherited from the ATIS dataset. The authors acknowledge that expanding to other domains is a crucial next step.
- Translation Artifacts: While Google Translate was effective, there's always a potential for subtle translation artifacts that might not be perfectly captured by manual annotation.
- Future Work: The authors plan to extend this dataset to other Dravidian languages (Malayalam, Kannada, Telugu) to promote multilingual TOD development. They also aim to incorporate multi-domain utterances to create a more comprehensive resource.
Summary updated