Towards Disfluency Annotated Corpora for Indian Languages

Chayan Kochar, Vandan Vasantlal Mujadia, Pruthwik Mishra, Dipti Misra SharmaCOLING 2024

TL;DR

This paper addresses the critical need for disfluency annotation and synthetic data generation for Indian languages, which are currently underserved in NLP research. The authors introduce detailed annotation guidelines tailored to Indian linguistic nuances and a novel algorithm for synthetically generating disfluent data, aiming to improve downstream NLP tasks like machine translation.

What Problem Does This Paper Address?

Spoken language is inherently messy. Unlike clean, written text, natural conversations are peppered with disfluencies – hesitations, repetitions, false starts, and self-corrections. These linguistic "noise" elements, while often filtered out in traditional NLP pipelines, are crucial for understanding the full spectrum of human communication and can significantly impact the performance of downstream tasks like machine translation (MT) and question answering.

While extensive research exists on disfluencies in English, Indian languages remain a significant gap. This is largely due to a lack of standardized annotation guidelines that account for the unique linguistic characteristics of India's diverse language landscape, and a scarcity of labeled data. Without this foundational work, developing robust NLP models for Indian languages is severely hampered. This paper aims to bridge this gap by providing detailed annotation guidelines and a method for generating synthetic disfluent data specifically for Indian languages.

Methods

The core of this work lies in two main contributions:

  1. Annotation Guidelines: The authors developed a comprehensive set of annotation guidelines specifically for Indian languages. This involved not only defining standard disfluency categories (filler words, pet phrases, repetitions, repair, false starts) but also meticulously analyzing and incorporating linguistic nuances unique to Indian languages. This includes addressing issues like code-mixing, reduplication, and emphasis, which can be easily mistaken for disfluencies but carry specific semantic or pragmatic weight.
  2. Synthetic Data Generation Algorithm: To overcome the data scarcity issue, the researchers propose a novel algorithm for synthetically generating disfluent data. This algorithm takes clean text and injects disfluencies based on learned probabilities and linguistic rules. It aims to create realistic disfluent samples by considering factors like sentence length, the probability of injecting multiple disfluencies, and the specific characteristics of different disfluency types (e.g., partial word repairs, phrase repairs, code-mixed repairs). The algorithm also accounts for the unique nature of pet phrases, which are speaker-dependent.

Datasets

The study utilizes a combination of real-world conversational data and synthetically augmented data:

  • Real-World Data: Simulated conversations were sourced from the IIT Madras SPRING lab, originally acquired from vendors. This data was manually filtered to focus on natural conversational audios exhibiting disfluencies. Data was collected for Hindi, Marathi, Bengali, Kannada, and Tamil. For Telugu, conversations were sourced from YouTube videos with a creative commons license.
  • PMIndia Corpus: Approximately 1 hour of data from the PMIndia corpus was used for each language to synthetically add disfluencies, further diversifying the dataset.

The paper does not explicitly state the total size of the real-world annotated dataset but provides a distribution of data in hours for synthetic and real data per language in Table 1.

Key Insights and Ideas

  • Indian Languages are Not Monolithic: The paper strongly emphasizes that a one-size-fits-all approach to disfluency annotation is insufficient for Indian languages. Nuances like code-mixing and specific repetition patterns require dedicated handling.
  • Disfluencies are Context-Dependent: The authors highlight that the same word or phrase might be a disfluency in one context but not in another, depending on its semantic contribution. This underscores the need for careful, context-aware annotation.
  • Synthetic Data is Crucial for Low-Resource Languages: Given the challenges in collecting and annotating large amounts of real-world data for Indian languages, synthetic augmentation is presented as a vital strategy to improve model generalization and address data scarcity.
  • Repair and False Starts are Complex: The paper notes that while agreement is high for simpler disfluency types like filler words and pet phrases, there are more disagreements in annotating repair and false start cases, often related to the precise span of the reparandum and alteration.

Key Results

  • High Inter-Annotator Agreement: The study achieved a strong inter-annotator agreement (IAA) score of 82-92% across languages using Cohen's kappa coefficient, indicating the robustness of their annotation guidelines. Pet phrases and filler words showed the highest agreement.
  • Performance on Downstream Tasks: While not explicitly detailing end-to-end task performance, the paper presents weighted F1 scores for disfluency identification experiments (Table 4). Hindi and Tamil achieved F1 scores around 59-60%, while Marathi and Telugu had lower scores (around 8-9%), attributed to less real-world data.
  • Synthetic Data Mimics Real-World Disfluencies: The authors' synthetic data generation algorithm, when tweaked for a disfluency percentage of around 21%, aimed to better mimic real-world data and improve the distribution of disfluency categories.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the synthetic augmentation process requires constant refinement to better emulate real-world disfluencies.
  • The performance of Marathi and Telugu was lower, potentially due to a smaller amount of real-world data available for these languages.
  • The authors note that while their tool is in-house developed, further improvements to ASR systems with lower Word Error Rates (WER) could enhance the overall workflow.
  • Future Work:
    • Continuously refining annotation criteria and gathering more datasets, especially from real-world sources.
  • Exploring the use of semantic knowledge (e.g., punctuation, grammatical chunks, POS tags) to improve algorithm performance for both synthetic generation and identification.
  • Further investigation into code-mixed disfluencies and their impact.

Summary updated

BibTeX
@inproceedings{kochar2024towards,
  author = {Chayan Kochar and Vandan Vasantlal Mujadia and Pruthwik Mishra and Dipti Misra Sharma},
  title = {{Towards Disfluency Annotated Corpora for Indian Languages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure