Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language

Bhavuk Singhal, Sindhuja Gopalan, Amrith Krishna, Malolan ChetlurEMNLP 2023

TL;DR

This paper tackles the challenge of Inverse Text Normalization (ITN) for Tamil, a morphologically rich, low-resource language. They explore three neural ITN models (seq2seq, non-autoregressive editor, and tagger+rules) and find that data augmentation and bootstrapping significantly improve performance, with the seq2seq model achieving the best overall results.

What Problem Does This Paper Address?

Inverse Text Normalization (ITN) is the process of converting spoken language, often containing verbalized numbers, dates, and times, into their written form. This is crucial for improving the readability and downstream processing of spoken language data. However, ITN presents significant challenges, especially for morphologically rich and agglutinative languages like Tamil. These challenges include:

  • Spelling Variations: Dialects and transcription errors lead to diverse ways of verbalizing the same concept.
  • Punarchi: Tamil's phonetic rules can obscure word boundaries, making it difficult to identify individual words.
  • Agglutination: Tamil's morphology allows for many suffixes and clitics to be attached to a root word, creating complex surface forms.
  • Low-Resource Nature: Limited availability of labeled data for Tamil makes training robust ITN models difficult.

The paper aims to address these challenges by developing and evaluating neural ITN systems for Tamil, focusing on numbers and temporal expressions.

Methods

The researchers experimented with three neural ITN approaches:

  1. Sequence-Tagger with Rules: This approach first uses a sequence tagger (based on XLM-Roberta) to identify ITN entities. These identified spans are then processed by a rule-based system for transformation. The tagging scheme is inspired by IOBES and BILOU.
  2. Non-Autoregressive Text Editor (NAR): This method, based on the FELIX model, uses a tagging model and an insertion model. The tagging model predicts whether to retain or delete tokens, and the insertion model generates the final output based on a masked sequence.
  3. Seq2Seq Model: A standard auto-regressive sequence-to-sequence model (using byT5) is employed to directly predict the desired written form from the verbalized input.

To overcome the low-resource challenge, the paper heavily utilizes bootstrapping (iteratively expanding a small seed set of labeled data) and data augmentation (creating new training examples through techniques like substitution matrices for spelling variations and generating numeral forms).

Datasets

The paper leverages several resources for data collection and augmentation:

  • In-house Speech Collection: Raw ASR transcriptions from code-mixed Tamil-English conversations form the base.
  • Publicly Available Resources:
    • Tamil Text-Normalization Corpus (Tyagi et al., 2021)
    • Unlabelled Tamil Corpus (Kakwani et al., 2020)
    • Parallel Tamil-English Corpus (Ramesh et al., 2022)
    • Tamil and English Dictionaries (Tamildict, Shabdkosh, Encyclopedia)
  • Generated Datasets: Through bootstrapping and data augmentation, they created a substantial training dataset. A "seed set" of 324 ITN entries was used for cold-start experiments, expanded to 10,000 words for warm-start scenarios.

Key Insights and Ideas

  • Agglutination is a Major Hurdle: The paper strongly emphasizes how Tamil's agglutinative nature, combined with phonetic variations (Punarchi), creates significant surface-form variations that are hard for ITN systems to handle.
  • Data Augmentation is Crucial: For low-resource languages like Tamil, bootstrapping and data augmentation are not just helpful but essential for achieving competitive performance. They significantly boost the effectiveness of all tested models.
  • Seq2Seq Models Shine with More Data: While a tagger-based approach might perform well in a fully supervised setting, the seq2seq model demonstrates superior scalability and performance gains when augmented with more data.
  • Temporal Expressions are Tricky: While numbers are a core ITN task, the paper also includes temporal expressions, highlighting their unique challenges, such as potentially affecting verb tenses and being non-contiguous.

Key Results

  • Seq2Seq Dominance: The seq2seq model achieved the best overall F-Score of 98.05% after data augmentation and bootstrapping, outperforming the tagger+rules system by 2.06%.
  • Bootstrapping Benefits: Bootstrapping alone provided significant performance improvements, ranging from 9.13% to 14.12%, even with a small seed set.
  • Data Augmentation's Impact: Data augmentation further boosted performance, with seq2seq seeing a 5.24% improvement on top of bootstrapping gains.
  • Tagger's Strength: In fully supervised settings and without data augmentation, the tagger+rules system often performed best.
  • Category Performance: All systems struggled most with the "Money" category, with over 56% of mispredictions going to the "Other Numerals" category.

Limitations and Future Work

  • Limited Semiotic Classes: The current work focuses on numerals and temporal expressions. The authors suggest expanding to other classes like abbreviations and acronyms.
  • Joint Modeling Potential: They believe that jointly modeling ITN with related tasks like grammatical error correction and punctuation restoration could further improve performance.
  • Automated vs. Human-in-the-Loop: While fully automated bootstrapping can yield improvements, the human-in-the-loop (HitL) approach in the cold-start setting and the automated warm-start setting showed further performance gains. The paper leaves open the exploration of optimal combinations.

Summary updated

BibTeX
@inproceedings{singhal2023scaling,
  author = {Bhavuk Singhal and Sindhuja Gopalan and Amrith Krishna and Malolan Chetlur},
  title = {{Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure