TransChat: Cross-Lingual Instant Messaging for Indian Languages

Diptesh Kanojia, Shehzaad Dhuliawala, Abhijit Mishra, Naman Gupta, Pushpak Bhattacharyya

Introduction

In a country with 22 official languages and over a billion people, cross-lingual communication remains a significant hurdle in critical sectors like agriculture, healthcare, and tourism. While Machine Translation (MT) has advanced significantly, it often fails in the context of instant messaging (IM) due to the informal nature of "chat-speak." TransChat is an open-source, cross-platform IM application designed to bridge this gap by integrating a specialized pre-processing pipeline with Statistical Machine Translation (SMT).

The Challenge of Informal Input

Standard MT engines typically struggle with IM for three primary reasons:

  1. Informal Input: Chat users frequently use abbreviations (e.g., "Hw r u?"), slang, and repeated characters for emphasis ("goooood"), which standard engines cannot parse.
  2. Resource Intensity: Many MT systems are too computationally heavy for real-time mobile use.
  3. User Experience: Most interfaces do not provide a natural environment that reconciles user expectations with the current limitations of MT technology.

System Architecture and Methods

The core of TransChat is a multi-stage server-side pipeline that cleans and standardizes text before it reaches the translation engine. This architecture moves the heavy computational lifting away from the mobile device and onto a scalable server (using XMPP via the ejabberd server).

The TransChat multi-stage processing pipeline for converting informal chat input into translated output.

The Processing Pipeline

  • Text Compression: To handle emotional emphasis, the system reduces character repetitions of three or more down to two (e.g., "pleeease" becomes "pleease") and uses a dictionary search to find the most likely valid word.
  • SMT-based Normalization: TransChat treats normalization as a translation task. It uses Moses, a Phrase-Based SMT system, to "translate" abbreviations like "2mrw" into "tomorrow." This allows the system to use context to resolve ambiguities, such as deciding if "yr" means "year" or "your."
  • Translation Engine: The system uses the Sata-Anuvadak framework for translation between English and five Indian languages: Hindi, Gujarati, Punjabi, Marathi, and Malayalam.
  • Post-processing Transliteration: For Out-of-Vocabulary (OOV) words or Named Entities, the system uses transliteration to ensure words appear correctly in the target script rather than remaining in English.

Data and Training

The normalization and translation models were trained using the Stanford Normalization Corpora (9,122 pairs), crowd-sourced chat data (215 pairs of acronyms), and internet slang dictionaries. A language model of 15,000 English words was harvested to support the normalization SMT.

Key Results and Insights

The research demonstrates that the primary bottleneck in chat translation is input quality. By cleaning the input first, standard engines see massive performance boosts. For Hindi translation, the BLEU score jumped from 0.0306 (unnormalized) to 0.1991 (normalized)—a nearly seven-fold improvement.

A sample chat session between a buyer and a farmer, demonstrating real-time translation and transliteration in a practical scenario.

Qualitatively, human evaluators rated the system highly on usability, fluency, and adequacy. Inter-annotator agreement (measured via Fliess’ Kappa) showed "substantial agreement," suggesting a consistent and helpful user experience.

Limitations and Future Work

Currently, the sophisticated pre-processing (compression and normalization) is only implemented for English input. Future iterations aim to expand coverage to all official Indian languages and incorporate advanced features such as emotional analysis, real-time word suggestions, and euphemism detection for sensitive dialogues.

Summary updated

BibTeX
@inproceedings{kanojia2015transchat,
  author = {Diptesh Kanojia and Shehzaad Dhuliawala and Abhijit Mishra and Naman Gupta and Pushpak Bhattacharyya},
  title = {{TransChat: Cross-Lingual Instant Messaging for Indian Languages}},
  booktitle = {ICON},
  year = {2015},
}
Figure