- On building voice enabled Banking Services for Multilingual Communities
Abstract
Tremendous progress in speech and language processing has brought language technologies closer to daily human life. Voice technology has the potential to act as a horizontal enabling layer across all aspects of digitization. It is especially beneficial to rural communities in scenarios like a pandemic. In this work we present our initial exploratory work towards one such direction -- building voice enabled banking services for multilingual societies. Speech interaction for typical banking transactions in multilingual communities involves the presence of filled pauses and is characterized by Code Mixing. Code Mixing is a phenomenon where lexical items from one language are embedded in the utterance of another. Therefore speech systems deployed for banking applications should be able to process such content. In our work we investigate various training strategies for building speech based intent recognition systems. We present our results using a Naive Bayes classifier on approximate acoustic phone units using the Allosaurus library.
Introduction
In many multilingual regions, speech is rarely "pure." People frequently engage in Code Mixing, where lexical items from one language (like English) are embedded into another (like Hindi), often peppered with filled pauses. Standard voice recognition systems often struggle with these linguistic nuances, especially in high-stakes environments like banking. For rural or multilingual communities, voice technology is a vital "horizontal enabling layer" for digitization. However, building robust systems for these users requires handling low-resource languages and non-standard speech patterns. This work aims to bridge this gap by creating a system that can recognize a user's intent (e.g., "Check my balance") directly from the phonetic sounds of their speech, bypassing the need for complex, language-specific text processing.
Methodology
The researchers propose a pipeline that focuses on acoustic units rather than words:
- Phonetic Extraction: The Allosaurus library, a universal phone recognizer, converts raw audio into a sequence of approximate acoustic phone units.
- Classification: These phone sequences are treated as "text" and processed by a Naive Bayes classifier.
- Smoothing Techniques: To handle the sparsity of the small dataset, two smoothing methods were compared: Add-1 (Laplace) Smoothing (a standard technique for unseen N-grams) and Absolute Discounting (which subtracts a fixed value, Delta, from seen counts to redistribute probability mass to unseen events).
- Model Combination: The final approach combines unigram, bigram, and trigram models to maximize recognition signal.
Datasets
A novel dataset was developed using a dummy banking application:
- Initial Collection: 100+ task-based dialogs in Hinglish (Hindi mixed with English) crowdsourced from fluent speakers.
- Pseudo-Parallel Data: Speakers "acoustically translated" interactions into other languages, resulting in data across six Indian languages: Hindi, Marathi, Gujarati, Punjabi, Telugu, and Bhojpuri.
- Evaluation Subset: This exploratory work used a subset of 25 utterances from 11 speakers covering five intents: Send Money, Check Balance, Check Last Transaction, Withdraw Money, and Deposit Money.
Experimental Results
The performance was measured by the system's ability to accurately classify the user's intent. Initial tests using Add-1 smoothing showed that accuracy decreased as the N-gram size increased. In a small dataset, specific three-phone sequences are so rare that the model fails to generalize, causing the Trigram model to struggle significantly compared to the Unigram model.
Moving from simple Add-1 smoothing to Absolute Discounting provided a significant jump in accuracy. For instance, the Unigram model improved from 0.56 to 0.69. The most robust results were achieved by combining the Unigram, Bigram, and Trigram models using optimized Delta values, reaching a peak accuracy of 0.83.
Discussion and Future Work
The core finding is that phonetic sequences (raw sounds) contain enough statistical signal to categorize banking intents effectively without requiring perfect text transcripts. However, the primary limitation is the current scale of the dataset; with only 25 utterances in the primary test set, these findings remain exploratory.
Future Directions: * Scaling experiments to the full multilingual dataset. * Leveraging "pseudo-parallel" data to design semi-supervised learning approaches. * Testing the system's robustness against diverse accents and noisy real-world environments.
Summary updated