Hierarchical Softmax for End-to-End Low-resource Multilingual Speech Recognition

Qianying Liu, Zhuo Gong, Zhengdong Yang, Yuhang Yang, Sheng Li, Chenchen Ding, Nobuaki Minematsu, Hao Huang, Fei Cheng, Chenhui Chu, Sadao KurohashiarXiv 2022 · arXiv

TL;DR

This paper introduces a novel approach for low-resource multilingual speech recognition by applying Hierarchical Softmax (H-Softmax) at the decoding stage. By organizing vocabulary tokens into a Huffman tree based on their frequencies across neighboring languages, the model enables cross-lingual knowledge sharing and significantly reduces computational complexity. Additionally, a new GPU vectorization algorithm ensures that this hierarchical approach runs faster than traditional vanilla softmax on modern hardware.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have made massive strides recently, but they still primarily cater to only about 100 of the world's 7,000+ languages. Developing ASR for low-resource languages is notoriously difficult due to the lack of sufficient training data.

While multilingual models help by learning universal features across languages, prior work has focused almost entirely on sharing knowledge within the encoder (the part of the model processing the audio). Explicit knowledge transfer at the decoding stage (turning features into text tokens) remains largely unexplored. Furthermore, as vocabulary sizes grow, calculating the standard "vanilla" softmax over all possible tokens becomes a major computational bottleneck, scaling linearly ($O(V)$) with the vocabulary size.

Methods

The authors propose a two-part solution integrated into a Conformer-Transformer architecture:

  1. Huffman Tree for Cross-Lingual Sharing: Based on the linguistic insight that neighboring or related languages share similar token frequency distributions, the authors pool monolingual text data to calculate token frequencies. They construct a binary Huffman tree where similar tokens across languages naturally cluster together. This shared structure allows low-resource tokens to benefit from the representations learned by their higher-resource neighbors.
  2. GPU-Vectorized Hierarchical Softmax (H-Softmax): Instead of calculating probabilities for every single token in the vocabulary, H-Softmax redefines the problem as a series of binary classification steps down the Huffman tree, reducing the decoding complexity to $O(\log V)$. Because traditional H-Softmax implementations are optimized for CPUs, the authors designed a novel vectorization algorithm using fixed Sign and Bias matrices to allow efficient parallel path calculations on GPUs.
Fig. 1. The flowchart of the proposed method. The blue lines stands for determination relation, and the red line stands for how the data goes through the model.

Datasets

The approach was evaluated on two distinct setups:

  • Speech-to-Text: Data was sampled from the Common Voice Corpus 11.0, focusing on three language groups: Romance (Catalan, Spanish, French, Italian, Portuguese), Slavic (Belarusian, Czech, Polish, Russian, Ukrainian), and Turkic (Bashkir, Kyrgyz, Turkish, Tatar, Uzbek). Training sets were artificially downsampled to an average of 30 hours per language to simulate a low-resource environment.
  • Speech-to-Phoneme: The UCLA Phonetic Corpus (comprising 97 languages) was used to evaluate zero-shot cross-lingual performance, training on 87 languages and testing on 10 completely unseen languages.

Key Insights and Ideas

  • Decoding as a Vehicle for Transfer Learning: The most striking insight is that H-Softmax isn't just an optimization trick to speed up computation; it acts as a regularizer. By forcing tokens with similar frequencies to share paths in a binary tree, the model implicitly transfers structural linguistic knowledge to data-scarce tokens.
  • Simplicity Wins: The Huffman tree relies purely on raw token frequencies from monolingual data. It requires no complex, hand-crafted cross-lingual phonetic mappings or expert linguistic alignment, yet it successfully captures cross-lingual similarities.
  • Overcoming the GPU Bottleneck: Tree-traversal algorithms are historically notoriously slow on GPUs due to their sequential nature. By padding the paths to a uniform depth and using matrix operations (shown in Figure 2), the authors successfully vectorized the log-probability calculations.
Fig. 2. A typical H-Softmax tree structure. Leaf w3 has a virtual child with the same probability of aligning each leaf node to the same depth, so it is conceptually possible for path vectorization.

Key Results

  • Accuracy Improvements: In speech-to-text tasks within the same language group, H-Softmax consistently outperformed vanilla softmax. For instance, the overall Character Error Rate (CER%) for the Turkic group dropped from 16.0% to 14.0%. When trained across all 15 languages simultaneously, the overall CER% for the Romance group dropped significantly from 12.4% to 9.7%.
  • Zero-Shot Phoneme Recognition: On the UCLA Phonetic Corpus, the proposed model outperformed the baseline by a massive margin, slashing the overall Phone Error Rate (PER%) from 85.2% to 64.7%.
  • Decoding Speedup: Thanks to the vectorization algorithm, the Real-Time Factor (RTF) improved across the board. For Russian decoding, the RTF dropped from 0.026 to 0.020, demonstrating that the model is both more accurate and computationally efficient.

Limitations and Future Work

While utilizing token frequency distributions via Huffman coding is highly effective, it only captures "shallow" statistical relationships. The authors note that frequency alone doesn't capture deeper semantic or phonetic nuances. As a next step, they plan to design binary tree codes that incorporate richer, more semantically meaningful features, such as token embeddings, directly into the tree structure.

Summary updated

Abstract

Low-resource speech recognition has been long-suffering from insufficient training data. In this paper, we propose an approach that leverages neighboring languages to improve low-resource scenario performance, founded on the hypothesis that similar linguistic units in neighboring languages exhibit comparable term frequency distributions, which enables us to construct a Huffman tree for performing multilingual hierarchical Softmax decoding. This hierarchical structure enables cross-lingual knowledge sharing among similar tokens, thereby enhancing low-resource training outcomes. Empirical analyses demonstrate that our method is effective in improving the accuracy and efficiency of low-resource speech recognition.
BibTeX
@misc{liu2022hierarchical,
  author = {Qianying Liu and Zhuo Gong and Zhengdong Yang and Yuhang Yang and Sheng Li and Chenchen Ding and Nobuaki Minematsu and Hao Huang and Fei Cheng and Chenhui Chu and Sadao Kurohashi},
  title = {{Hierarchical Softmax for End-to-End Low-resource Multilingual Speech Recognition}},
  year = {2022},
  eprint = {2204.03855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.03855},
}
Figure