Introduction
Automatic Speech Recognition (ASR) has reached impressive milestones in transcribing "read" speech, such as audiobooks or dictated commands. However, these systems often struggle with spontaneous, conversational speech characterized by disfluencies, fillers ("ums" and "ahs"), and mumbled articulations. This performance gap is particularly pronounced in low-resource languages where labeled spontaneous speech data is scarce.
Traditional ASR models are trained for strict acoustic faithfulness. When audio is noisy or poorly enunciated, these models lack a "semantic safety net," often substituting acoustically similar but contextually nonsensical words. To address this, researchers from IIT Bombay proposed AMPS (ASR with Multimodal Paraphrase Supervision), a technique that leverages the semantic flexibility of paraphrasing to improve ASR robustness.
Methodology: The Multimodal Paraphrase Framework
The AMPS framework is built upon SeamlessM4T, a multimodal architecture featuring dual encoders for speech and text that feed into a shared decoder. The core innovation lies in introducing a secondary training objective: paraphrasing.
Dual Pathways
The model operates through two simultaneous pathways:
- Speech-to-Text (S2T) Pathway: The standard ASR route where the model learns to transcribe audio input $X$ into a target transcript $Y$.
- Text-to-Text (T2T) Pathway: An auxiliary route where the model learns to transform the transcript $Y$ into a semantically equivalent paraphrase $Y'$.
By sharing the decoder across both tasks, the model learns to associate acoustic features not just with specific word sequences, but with underlying semantic meanings.
Thresholded Supervision ($\text{AMPS}_\tau$)
A key challenge in auxiliary task training is balancing faithfulness with flexibility. Forcing a model to paraphrase clear audio can lead to unnecessary deviations from the ground truth. To mitigate this, the researchers introduced a thresholding mechanism ($\tau$). The paraphrasing loss is only activated when the ASR loss exceeds a certain value, indicating that the model is struggling with the audio input.
This "selective triggering" ensures that the model remains strictly faithful to clear audio while utilizing semantic guidance from the paraphrase objective when the signal is degraded or ambiguous.
Experimental Setup
The study evaluated AMPS across several Indic languages (Hindi, Marathi, Malayalam, and Kannada) using the IndicVoices dataset, which contains natural, extempore speech. To test cross-family generalizability, experiments were also conducted on Nyanja, an African language from the Zambezi-Voice dataset. Training typically utilized approximately 50 hours of mixed conversational and read speech, with additional low-resource simulations using as little as 5 hours.
Key Results and Analysis
The AMPS approach consistently outperformed standard ASR-only baselines across all tested languages and metrics.
Performance Gains
The system achieved relative Word Error Rate (WER) reductions of up to 5%. Beyond WER, improvements were observed in semantic metrics like METEOR and BERTScore, suggesting that the transcripts produced by AMPS are more contextually coherent.
Robustness on Difficult Samples
The "Hard 100" test—evaluating the 100 most difficult sentences for a standard ASR model—revealed the true strength of the AMPS approach. As shown in the results table, the improvement on these challenging samples ($\Delta$Hard) was significantly higher than the average improvement across the entire dataset. This confirms that the paraphrase supervision effectively acts as a fallback mechanism for difficult acoustic signals.
Human Evaluation
In addition to automated metrics, human annotators rated the outputs on a 0-5 scale. AMPS consistently received higher scores, with annotators noting that even when the model did not match the ground truth verbatim, it often produced more meaningful and grammatically correct sentences than the baseline.
Discussion and Future Directions
While AMPS shows significant promise, several areas for improvement remain. The researchers noted that WER is a conservative metric for this approach; a perfect synonym is counted as an error even if the meaning is preserved. Future evaluations may incorporate more sophisticated LLM-based semantic scoring.
Furthermore, the threshold $\tau$ is currently a manually tuned hyperparameter. Future iterations could explore making this parameter learnable, allowing the model to dynamically adjust its reliance on semantic supervision. Finally, addressing code-switching (e.g., "Hinglish") by incorporating code-switched paraphrases remains a priority for improving real-world applicability in multilingual contexts.
Conclusion
AMPS demonstrates that multimodal paraphrase supervision can provide a vital "semantic safety net" for ASR systems. By selectively integrating text-to-text knowledge, the model becomes more resilient to the disfluencies and noise inherent in spontaneous speech, paving the way for more robust speech recognition in low-resource and real-world scenarios.
Summary updated