Abstract
Siamese networks have shown effective results in unsupervised visual representation learning by learning invariant representations of two augmentations. We propose an effective Siamese network to improve the robustness of End-to-End automatic speech recognition (ASR). We introduce spatial-temporal dropout for more violent disturbance and relax similarity regularization to focus on frames where connectionist temporal classification (CTC) spikes occur. Evaluated on AISHELL-1 and Librispeech, our approach achieves 7.13% and 6.59% relative error rate reductions, respectively, by improving model uniformity and enlarging CTC spikes.
Introduction
Siamese networks learn by maximizing the similarity between two different "views" of the same input. While successful in computer vision, they face challenges in ASR due to "weak disturbances" and "inefficient regularization." Standard dropout is often insufficient because speech features are highly correlated; the model can easily reconstruct missing information from adjacent frames. Furthermore, standard Siamese models regularize across all frames, including uninformative silence or noise, which can be counterproductive for phonetic alignment.
Methodology
The proposed architecture utilizes a Conformer encoder and Transformer decoder, enhanced by two core innovations designed to address the specific challenges of speech data:
- Spatial-Temporal Dropout: Instead of dropping individual nodes, this method removes adjacent "chunks" in the frequency (spatial) or time (temporal) domains. This "violent" disturbance forces the model to rely on broader context to maintain consistency between the two branches.
- CTC Triggered Similarity Regularization: A "spike filter" identifies high-probability peaks (spikes) produced by the CTC algorithm. Similarity loss is calculated only on these informative frames and their counterparts, ignoring "blank" frames.
During training, the model optimizes a joint loss consisting of standard ASR loss (CTC + Attention) and the new Siamese similarity loss. At inference, the Siamese components are removed, ensuring no additional computational overhead for the end-user.
Experimental Results
The model was evaluated on AISHELL-1 (Mandarin) and Librispeech (English) using the Wenet toolkit as a baseline framework.
Performance
The "BiCTC-T-DropC" variant achieved significant improvements over standard Siamese methods like SimCLR or MoCo:
- AISHELL-1: 7.13% relative reduction in Character Error Rate (CER).
- Librispeech: 6.59% relative reduction in Word Error Rate (WER) on the "test-other" set.
Analysis of Representation Learning
Analysis indicates that the proposed method improves the "uniformity" of learned representations, meaning the model utilizes the available feature space more effectively, preventing different sounds from collapsing into the same representation. Additionally, the CTC spikes become more pronounced, indicating higher model confidence at critical phonetic transitions.
Conclusion
This work demonstrates that Siamese networks can be highly effective for ASR when paired with aggressive internal augmentations and targeted regularization. By focusing the model's attention on informative spikes rather than the entire sequence, the proposed architecture achieves state-of-the-art robustness without increasing inference latency. Future work will explore scaling these techniques to even larger, industrial-scale datasets.
Summary updated