A CTC Triggered Siamese Network with Spatial-Temporal Dropout for Speech Recognition

Yingying Gao, Junlan Feng, Tianrui Wang, Chao Deng, Shilei ZhangarXiv 2022 · DOI · arXiv

Abstract

Siamese networks have shown effective results in unsupervised visual representation learning by learning invariant representations of two augmentations. We propose an effective Siamese network to improve the robustness of End-to-End automatic speech recognition (ASR). We introduce spatial-temporal dropout for more violent disturbance and relax similarity regularization to focus on frames where connectionist temporal classification (CTC) spikes occur. Evaluated on AISHELL-1 and Librispeech, our approach achieves 7.13% and 6.59% relative error rate reductions, respectively, by improving model uniformity and enlarging CTC spikes.

Introduction

Siamese networks learn by maximizing the similarity between two different "views" of the same input. While successful in computer vision, they face challenges in ASR due to "weak disturbances" and "inefficient regularization." Standard dropout is often insufficient because speech features are highly correlated; the model can easily reconstruct missing information from adjacent frames. Furthermore, standard Siamese models regularize across all frames, including uninformative silence or noise, which can be counterproductive for phonetic alignment.

Methodology

The proposed architecture utilizes a Conformer encoder and Transformer decoder, enhanced by two core innovations designed to address the specific challenges of speech data:

  1. Spatial-Temporal Dropout: Instead of dropping individual nodes, this method removes adjacent "chunks" in the frequency (spatial) or time (temporal) domains. This "violent" disturbance forces the model to rely on broader context to maintain consistency between the two branches.
  2. CTC Triggered Similarity Regularization: A "spike filter" identifies high-probability peaks (spikes) produced by the CTC algorithm. Similarity loss is calculated only on these informative frames and their counterparts, ignoring "blank" frames.
The proposed Siamese-ASR architecture featuring spatial-temporal dropout and CTC-triggered similarity loss.

During training, the model optimizes a joint loss consisting of standard ASR loss (CTC + Attention) and the new Siamese similarity loss. At inference, the Siamese components are removed, ensuring no additional computational overhead for the end-user.

Experimental Results

The model was evaluated on AISHELL-1 (Mandarin) and Librispeech (English) using the Wenet toolkit as a baseline framework.

Performance

The "BiCTC-T-DropC" variant achieved significant improvements over standard Siamese methods like SimCLR or MoCo:

  • AISHELL-1: 7.13% relative reduction in Character Error Rate (CER).
  • Librispeech: 6.59% relative reduction in Word Error Rate (WER) on the "test-other" set.

Analysis of Representation Learning

Analysis indicates that the proposed method improves the "uniformity" of learned representations, meaning the model utilizes the available feature space more effectively, preventing different sounds from collapsing into the same representation. Additionally, the CTC spikes become more pronounced, indicating higher model confidence at critical phonetic transitions.

Quantitative analysis showing (a) improved uniformity across datasets and (b) increased frequency of high-magnitude CTC spikes compared to the baseline.

Conclusion

This work demonstrates that Siamese networks can be highly effective for ASR when paired with aggressive internal augmentations and targeted regularization. By focusing the model's attention on informative spikes rather than the entire sequence, the proposed architecture achieves state-of-the-art robustness without increasing inference latency. Future work will explore scaling these techniques to even larger, industrial-scale datasets.

Summary updated

Abstract

Siamese networks have shown effective results in unsupervised visual representation learning. These models are designed to learn an invariant representation of two augmentations for one input by maximizing their similarity. In this paper, we propose an effective Siamese network to improve the robustness of End-to-End automatic speech recognition (ASR). We introduce spatial-temporal dropout to support a more violent disturbance for Siamese-ASR framework. Besides, we also relax the similarity regularization to maximize the similarities of distributions on the frames that connectionist temporal classification (CTC) spikes occur rather than on all of them. The efficiency of the proposed architecture is evaluated on two benchmarks, AISHELL-1 and Librispeech, resulting in 7.13% and 6.59% relative character error rate (CER) and word error rate (WER) reductions respectively. Analysis shows that our proposed approach brings a better uniformity for the trained model and enlarges the CTC spikes obviously.
BibTeX
@misc{gao2022ctc,
  author = {Yingying Gao and Junlan Feng and Tianrui Wang and Chao Deng and Shilei Zhang},
  title = {{A CTC Triggered Siamese Network with Spatial-Temporal Dropout for Speech Recognition}},
  year = {2022},
  eprint = {2206.08031},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.08031},
}
Figure