Optimizing Speech Multi-View Feature Fusion through Conditional Computation

Weiqiao Shan, Yuhao Zhang, Yuchen Han, Bei Li, Xiaofeng Zhao, Yuang Li, Min Zhang, Hao Yang, Tong Xiao, Jingbo Zhu

Abstract

Modern speech translation systems often rely on either traditional spectral features (FBanks) or self-supervised learning (SSL) features (S-features). While FBanks offer stability and high peak performance, S-features provide rapid convergence but may suffer from information loss. Fusing these two views is desirable but hampered by "gradient conflicts"—opposing update directions that lead to training instability. We propose a generalized fusion framework using a Gradient-Sensitive Gating Network (GSGN) to dynamically deconflict gradients and a multi-stage dropout strategy to handle training instability. Our method achieves a 1.24x training speedup and consistently improves BLEU scores on the MuST-C benchmark.

Introduction: The Challenge of Multi-View Fusion

In speech processing, representations typically fall into two categories: traditional Filter Banks (FBanks) and learned Self-Supervised Learning (SSL) features. FBanks are reliable but slow to converge. Conversely, SSL features (S-features), pre-trained on vast unlabeled datasets, enable rapid initial learning but often plateau below FBank performance due to quantization-induced information loss.

Figure 1: (a) Comparison of validation BLEU scores for FBanks and S-features over training steps. (b) Illustration of gradient conflict and instability during multi-view training.

As shown in Figure 1, while both features have merits, a simple combination is not straightforward. The primary obstacle identified is Gradient Conflict: the update directions derived from FBanks and S-features often oppose one another, creating a "tug-of-war" effect. Empirical analysis reveals that up to 32% of gradients contain conflicting components, significantly hindering joint optimization.

Proposed Method: Gradient-Sensitive Gating

To address these challenges, we introduce a fusion framework based on conditional computation. The architecture centers on two innovations: the Gradient-Sensitive Gating Network (GSGN) and a multi-stage dropout strategy.

Gradient-Sensitive Gating Network (GSGN)

The GSGN replaces standard concatenation or addition with a dynamic weighting mechanism. It incorporates a "Deconflict" operation: when the cosine similarity between the FBank gradient and the S-feature gradient is negative (indicating an angle greater than 90 degrees), the network adjusts the weights to project out the conflicting components. This ensures that the shared parameters of the acoustic encoder receive a harmonious update signal.

Figure 2: Overview of the proposed architecture, highlighting the Fusion Layer and the internal structure of the Gradient-Sensitive Gating Network (GSGN).

Multi-stage Dropout Strategy

The relative importance of FBanks and S-features shifts during training. To manage this transition and increase robustness, we employ a three-stage training process. By dynamically sampling between individual features and the fused representation at varying thresholds, the model learns to be robust to each input view while eventually converging on the optimized fused representation.

Experimental Setup

The framework was evaluated on the MuST-C dataset for English-to-German (En-De), English-to-Spanish (En-Es), and English-to-French (En-Fr) translation tasks. We utilized LibriSpeech for acoustic encoder initialization and WMT datasets for textual component pre-training.

Key Insights and Results

Our analysis yielded several critical findings:

  • The "Seesaw Phenomenon": Improving performance on one feature view often degrades the other without explicit deconfliction.
  • Dynamic Feature Prioritization: The gating network initially prioritizes S-features to exploit their fast convergence (acting as a "starter motor") before shifting focus to FBanks for final high-quality refinement.
  • Performance Gains: The proposed "S2T-GSGN+Drop" model outperformed the FBank-only baseline across all tasks, notably achieving a +0.20 BLEU gain in En-Es.
  • Efficiency: The method delivered an average 1.24x training speedup, reaching peak performance significantly faster than single-feature baselines.

Conclusion

By explicitly addressing gradient conflicts through conditional computation, our framework successfully combines the rapid convergence of SSL features with the high-fidelity performance of spectral features. This approach not only improves translation quality but also significantly reduces the computational resources required for training state-of-the-art speech translation models.

Summary updated

BibTeX
@inproceedings{shan2025optimizing,
  author = {Weiqiao Shan and Yuhao Zhang and Yuchen Han and Bei Li and Xiaofeng Zhao and Yuang Li and Min Zhang and Hao Yang and Tong Xiao and Jingbo Zhu},
  title = {{Optimizing Speech Multi-View Feature Fusion through Conditional Computation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890091},
}
Figure