Spend Scarce Code-Switched Speech Only on the Router: Alternating Block Training for Bilingual Mixture-of-Experts ASR

Speech recognizers for people who mix languages mid-sentence, a practice called code-switching, are short of the one kind of data they need most. Monolingual Mandarin and English recordings run to thousands of hours. Natural Mandarin-English code-switched speech runs to tens or a few hundred, and it is lopsided: the ASRU 2019 corpus is 89% Mandarin tokens, and in SEAME about half to 70% of English switches are a single inserted word. Recent systems split the model into language-specific "experts" (sub-networks specialized for one language) plus a small "router" that decides, frame by frame, which expert should handle the audio. They then train everything jointly on a mixture of monolingual and code-switched data. Under that setup the scarce switched data is spread across millions of expert parameters. Meanwhile the router, which actually has to learn where switches happen, gets pulled toward the dominant language.

We propose to train these models the way signal-processing engineers solve coupled beamforming problems: by alternating optimization, meaning the parameters are split into blocks and the solver repeatedly fixes one block while optimizing the other. Concretely, the experts are updated on abundant monolingual speech while the router is frozen. The router and language-boundary components are then updated only on scarce code-switched speech while the experts are frozen, and the two phases repeat. The claim is that routing scarce data to a small parameter block, while abundant data feeds the large block, beats joint training. We also predict that the advantage grows as code-switched data shrinks. That prediction is what separates this idea from a generic training trick.

Core Claim

In a bilingual mixture-of-experts layer the output is a router-weighted sum of expert outputs,

$$h(x) = \sum_{e \in \{\text{zh}, \text{en}\}} g_e(x;\phi)\, f_e(x;\theta_e),$$

where $x$ is an acoustic frame representation, $g_e(x;\phi)$ is the router's weight on expert $e$ with router parameters $\phi$, and $f_e(x;\theta_e)$ is the output of expert $e$ with parameters $\theta_e$. The output is bilinear in $\phi$ and $\theta$. This is structurally the same coupling as the passive phase shifts and the active beamformer in reflecting-surface systems. There, alternating optimization handles the coupling by solving each block with the other fixed (Beamforming Design with Partial Channel Estimation and Feedback for FDD RIS-Assisted Systems). Hybrid Digital and Analog Beamforming Design for Large-Scale Antenna Arrays splits a precoder into a small digital stage and a large analog stage and optimizes them by coordinate descent. We borrow only the decomposition, not the solvers.

The new ingredient is the pairing of parameter blocks with data sources. Joint training lets the router chase whichever expert currently lowers loss on Mandarin-heavy batches, and lets the experts absorb the other language's phonetics, so the two co-adapt toward collapse. Freezing the router while experts train on monolingual data holds language assignment fixed, so each expert sees only its own language. Freezing the experts while the router trains on code-switched data limits the switched data's job to a low-dimensional problem: locating boundaries, including one-word insertions. We expect the parameter count fitted to scarce data to set how much it overfits. That reasoning predicts the gap between alternating and joint training should widen as code-switched hours fall from 200 toward 10.

Relation to Prior Work

Three code-switching systems supply the architecture we reuse, and all three train jointly. Improving End-to-End Modeling For Mandarin-English Code-Switching Using Lightweight Switch-Routing Mixture-of-Experts places a top-1 router with Mandarin and English experts in the last Conformer encoder block and a decoder block. It trains all parameters together on monolingual corpora plus 200h of ASRU data. CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition adds MoE adapters with gated cross-attention, a language-diarization decoder, and language-wise CTC losses on pseudo-labels to force experts to specialize. Aligning Speech to Languages to Enhance Code-Switching Speech Recognition derives frame-level language pseudo-labels from decoder cross-attention to train a language-ID head, and needs a weighted variant of its loss to cope with Mandarin dominance.

These papers motivate the proposal in two ways. CAMEL's masking and the weighted alignment loss are evidence that plain joint gradients do not produce clean specialization under imbalance. We keep their components but change which data updates which parameters, and when.

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception supplies the closest optimization precedent. It alternates across dataset-objective pairs at each step and reports that this beats summing losses. It alternates data, however, and updates all parameters every step, in a data-rich multimodal regime. We alternate parameter blocks tied to data sources, in a data-poor regime. Whether its data-only alternation already captures our gain is one of our central tests.

On the data side, Speech Collage: Code-Switched Audio Generation by Collaging Monolingual Corpora builds synthetic switched audio by splicing monolingual units. It reports up to 34.4% relative MER reduction over a monolingual baseline, but also an MER gap of up to 8.9% versus real SEAME data and some monolingual degradation. These results constrain us: synthetic data is useful but acoustically mismatched. A router-only update on collaged data may be less hurt by that mismatch than a full-model update. We treat that as a secondary hypothesis.

Research Program

The backbone follows the switch-routing Conformer/Transformer design, with expert layers in the upper encoder and decoder blocks. We also add CAMEL-style gated cross-attention and a diarization decoder in a second configuration. Parameters fall into two blocks. Block A holds the language experts, plus optionally the shared lower encoder. Block B holds the router, the gated cross-attention, and the diarization and language-ID heads.

Training begins with a short joint warm-up on monolingual data, so that the experts are not random when the router first trains. It then cycles. For $K$ steps, Block B is frozen while Block A trains on Mandarin (AISHELL-1) and English (LibriSpeech 460h) batches. During this phase the router is frozen as well, so expert assignment is fixed and the experts learn only their own language. For $M$ steps, Block A is frozen while Block B trains on code-switched batches. The router phase uses the ASR loss together with token-language supervision derived from transcripts, as in the alignment-loss work.

We study three questions. The first is how the advantage depends on code-switched data volume, using ASRU subsets of 10, 30, 100, and 200 hours and SEAME. The second is how the alternation period $K/M$ matters, ranging from one cycle, which is plain two-stage training, to per-step alternation. The third is whether the block-data pairing itself is the active ingredient. Alternating optimization guarantees only stationary points, so we report seed variance and examine router entropy and expert-usage statistics over training to see whether alternation avoids the collapse joint training shows.

Baselines

The naive reference point is a single dense Conformer trained on the same monolingual plus code-switched mixture with no experts. We also report a monolingual-only model to anchor how much code-switched data helps at all.

The strongest existing method on this exact question is the jointly trained switch-routing MoE (Improving End-to-End Modeling For Mandarin-English Code-Switching Using Lightweight Switch-Routing Mixture-of-Experts). It reports a total MER of 10.30% on the ASRU code-switching test set, against 10.57% for a bi-encoder and 10.62% for a multilingual baseline, and a pure-Mandarin CER of 2.47% versus 2.73%. CAMEL is the strongest richer architecture, reducing ASRU200 MER from 11.35% to 10.55% (CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition). For SEAME, CAMEL reports only an approximately 5% relative gain and gives no absolute numbers in the file. These published figures come from different data mixtures, so we re-implement both under our exact data, step count, and tuning budget rather than comparing to the printed numbers.

Fairness is enforced in three ways. Every method sees the same examples for the same total optimizer steps. Each method gets the same number of hyperparameter trials, about twenty, over learning rate and loss weights. Alternating runs count frozen-block steps against the same total.

Two ablations isolate the key ingredient. The first, the swapped pairing, trains the router on monolingual data and the experts on code-switched data. The second, the shuffled pairing, alternates blocks but feeds both phases the full mixture. If either matches the proposed method, the claim about data-to-block assignment fails.

Beating the baseline means a reproducible MER reduction on ASRU and SEAME at 200h, with no loss on AISHELL-1 or LibriSpeech. The more important target is a relative advantage that clearly grows as code-switched hours shrink, outside seed variance across three seeds.

Why not just…?

Why not just pretrain the experts on monolingual data, freeze them, and fine-tune the router once on code-switched data? This is one round of alternation and is the most serious threat. If experts pretrained in isolation already sit in a good place, repeated alternation adds nothing. The weakness we expect is that experts never adapt to the router's eventual assignment, especially for single-word insertions, where acoustic context crosses expert boundaries. The $K/M$ sweep includes this one-cycle point explicitly. If it matches multi-cycle alternation, we will report that the block-data pairing, not the alternation, is what matters. That would still be a useful and simpler recipe.

Why not just keep joint training and rebalance it with oversampling or loss weights? Prior evidence suggests static reweighting helps but needs per-dataset tuning: the weighted alignment loss gave 8.6% relative improvement on ASRU over its baseline (Aligning Speech to Languages to Enhance Code-Switching Speech Recognition). We include a tuned oversampling-plus-reweighting baseline under the same trial budget. Our prediction is that reweighting closes the gap at 200h but not at 10–30h, because it still fits all parameters to the scarce data.

Why not just alternate datasets per step, as in the multimodal alternating-gradient work, without freezing anything? That method beats loss summing at scale (Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception). But every step still updates the router on Mandarin-dominated batches and the experts on switched data. We run exactly this variant with matched schedules. If it equals our method, block freezing is unnecessary.

Why not just generate more synthetic code-switched audio? Collaging and TTS help but leave a real-data gap of up to 8.9% MER on SEAME and can hurt monolingual accuracy (Speech Collage: Code-Switched Audio Generation by Collaging Monolingual Corpora). The method is orthogonal to synthetic data rather than a replacement, so we test it with and without collaged data added to the router phase. A good outcome is that alternation recovers part of the synthetic-to-real gap.

Proposed Experiments

The main grid crosses method (dense, joint MoE, CAMEL-style joint, tuned reweighting, data-only alternation, two-stage, multi-cycle block alternation, both pairing ablations) with code-switched data volume (ASRU 10/30/100/200h). All runs share AISHELL-1 and LibriSpeech 460h as monolingual data. We report MER on the ASRU test set, CER on AISHELL-1, and WER on LibriSpeech.

SEAME serves as the spontaneous, short-switch test. There we break MER down by switch length and score English single-word insertions separately, since that is where router uncertainty should concentrate. A second language pair, Arabic-English, uses MGB-2 and Tedlium3 as monolingual sources, collaged data, and ArzEn or ESCWA for evaluation. This checks whether the scaling claim transfers. The file gives no reported ArzEn result, and for ESCWA it gives only Speech Collage's relative reductions.

The decisive checks are three. First, the method-versus-data-volume curve must show the alternating advantage widening as code-switched hours drop. Second, both pairing ablations must lose to the proposed pairing. Third, monolingual CER and WER must not degrade relative to the monolingual-only model. If the curve is flat, the hypothesis about scarce data fitting a small block is wrong, and we will say so.