How Small a Code-Switching Effect Can We Actually Measure?

Code-switching means alternating between languages inside one sentence or conversation, such as Mandarin and English in Singapore or Egyptian Arabic and English in Cairo. Research on systems that handle it routinely reports small differences and treats them as real. One system "preserves" switches better than another. A new speech recognizer lowers word error on mixed-language segments by a few relative percent. A language model "prefers" natural switches at slightly above chance. These differences are measured on very few of the events that matter. Switch points are sparse inside mostly monolingual speech. Test sets come from a few dozen speakers. Often the measuring tool is itself badly broken: in First Steps Towards Voice Anonymization for Code-Switching Speech, the Whisper recognizer used to score utility reaches a match error rate (MER) of about 39% on code-switched speech, against 1.5–3% on monolingual benchmarks. Rare-event physics has faced this situation for decades: counting a handful of events with an imperfect detector. Its answer is not just "compute a p-value." Physicists calibrate how the detector responds to a known injected signal. They report the expected sensitivity before looking at the result. They charge themselves for every subgroup they searched.

This note proposes importing that practice into code-switching evaluation as a measurement protocol and auditing published claims with it. The protocol has three parts. First, it calibrates the evaluation instrument (a recognizer or a language-model scorer) by injecting known changes at switch points and measuring how much of each change the instrument registers. Second, it reports a "minimum detectable effect": the smallest true difference the benchmark could reveal given the counted switch points, speaker clustering and instrument noise. Third, it corrects significance for the many language pairs, word categories and subsets that are inspected. The output is a re-assessment of whether specific headline findings exceed the sensitivity floor of the benchmarks they were measured on.

Core Claim

The central claim is that, in code-switching evaluation, the dominant uncertainty is often the instrument rather than sampling. Sampling variance is what standard resampling captures. What it misses is how much of a true change the degraded instrument actually detects (its efficiency) and how many switches it invents on its own (its background, in physics terms). These two quantities can be measured by injection.

A simple model makes this concrete. Let $s$ be the true number of switch points a system preserves, $\varepsilon$ the probability that the instrument transcribes a preserved switch as a switch, and $b$ the expected number of spurious switches the instrument produces. The observed switch count is then approximately

$$n_{\text{obs}} \sim \text{Poisson}(\varepsilon\, s + b).$$

Here $\varepsilon$ and $b$ are not constants. They depend on how degraded the audio is, which is exactly what a voice anonymizer changes. If Whisper's tendency to collapse into one language worsens on anonymized audio, $\varepsilon$ falls. A measured 65% drop in switch points then mixes system behaviour with instrument behaviour.

The minimum detectable change in preservation rate, at significance $z$, has the following form:

$$\Delta_{\min} \approx z\,\sqrt{\frac{D}{\varepsilon N}},$$

where $N$ is the number of reference switch points and $D \ge 1$ is the variance inflation from switches clustering within speakers. The $1/\sqrt{N}$ comes from counting statistics. The $\varepsilon$ in the denominator says a less efficient instrument wastes events. Neither $\varepsilon$ nor $D$ is reported for any code-switching benchmark in the literature we reviewed. The protocol makes both explicit, measurable quantities.

Relation to Prior Work

Several papers motivate this proposal by exposing the gap.

First Steps Towards Voice Anonymization for Code-Switching Speech evaluates three anonymizers on the SEAME (Mandarin–English) and MIAMI (Spanish–English) corpora:

  • GANmulti degrades MER by only 1–6% but cuts switch points by 65% and 64%.
  • B2 and SSL degrade MER by 20–71%.

The authors themselves flag that the instrument is compromised, but they do not quantify what that does to their numbers. We use their setting as the primary testbed.

Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech introduces ArzEn, a 12-hour, 38-speaker corpus. It reports a 30.6% word error rate (WER) and a 4.8% relative gain on intra-sentential switches without uncertainty estimates. That is a candidate claim to audit.

Minimal Pair-Based Evaluation of Code-Switching builds the ACS benchmark, which pairs natural switched sentences with one-word-altered versions across 11 language pairs and up to 1,000 pairs each. It reports Llama-3 405B at 74.8% on German–English against a human ceiling of about 80.6%, sub-10B models near 50%, and differences between closed-class and open-class word mutations. At $p=0.5$ and 1,000 pairs, the binomial standard error is about 1.6 points. So "near chance" cannot be distinguished from roughly 53% at 95% confidence, and category-level subsets are wider still. The paper reports no such bounds.

Other papers constrain the design. Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition shows that a better metric changes how well scores track humans: WER correlates 0.55 with human post-editing effort, while CER on text transliterated into Arabic and normalized reaches 0.80. The protocol must therefore be metric-agnostic and applicable on top of the best metric, not in place of it. That paper also warns that transliteration and translation tools add their own errors. In our model these become part of the instrument's response, to be calibrated rather than ignored.

The physics papers supply the method:

None of these physics papers concern language. The contribution here is the instrument model and injection design for code-switching, not the statistics themselves.

Research Program

Instrument calibration by injection

Take utterances with gold transcripts from SEAME, MIAMI and ArzEn. Create a ladder of degraded versions whose spoken content is unchanged:

  • content-preserving vocoder resynthesis;
  • the parameter-free LPC transform used by B2 at graded strengths;
  • additive noise at controlled levels.

Because the truth is still the gold transcript, transcribing each rung with Whisper gives $\varepsilon$ and $b$ as functions of the instrument's own baseline MER.

A second injection type changes content in a known way. We synthesize gold transcripts in which a known fraction of switches has been removed, by translating the switched span into the matrix language, using the multilingual synthesis stack. This tests whether the instrument registers a known 30% or 60% switch loss as 30% or 60%.

The same idea applies to the language-model scorer on ACS. Natural sentences that bilingual annotators rejected act as background. The human agreement ceiling bounds the attainable accuracy, so measured accuracy can be rescaled to a band between chance and that ceiling.

Counting statistics with speaker structure

Switch points are treated as counted events grouped by speaker and conversation. The variance inflation $D$ is estimated empirically from between-speaker dispersion, and switch counts are allowed to be overdispersed relative to Poisson. Every reported quantity carries two things: an interval, and the benchmark's $\Delta_{\min}$ computed before the comparison is made.

Trial-factor accounting

For ACS-style analyses spanning 11 pairs, two word classes and several model sizes, the protocol requires declaring the search space. Significance is reported after correcting for the number of looks, following the post-trial logic of the anisotropy search.

Audit

We re-run the reported comparisons from the voice-anonymization, ArzEn and ACS papers under the protocol. Each claim is classified as above, near, or below the sensitivity floor.

Baselines

Naive reference. The as-reported point estimate with no uncertainty, which is the current norm in the cited papers. The reference figures are:

  • for anonymization: GANmulti's 1–6% MER degradation and 64–65% switch-point reduction;
  • for ArzEn: 30.6% WER and 4.8% relative gain on intra-sentential switches;
  • for ACS: 74.8% versus a human ceiling of about 80.6% on German–English.

Strongest statistical competitor. A paired bootstrap over utterances combined with a paired permutation test. This is what a careful NLP reviewer would demand, and it already captures sampling noise. It is run on exactly the same utterances, metric and system outputs as our protocol, with 10,000 resamples and no extra tuning.

Speaker-clustered bootstrap. A stronger variant that resamples speakers rather than utterances. Comparing against both bootstraps shows how much of our gain comes from clustering alone, since clustering is the ingredient a reviewer would most easily add without any physics.

Strongest instrument fix. Replacing WER with transliterated-and-normalized CER, the best metric in the Hamed et al. benchmark with a 0.80 correlation with human judgment on the Human Acceptability Corpus. This corpus is Arabic–English, so the comparison is cleanest on ArzEn. On SEAME and MIAMI we use the analogous script-normalized error rate, for which no reported result was found on Lacuna.

Ablations. The key ablation removes injection calibration (fixing $\varepsilon=1$, $b=0$) while keeping clustered counting statistics. A second ablation keeps calibration but drops clustering.

What counts as beating the baselines. On the injection ladder, where the true effect is known, the protocol's intervals should reach their nominal coverage when the clustered bootstrap's do not. In replication tests, claims the protocol labels significant should replicate at a materially higher rate than claims the bootstrap labels significant.

Why not just…?

Why not just bootstrap, clustered by speaker? On the question of sampling variance, this is a genuine threat. If instrument efficiency turns out to be near one and stable across degradation levels, the clustered bootstrap will match the protocol, and we will say so. What the bootstrap cannot do is detect bias. Resampling a biased instrument yields a tight interval around a wrong value. The injection ladder settles this directly. If a known 60% switch removal is measured as roughly 60% on anonymization-style audio, calibration adds nothing. If it is measured as 75% because Whisper drops switches on degraded audio anyway, the bootstrap's coverage of the true effect collapses while the calibrated estimate's does not.

Why not just use a better metric such as transliterated CER? Better metrics raise agreement with humans (0.55 to 0.80 in the Hamed et al. benchmark). However, they still count switches through the same recognizer, and they introduce transliteration errors that the same paper flags. A better metric lowers background and raises efficiency, which shrinks $\Delta_{\min}$. It does not remove the need to know $\Delta_{\min}$. We therefore apply the protocol on top of the best metric and report how much it shrinks the floor.

Why not just use a stronger recognizer or human transcription as the instrument? Human transcription of anonymized audio is the ideal reference. We use it on a small calibration subset to validate the injection estimates. At full benchmark scale it is too costly, and it is not free of disagreement either: ACS human agreement is only 76–87%. A stronger recognizer simply changes $\varepsilon$ and $b$, which the protocol then measures.

Why not just collect more data? The $1/\sqrt{N}$ scaling says how much more data is needed. For example, halving the ACS interval requires four times the pairs. Natural code-switched speech from a given community is limited, and the Cline acceptability study finds that synthetic code-mixing differs from natural social-media text. Reporting $\Delta_{\min}$ is what tells a community how much collection would change a conclusion.

Decisive Checks

The first check is coverage on known truth. Across the injection ladder on SEAME, MIAMI and ArzEn, we compare the empirical coverage of 95% intervals from the naive estimate, both bootstraps and the full protocol. The prediction is that the bootstraps under-cover as the instrument's baseline MER approaches the roughly 39% regime and the protocol does not. If this fails, the instrument model is wrong.

The second check is replication by speaker split. We split each corpus into disjoint speaker halves and test every comparison independently in each half. We then measure how often a claim significant in one half is significant, with the same sign, in the other. The protocol should label fewer claims significant, and a larger fraction of those should replicate.

The third check is the audit itself. We report whether GANmulti's switch-point reduction remains after correcting for Whisper's efficiency on resynthesized speech, and how much of it remains. We report whether ArzEn's intra-sentential gain clears its speaker-clustered $\Delta_{\min}$. For ACS, we report which sub-10B models and which closed-class versus open-class differences survive trial-factor correction.

A clean negative result is also informative. If every audited claim survives, the field's current reporting is adequate for these benchmarks, and the protocol reduces to a sensitivity table that future papers can cite before running comparisons.