When people build speech corpora of code-switching (bilingual speakers alternating languages mid-conversation), they now harvest thousands of hours from broadcasts, podcasts and video, as in KSC2. No one listens to all of it. Instead a speech recognizer and a language identifier label the audio, and the corpus builder counts how often each kind of switch appears: which language pair, which direction, a single borrowed noun or a whole switched clause. Those counts decide what gets annotated and how the corpus is split. They are also the evidence behind linguistic claims about where switching happens. The counts are biased, though. The detector is worst exactly where it matters: on the DECM German-English benchmark, Whisper's word error rate rises from 15.02% on German stretches to 34.68% on embedded English words, and MMS rises from 24.79% to 72.78% (DECM). A naive count therefore undercounts real switches, invents false ones, and has no error bar. For the rarest switch types it cannot tell "absent" from "missed."
Experimental particle physics has faced the same problem for decades. It searches for rare decay modes with detectors that miss some real events and accept some fake ones. The field's answer is a small set of habits. Fit the signal yield against a variable that separates real from fake. Measure detection efficiency on simulated signal and check it against a well-understood control channel. Quote rare rates as ratios to a common sibling channel so that shared systematic errors cancel. Estimate fakes from data collected where no signal can exist. Unfold the true distributions of other variables with event weights. Decide on discovery or an upper limit only after the analysis is frozen. This note proposes carrying that toolkit into code-switching corpus analysis. The goal is a bias-corrected "switch census": rates of each switch type with calibrated uncertainty, and a principled rule for declaring that a rare switch type is present in harvested audio.
Core Claim
An automated census of code-switching in unlabeled audio is a counting experiment with an imperfect detector. If it is analyzed the way rare-decay searches are analyzed, the rate estimates for rare switch types should be close to unbiased, and their confidence intervals should achieve nominal coverage. This should hold using only about one hour of hand-labeled calibration audio, and in the regime where plain confusion-matrix correction with the same labels breaks down: switch types that appear fewer than a handful of times in the labeled sample. The new element is not a better recognizer. It is a measurement protocol that sits on top of any recognizer and turns its biased output into a defensible estimate. The protocol can be scored exactly on corpora that already have gold transcripts.
What the physics toolkit actually contributes
Four specific techniques transfer. They are not loose analogies.
The first is ratio-to-reference measurement. BABAR reports the three-body to two-body ratio for $\Lambda_c^+\bar p$ final states as $15.4 \pm 1.8 \pm 0.3$, and the systematic error is far smaller than on either absolute rate because the poorly known $\Lambda_c^+$ branching fraction cancels (BABAR, B → Λc p̄ (π); BABAR, B̄0 → Λc p̄ π0). CLEO's comparison of sibling charmonium channels works the same way (CLEO ψ(2S) → baryon pairs). The speech version measures a rare switch type relative to a common one in the same language pair, speaker pool and recording domain. The recognizer's domain-level failure then largely cancels.
The second is efficiency from injected simulated signal. Physicists estimate how often a real decay would be reconstructed by passing simulated decays through the detector model. The speech version injects synthetic switches of a given type into real audio from the target domain and measures how often the pipeline recovers them. Generators such as UniCoM (word-level substitution with voice conversion to keep the speaker consistent) and the phrase-level constrained generator of Umar et al. serve as the simulator. Their known artifacts, including missing spontaneous prosody and splice boundaries, are the "simulation mismodeling" that the ratio measurement is meant to cancel.
The third is background estimation from signal-free data. BABAR models continuum background with data taken 40 MeV below the resonance, where the signal cannot occur (BABAR, B̄0 → Λc p̄ π0). The speech version uses audio from the same channel or speakers that is known to be monolingual. Every "switch" detected there is fake, so this audio supplies the fake-switch template directly.
The fourth is yield fitting with event weights. Each candidate switch gets a discriminating variable, for example the language-identification posterior margin at the switch point combined with recognizer confidence. A maximum-likelihood fit with a signal template (from injection) and a background template (from monolingual control audio) extracts the true yield. The sPlot method, used by BABAR to study resonant structure without modeling the background (BABAR, B → Λc p̄ (π)), then gives each candidate a weight. Those weights recover the true distribution of a second variable, such as distance to the nearest prosodic boundary or the part-of-speech class of the switched span. This step only works if the second variable is independent of the discriminating variable within signal and within background. That condition is testable, and the program tests it.
Discovery decisions follow the Belle and BESIII style: a significance for the presence of a switch type, or a 90% upper limit when it is not found (Belle Ω(2012); BESIII W-annihilation). All selection and template choices are frozen on development corpora before the target channel counts are examined.
What counts as a "switch type"
Each detected switch point is labeled by language pair and direction, by whether the switched span is one word or several, and by a syntactic class of the switched material. Single-word noun insertions are separated from switched verbs, pronouns or conjunctions, following the syntactic depth idea behind SyMCoM (Goel et al.). The word-count distinction matters because Pattichis et al. show that single-word borrowings can inflate apparent within-intonation-unit switch rates by up to 300%. Their finding has so far been established only on 4.8 hand-transcribed hours. Aggregate indices such as the M-Index and I-Index (Metrics for Quantifying Code Switching Structure) are all computed from gold transcripts. This proposal estimates the same quantities from audio when no gold exists and attaches honest error bars to them.
Research Program
The work has three parts.
The first part is the measurement pipeline. It is recognizer-agnostic and runs on fine-tuned Whisper as the main detector, since Yang et al. show that even one hour of in-domain code-switched fine-tuning beats zero-shot Whisper-Large. It consists of the injection engine, the monolingual control sample, the template fit, ratio normalization to a reference switch type, sPlot unfolding, and significance or upper-limit computation.
The second part is closed-book validation. The pipeline runs on corpora whose gold language-tagged transcripts are hidden from it, except for a one-hour labeled calibration slice. Its estimates are then compared with the gold counts.
The third part is two uses of the census. One is steering annotation: harvesting audio that meets the minimum code-switched quotas per language pair that the mixed-integer partitioning of the South African soap opera corpus requires. The other is scaling linguistic questions, such as whether multi-word switches cluster at prosodic boundaries across many language pairs. That second use bears directly on the disagreement between UniCoM, which argues for word-level substitution, and Umar et al., who report 28.1% versus 31.7% mixed error rate for phrase-level versus word-level synthetic data. A census of how often each form occurs in real speech, by language pair, tells generator designers which one to imitate.
Baselines
The naive reference point is direct counting. The fine-tuned recognizer and language identifier are run over the target audio, and every detected switch is counted by type with no correction. No reported result on Lacuna was found for rate-estimation accuracy under this procedure. Its expected failure can be inferred from the embedded-word error rates DECM reports: 34.68% for Whisper and 72.78% for MMS. To beat it, the census must cut mean absolute relative error in estimated per-type rates at least in half on held-out gold corpora.
The strongest simple competitor is confusion-matrix correction. The same one-hour labeled calibration slice is used to estimate, for each switch type, the probability that a true switch is detected and the probability that a detection is false, and the raw counts are inverted accordingly. This is the standard misclassification correction from epidemiology and survey statistics. It gets exactly the same labels, the same recognizer and the same tuning budget as the proposal. For abundant types it should match the proposal. The proposal beats it only if, on switch types with fewer than about five calibration examples, it gives lower bias and near-nominal 90% interval coverage where confusion-matrix correction is undefined or badly miscalibrated.
A second competitor is pure human sampling. Annotators spend the same time as goes into the calibration slice and the templates, using stratified random sampling of harvested audio, and rates are estimated with standard survey estimators. This baseline is unbiased by construction, so it is the reference for whether automation helps at all.
Three ablations isolate the ingredients. Removing ratio normalization and reporting absolute rates tests whether cancellation actually happens. Replacing monolingual-control background templates with background estimated from the labeled slice alone tests the "off-resonance" idea. Replacing injected-signal efficiency with calibration-slice efficiency tests whether synthetic injection adds information or only adds mismodeling.
Why not just…?
Why not just use a better recognizer and count? Better recognizers shrink the bias but do not remove it, and they never produce an interval. On SEAME, fine-tuned Whisper-Small reaches about 14–15% mixed error rate (Yang et al.), yet errors concentrate on the embedded language, which is exactly where switch types are defined. The experiment runs the census on top of two recognizers of different quality. The claim holds only if the corrected estimates agree with each other and with gold while the naive counts disagree.
Why not just correct with a confusion matrix from the labeled slice? This is the real threat. For common switch types it will probably do as well as the proposal, and the note concedes that in advance. Its weakness is structural. A switch type with zero or two examples in one hour of labels has no usable detection-rate estimate. The proposal gets that rate from injected synthetic signal, calibrated through a common sibling type. The decisive comparison is therefore stratified by how many calibration examples each type has. If confusion-matrix correction matches the census on rare types too, the physics machinery is unnecessary and the conclusion will say so.
Why not just hand-annotate a random sample? Random annotation is unbiased but inefficient for rare types. A switch type at 0.1% of utterances needs hundreds of annotated hours before a handful of examples appear. The soap opera corpus, with 94 language combinations across 307 speakers, is exactly the setting where that cost dominates (Jansen van Vüren et al.). The experiment compares annotator hours needed to reach a fixed interval width on rare types under each approach.
Why not skip the census and train better models directly, for example with language-state tracking? CP-LSTM-2 improves recognition around switch boundaries by tracking a separate state for each language, but it answers a different question. A better model improves transcripts. It does not tell a corpus builder whether a rare pair exists in 2,000 unlabeled hours, and it does not tell a linguist what the switching distribution is. The census can use such a model as its detector.
Proposed Experiments
The main test uses the 50-hour South African soap opera corpus (Jansen van Vüren et al.). It has gold language tags, about 40% code-switched utterances and many rare language combinations. Fine-tuned XLS-R reaches roughly 37% development word error rate on it, which makes it a hard detector regime. All gold labels outside a one-hour calibration slice are hidden. Per-combination rates are estimated with the census and with every baseline, and three things are scored: relative error, 90% interval coverage, and false-discovery rate for the claim "type present."
Replication runs on DECM (DECM) using its low-, mid- and high-switching splits. For reference, Whisper's word error rate is 11.30% on mid and 22.10% on high. This tests whether correction stays accurate as switch density changes. Further replication runs on SEAME (SEAME) and TALCS (TALCS) for Mandarin-English.
The sPlot test uses NMSEB via Pattichis et al.. Weights are fitted on the language-identification margin, and the unfolded distribution of switch position relative to intonation-unit boundaries is compared with the gold distribution. The independence assumption is checked first on the gold data. If it fails, that is reported as a limit on sPlot use, not tuned away.
A synthetic-mismodeling stress test deliberately degrades the injection generator, for example by turning off voice conversion in the UniCoM-style pipeline. The prediction is that absolute efficiency estimates shift while ratio-normalized rates barely move. If ratio rates also shift by more than their stated systematic, the cancellation argument fails.
The decisive check is simple. On rare switch types, the census must show both lower bias and better coverage than confusion-matrix correction at equal labeling budget. If it does not, the proposal reduces to "use confusion-matrix correction," which is still a useful and publishable negative result for corpus builders.
References
- KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus
- DECM: Evaluating Bilingual ASR Performance on a Code-switching/mixing Benchmark
- Measurements of B((B)over-bar(0) -> Lambda(+)(c)(p)over-bar and B(B(-) -> Lambda(+)(c)(p)over-bar pi(-)) and studies of Lambda(+)(c)pi(-) resonances RID C-5223-2009
- Observation of the decay<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mover accent="true"><mml:mi>B</mml:mi><mml:mo>¯</mml:mo></mml:mover><mml:mn>0</mml:mn></mml:msup><mml:mo>→</mml:mo><mml:msubsup><mml:mi mathvariant="italic">Λ</mml:mi><mml:mi>c</mml:mi><mml:mo>+</mml:mo></mml:msubsup><mml:mover accent="true"><mml:mi>p</mml:mi><mml:mo>¯</mml:mo></mml:mover><mml:msup><mml:mi>π</mml:mi><mml:mn>0</mml:mn></mml:msup></mml:math>
- Branching-fraction measurements of<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi>ψ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>2</mml:mn><mml:mi>S</mml:mi><mml:mo stretchy="false">)</mml:mo></mml:math>decay to baryon-antibaryon final states
- UniCoM: A Universal Code-Switching Speech Generator
- Improving ASR with Synthetic Intra-Sentential Code-Switched Speech Generated Using Linguistically-Constrained LLMs and Multilingual TTS
- Evidence for the decay <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msubsup><mml:mi mathvariant="normal">Ω</mml:mi><mml:mi>c</mml:mi><mml:mn>0</mml:mn></mml:msubsup><mml:mo stretchy="false">→</mml:mo><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mi mathvariant="normal">Ω</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>2012</mml:mn><mml:msup><mml:mo stretchy="false">)</mml:mo><mml:mo>−</mml:mo></mml:msup><mml:mo stretchy="false">→</mml:mo><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mo stretchy="false">(</mml:mo><mml:mover accent="true"><mml:mi>K</mml:mi><mml:mo stretchy="false">¯</mml:mo></mml:mover><mml:mi mathvariant="normal">Ξ</mml:mi><mml:msup><mml:mo stretchy="false">)</mml:mo><mml:mo>−</mml:mo></mml:msup></mml:math>
- Observation of the <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>W</mml:mi></mml:mrow></mml:math>-annihilation decay <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msubsup><mml:mrow><mml:mi>D</mml:mi></mml:mrow><mml:mrow><mml:mi>s</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msubsup><mml:mo stretchy="false">→</mml:mo><mml:mi>ω</mml:mi><mml:msup><mml:mrow><mml:mi>π</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow></mml:math> and evidence for <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msubsup><mml:mrow><mml:mi>D</mml:mi></mml:mrow><mml:mrow><mml:mi>s</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msubsup><mml:mo stretchy="false">→</mml:mo><mml:mi>ω</mml:mi><mml:msup><mml:mrow><mml:mi>K</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow></mml:math>
- SyMCoM - Syntactic Measure of Code Mixing A Study Of English-Hindi Code-Mixing
- Code-Switching Metrics Using Intonation Units
- Adapting OpenAI's Whisper for Speech Recognition on Code-Switch Mandarin-English SEAME and ASRU2019 Datasets
- Automatic Partitioning of a Code-Switched Speech Corpus Using Mixed-Integer Programming
- Improving Under-Resourced Code-Switched Speech Recognition: Large Pre-trained Models or Architectural Interventions
- Mandarin-English code-switching speech corpus in South-East Asia: SEAME
- TALCS: An open-source Mandarin-English code-switching corpus and a speech recognition baseline