Bilingual speakers do not mix languages at random. Hindi-English tweeters move into Hindi when they turn negative, Kenyan peer supporters use English for medical facts and Kiswahili or Sheng for comfort, and a switch point often marks the moment a speaker stops narrating and starts judging. This proposal asks whether that division of labor can be used to build better model committees. The idea is simple. Instead of making ensemble members differ by random seed or learning rate, make each member read a different functional view of the same message: the whole text, only the spans in the speaker's native (non-dominant) language with the rest masked, only the dominant-language spans, or only a small window around each switch point. If speakers really put different kinds of meaning in different languages, these members should make different mistakes. Averaging them would then help, where averaging near-identical fine-tuned transformers usually does not.
This matters because ensembling large fine-tuned models often fails for a known reason. When are ensembles really effective? shows that the gain from an ensemble tracks the disagreement-error ratio: how often members disagree with each other, divided by how often they are wrong. High-capacity models that fit their training set exactly tend to agree on their errors, which pushes this ratio below 1 and makes ensembling useless. Code-mixed text is a natural place to test whether diversity grounded in linguistic structure escapes that cliff. It is also where naive combinations have failed badly. In Bridging Linguistic Complexity, a Random Forest stacked on base-model outputs scored 0.99 on dev and 0.1333 on test for Tamil code-mixed sentiment.
Core Claim
Our central claim is falsifiable. On code-mixed classification, a committee of transformers fine-tuned on functional views will show a higher disagreement-error ratio than an equal-size, equal-compute committee built from seeds and hyperparameters. It will also improve F1 and calibration more. The extra disagreement should concentrate on messages that contain a switch, and should vanish on monolingual messages. A second, smaller claim is that combining members with a deliberately low-capacity gate avoids the stacking collapse seen on Tamil. The gate is a multinomial logistic model over roughly ten switching statistics, trained with cross-fitting. We only keep the gate if it beats plain averaging and greedy selection.
Relation to Prior Work
Three kinds of prior work motivate the claim by showing that function is split across languages. Understanding Language Preference for Expression of Opinion and Sentiment reports two relevant findings on 430k tweets. Speakers are 10.27 times more likely to switch into Hindi when moving from positive or neutral to negative than the reverse. In 46.1% of code-switched tweets, the switch separates narrative from evaluation. Discourse-Driven Code-Switching finds on the Bangor Miami corpus that dialogue function predicts switching better than topic. A Bilingual Attention Network finds that 20.1% of Chinese-dominant Weibo posts carry their emotion entirely in English. That paper's model has separate Chinese, English and bilingual attention channels inside one LSTM, merged by concatenation. We instead train separate pretrained-transformer members on each view and combine them as a committee. This lets us measure their error correlation directly, which a single network with internal channels cannot expose.
Two lines of work already inject switching information into a single model. Code-Switching Patterns Can Be an Effective Route appends nine run-length switching features to a hierarchical attention network. TueMix adds language-ratio and lexicon features to logistic regression. Both use switching as an input feature. We use it to decide what each member is allowed to see, and to weight members. Adapting Multilingual Models to Code-Mixed Tasks via Model Merging is the closest adaptation method. It merges continued-pretraining weights into the base model with task arithmetic or TIES, producing one network with no condition on function. We reuse its merged checkpoint as a strong single-model baseline and as an optional member backbone.
On the ensembling side, Ensembling Finetuned Language Models for Text Classification supplies the method we compete against. It applies greedy post-hoc selection over a pool of 1,134 fine-tuned configurations and shows gains in error and especially in negative log-likelihood. It was run on monolingual benchmarks and never asks whether the kind of diversity matters. NITS-Hinglish-SentiMix averages four shallow networks and reaches 0.617 F1 against 0.576 for its best member. That result is evidence that diversity helps on this data, though only with weak members.
Two results constrain the design. The Tamil stacking collapse rules out flexible meta-learners. BA-MoE shows that routing on language boundaries helps speech recognition. However, its routing is acoustic and internal to one encoder, so it tells us little about pragmatic boundaries in text.
Research Program
Every member is XLM-R fine-tuned on the task. For a robustness pass, members are also built on mBERT and on the merged checkpoint from Kodali et al. Members differ only in the view they read. The full view is the raw text. The native-span view keeps tokens tagged as the non-dominant language and replaces all others with a mask token. The dominant-span view does the opposite. The switch-window view keeps tokens within $k$ positions of any switch point, with $k$ tuned on dev from {2, 4}. Masking keeps sentence length and position intact, so a member still sees where the switches occur even when it cannot read the other language.
Language tags come from different sources depending on the dataset. They are gold where available (SemEval-2020 Task 9, MaCmS spans) and come from script for Chinese-English Weibo. Elsewhere, a token-level language identifier trained on SemEval tags supplies them, and we report results with both gold and predicted tags. If a message has no content in a view, that member returns its training-set class prior. The combiner then learns to discount it.
We measure diversity with the disagreement-error ratio
$$\mathrm{DER}=\frac{\mathbb{E}_{i\neq j}\,\Pr_x[h_i(x)\neq h_j(x)]}{\frac{1}{M}\sum_i \Pr_x[h_i(x)\neq y]},$$
where $h_i$ is member $i$'s predicted label, $M$ the number of members, and $y$ the gold label. The ratio is computed overall and separately on switch-bearing and monolingual test messages.
The gate assigns member weights
$$w(x)=\mathrm{softmax}(A\,\phi(x)),$$
where $\phi(x)$ holds about ten switching statistics. These include the native-language fraction, switch count, longest run, whether the final segment is in the native language, and the presence of a swear-lexicon hit. $A$ is an $M\times|\phi|$ matrix fit with an L2 penalty on out-of-fold member predictions from five-fold cross-fitting, so the gate never sees a member's predictions on that member's own training data.
Baselines
The do-nothing reference is a single XLM-R fine-tuned on the full text. We also report the task's M-BERT baseline, which reaches 65.4% F1 on Hinglish and 65.6% on Spanglish according to SemEval-2020 Task 9: Overview. The strongest published system on Hinglish is KK2018's XLM-R with adversarial training at 75.0% F1, and on Spanglish LiangZhao's translation-augmented XLM ensemble at 80.6% F1, both reported in the same overview. We cite these numbers but cannot rerun those systems exactly. Our in-house comparisons are therefore the decisive ones.
The strongest competitor on our specific question is greedy post-hoc selection, following Ensembling Finetuned Language Models, over a pool of full-view XLM-R and mBERT models that vary seed, learning rate and LoRA rank. Unlike the rest, this baseline gets diversity for free with no linguistic assumptions. It will receive the same total fine-tuning GPU budget as the functional pool and the same dev set for selection. It will be evaluated at the same committee sizes of 3, 5 and 10.
A third baseline is the merged model from Kodali et al., the strongest single network for code-mixed adaptation. A fourth is a single XLM-R with the nine Bansal-style switching features concatenated before the classifier, which tests whether switching information helps just as much inside one model.
The key ablation replaces each functional view with a random-span view that masks the same number of tokens chosen at random. This isolates whether gains come from linguistic function or from input-dropout diversity alone. A second ablation swaps the gate for uniform averaging and for greedy selection over the functional members.
Counting as a win requires three things at matched member count and compute. The functional committee must beat greedy selection over the seed/hyperparameter pool in macro-F1 by more than the seed-to-seed standard deviation across five repeats. It must beat the random-span committee by the same margin. And it must show a higher disagreement-error ratio, with the gap concentrated on switch-bearing messages.
Why not just…?
Why not just run greedy selection over many ordinary fine-tuned models? This is the real threat. On monolingual benchmarks, Ensembling Finetuned Language Models shows it reliably beats the single best model, and it may do so here too. The proposal settles the question head-on. We place functional-view members and ordinary members in one pool and let greedy selection choose. If the selector almost never picks view members, or picks them no more than random-span members, the core claim is wrong and we will report it as such.
Why not just feed switching features into one transformer? Bansal et al. got a 17.7% relative macro-F1 gain on hate speech this way, although the file gives no absolute number. A single network, however, can learn to ignore a small appended feature vector, and it offers no separate error profiles to average. The feature-concatenation baseline and the per-slice disagreement analysis test whether separation adds anything beyond the information itself.
Why not just use a merged model or a large prompted model? Kodali et al. report merging gains of 2–5 F1 over plain fine-tuning. They also report that a merged 1B model beats prompted Llama-3.3-70B, so prompting is a weak competitor. Merging is complementary rather than rival. We therefore also build view members on the merged checkpoint, which tests whether functional diversity survives a stronger, better-adapted backbone.
Why not just stack a flexible meta-model? The Tamil collapse from 0.99 dev to 0.1333 test accuracy is the answer. We run a cross-fitted gradient-boosted stacker as a negative control on the DravidianLangTech Tamil data to check that our low-capacity gate does not reproduce that failure.
Proposed Experiments
The main benchmarks are SemEval-2020 Task 9 Hinglish and Spanglish sentiment, and the Weibo Chinese-English emotion data, where the Bilingual Attention Network reports 0.672 average F1. We add the Bansal humour, sarcasm and hate-speech sets, and MaCmS Magahi sentence sentiment, where XLM-R reports 0.75 F1. The DravidianLangTech Tamil set serves as the stacking stress test. Metrics are macro- and weighted-F1, negative log-likelihood and expected calibration error, and the disagreement-error ratio overall and by slice.
We make several decisive checks. First, whether the functional committee's disagreement-error ratio exceeds the seed committee's, and does so specifically on switch-bearing messages. If it does not, the mechanism is absent. Second, on SemEval Hinglish, the gate should up-weight the native-span member on messages whose final segment is in Hindi and negative, as the 10.27 polarity-switch ratio would predict. The functional story should be visible in the learned weights, not only in the scores. Third, gold versus predicted language tags, to show the method survives realistic tagging noise. Finally, Hinglish has only 55% annotator agreement, so we will check that ensemble gains hold on the subset where multiple predictions of a strong model agree with the gold label. This guards against the ensemble merely fitting label noise more smoothly.
References
- When are ensembles really effective?
- Bridging Linguistic Complexity: Sentiment Analysis of Tamil Code-Mixed Text Using Meta-Model
- Understanding Language Preference for Expression of Opinion and Sentiment: What do Hindi-English Speakers do on Twitter?
- Discourse-Driven Code-Switching: Analyzing the Role of Content and Communicative Function in Spanish-English Bilingual Speech
- A Bilingual Attention Network for Code-switched Emotion Prediction
- Code-Switching Patterns Can Be an Effective Route to Improve Performance of Downstream NLP Applications: A Case Study of Humour, Sarcasm and Hate Speech Detection
- TueMix at SemEval-2020 Task 9: Logistic Regression with Linguistic Feature Set
- Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
- Ensembling Finetuned Language Models for Text Classification
- NITS-Hinglish-SentiMix at SemEval-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text Using an Ensemble Model
- BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition
- SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets