Noise-Calibrated Smoothing for Robust Fine-Tuning on Code-Mixed Text

Robust training objectives tell a model to do well on its worst-performing slice of the data rather than on average. They are an appealing fix for code-mixed social media text, meaning posts that switch between languages mid-sentence, such as Hindi-English "Hinglish" tweets written in Roman script. Standard fine-tuning of multilingual transformers on such text collapses on minority slices: the Neutral class, heavily switched posts, and unusual transliterations. The catch is that a robust objective is itself an optimizer. It searches for the maximum over estimated slice losses, and code-mixed labels are very noisy: annotators on the SemEval-2020 Hinglish task agreed only 55% of the time (SemEval-2020 Task 9 overview). A max over noisy estimates chases the noise. The adversary piles weight onto whichever slice looks worst by chance or because of label errors. This is the same error-maximization pathology that makes Markowitz portfolios bet on assets whose returns were overestimated.

We propose replacing the hard "worst slice" maximum with a smoothed maximum, a softmax-weighted average over slices, and setting the smoothing temperature from the measured statistical uncertainty of each slice's loss. The temperature is then no longer an optimization convenience tuned by grid search. It becomes a shrinkage estimator for the worst case: large while slice-loss estimates are noisy, and shrinking toward the hard maximum as evidence accumulates. We test this on XLM-RoBERTa fine-tuned on Hinglish, Spanglish, and Tamil-English sentiment. Slices are defined by how densely a post switches languages, crossed with its label.

Core Claim

Let $\hat L_g(w)$ be the running estimate of the average loss on slice $g$ (one of $m$ slices) for model weights $w$, and $\pi_g$ its training share. The robust objective we train on is the entropy-smoothed maximum

$$ \Phi_\mu(w) = \mu \log \sum_{g=1}^{m} \pi_g \exp\!\big(\hat L_g(w)/\mu\big), $$

which is Nesterov's dual smoothing of the max with an entropic regularizer. It equals average risk as $\mu\to\infty$ and hard worst-slice risk as $\mu \to 0$, and it sits within $\mu \log m$ of the hard maximum. The estimation-error side supplies the matching quantity. If each $\hat L_g$ carries noise of standard error $s_g$, the hard maximum over slices is biased upward by roughly $s \sqrt{2\log m}$. This is the standard bound on the maximum of $m$ sub-Gaussian errors. Because the two terms have the same form, we set

$$ \mu_t = \kappa \cdot \bar s_t, \qquad \bar s_t = \Big(\textstyle\sum_g \pi_g s_{g,t}^2\Big)^{1/2}. $$

Here $s_{g,t}$ is the bootstrap standard error of slice $g$'s loss at step $t$, computed from a buffer of recent per-example losses. The single constant $\kappa$ is of order one and is fixed across datasets. The smoothing error is then matched to the noise-induced bias, and $\mu_t$ decays on its own as slice estimates sharpen. This gives a data-driven smoothing schedule for the non-convex stochastic setting, where the smoothing literature lacks principled rules.

The claim is falsifiable in three ways. First, this rule should match or beat a fixed temperature tuned per dataset by grid search. Second, it should beat hard group robustness on both worst-slice F1 and overall macro-F1. Third, when we inject label noise into one slice, hard robustness should shift weight onto it and degrade, while the calibrated version should not.

Where This Sits

Three lines of prior work motivate the proposal. Distributionally Robust Language Modeling trains language models against the worst mixture of latent topics (Topic CVaR). It reports that the sentence-level version failed because the adversary upweighted ungrammatical sentences, and it moved to coarse LDA topics as a workaround. We read that failure as error maximization by the adversary, and we treat it with calibrated smoothing rather than coarsening. Unlike that work, we target short, non-standard, code-mixed posts and transformer fine-tuning, which its authors left to future work.

Stochastic Approximation Approaches to Group DRO supplies the algorithmic template: a mirror-prox game between model and group weights with $O(\log m/\sqrt{n_i})$ rates. Its guarantees assume convex losses, and it treats noise in the adversary's loss estimates only through rates, never as a quantity that should set the adversary's aggressiveness.

Risk-Sensitive Submodular Optimization shows that unsmoothed CVaR optimization stalls. There, standard Frank-Wolfe returned a CVaR of zero at low risk levels, and smoothed gradients and thresholds were required. We reuse that lesson but tie the smoothing amount to estimation noise, not only to optimization stability.

The code-mixing papers constrain the design. kk2018 adds norm-ball embedding perturbations to XLM-R and still reports poor Neutral performance. Its robustness is local and input-level; ours is distributional and slice-level, so the two can be combined. The same paper found that mixing Spanglish into Hinglish training hurt, so we train one language pair at a time. Reed and TueMix show that bag-of-words and feature-based linear models stay competitive because subword tokenizers fracture Romanized words. This warns us that gains from any loss change may be small next to representation problems, so we report the linear baselines too. Advancing Sentiment Analysis in Tamil-English Code-Mixed Texts supplies the Tamil evaluation and an error breakdown: sarcasm and idioms account for 35% of errors, and phonetic errors for 22%.

Research Program

The work has three parts.

The first part defines slices. On SentiMix Hinglish we use the word-level language tags provided with the data to compute each post's switching density: the fraction of adjacent word pairs that change language, plus the fraction of non-English tokens. We bin posts into three switching-density tertiles and cross them with the three sentiment labels, giving $m=9$ slices. Group labels are used only in training. At test time we report per-slice metrics. For Spanglish, and for Tamil where tags are absent, we use an automatic word-level tagger and report tagger agreement on a hand-checked sample. We will not claim results that depend on slices the tagger cannot reproduce.

The second part is the estimator. We keep a per-slice buffer of recent per-example losses. From each buffer we compute $\hat L_g$ and a bootstrap standard error $s_g$, form $\mu_t$, and update slice weights by exponentiated gradient toward the softmax of $\hat L_g/\mu_t$. Model weights take ordinary AdamW steps on the weighted loss. We also study a per-slice variant that shrinks each $\hat L_g$ toward the pooled mean in proportion to $s_g^2$, to test whether a single global temperature loses information.

The third part is analysis. For a fixed model we bound the excess worst-slice risk of the smoothed adversary's weights by the sum of the smoothing gap $\mu\log m$ and the noise term $s\sqrt{2\log m}$, and show that the rule balances the two. We make no convergence claim for the non-convex transformer. We verify empirically that training loss curves are stable and that slice weights do not oscillate.

Baselines

The do-nothing reference is standard cross-entropy fine-tuning of XLM-R. On SentiMix Hinglish, kk2018 reports a test F1 of 0.7177 for plain XLM-R. We rerun it rather than copying the number, so that seeds and preprocessing match. We also report the task's M-BERT baselines, F1 65.4% on Hinglish and 65.6% on Spanglish (SemEval-2020 Task 9 overview), and the bag-of-words network from Reed, which reports 60.0% validation accuracy. These anchor the scale of any improvement.

The strongest existing single-model method on this benchmark is XLM-R with embedding adversarial training. kk2018 reports a test F1 of 0.7349 with $\epsilon=5$, and a 10-fold ensemble reaching macro-F1 0.7526 (the task overview lists 75.0%). On Spanglish, the top system (MT augmentation plus XLMs and a CNN) reports F1 80.6%. On the combined Tamil-English corpora, RemBERT reports accuracy 87.5% and F1 86.4% (Krasitskii et al.). We found no reported worst-slice or Neutral-class F1 for any of these systems on Lacuna, so those comparisons rest on our own reruns.

Beating the strongest method means the following. Relative to single-model adversarial training, our method must improve worst-slice F1 by more than the across-seed standard deviation, while overall macro-F1 is no lower than 0.7349 within seed noise. Adding our objective on top of adversarial training should then improve on adversarial training alone. We do not treat an ensemble's number as a single-model target.

The key robust-training competitors are hard group DRO (the Zhang et al. mirror-prox procedure) and the smoothed objective with a fixed temperature chosen by validation grid search. The comparison with the fixed temperature is the ablation that isolates our contribution, which is the noise calibration rather than the smoothing itself. A second ablation keeps the calibration rule but shuffles the slice assignment. This tests whether switching-density slices matter or whether any reweighting of hard examples helps.

All methods share the same backbone (XLM-R base for development, XLM-R large for final numbers), data splits, five seeds, and a fixed budget of 20 hyperparameter trials selected on development macro-F1. Our method's single constant $\kappa$ counts against that budget.

Why not just…?

Why not just use class-balanced reweighting? Much of the Neutral collapse is class imbalance, and inverse-frequency weighting or a focal loss addresses it cheaply. This is a real threat on the Neutral-F1 metric alone. Class weights are fixed from counts, however, and ignore the switching-density axis. Nothing in them responds to which slices the model is currently failing. Our slices cross label with switching density, so the experiments show whether the high-switching Neutral slice improves beyond what class balancing gives. If class balancing matches us on every slice, we will report that the robust machinery is unnecessary for this data.

Why not just tune the temperature, or use hard group DRO? Hard group DRO is the method our argument says will chase label noise. The Oren et al. sentence-level failure is prior evidence of this, and the noise-injection experiment tests it directly. A grid-tuned fixed temperature is the more serious alternative. It may do as well as our rule on in-distribution splits, since tuning implicitly absorbs the average noise level. Our method has to win on three things instead. The first is transfer: one $\kappa$ must work across all three language pairs, whereas a tuned $\mu$ must be re-searched for each. The second is noise shift: under injected noise, a temperature tuned on clean data should overreact. The third is training dynamics: early in training, estimates are noisy, and a decaying $\mu_t$ should help. If a tuned constant matches us on all three, the contribution reduces to "entropy-smoothed group DRO works on code-mixed text," and we will say so.

Why not just normalize transliterations or use adversarial training? Normalization attacks the tokenization problem rather than slice-level error concentration. Even with normalization, subword segmentation, and back-translation, the Tamil study still attributes 22% of errors to phonetic variants and 15% to script differences (Krasitskii et al.). Adversarial training smooths the model locally around each input, and kk2018 still reports weak Neutral performance with it. Both methods are complementary to ours. We run adversarial training combined with our objective to check additivity, and we apply the same normalization to every method in one controlled condition.

Proposed Experiments

The main experiment trains each method on SentiMix Hinglish, SentiMix Spanglish, and the combined Tamil-English sentiment corpora (SAIL, CMD-Tamil, DravidianCodeMix). We report overall macro-F1, Neutral F1, worst-slice F1, and the entropy of the learned slice weights, with means and standard deviations over five seeds.

The decisive check is controlled label noise. We randomly flip 20% and then 40% of labels in one high-switching slice of the Hinglish training set, leaving the test set clean. Hard group DRO should put increasing weight on the corrupted slice, and its clean-test worst-slice F1 should drop. Our method should hold both its weights and its F1 roughly steady. The optimal fixed temperature found by grid search should also rise with the noise level in the way the rule $\mu = \kappa \bar s$ predicts. If the tuned optimum does not track measured standard error, the core calibration claim is wrong.

The second check trains on 25%, 50%, and 100% of the Hinglish training data. Smaller data means larger $s_g$, so the rule predicts stronger smoothing is needed; we test whether the best fixed temperature grows as the data shrink.

The third check is a qualitative audit of the 200 examples the hard adversary upweights most. We measure what fraction are annotator-ambiguous or mislabeled, compared with the examples our method upweights. This ties the numbers back to the 55% annotator-agreement figure that motivated the work.