Sentiment classifiers for code-mixed social media text, meaning posts that switch between languages mid-sentence such as Hindi-English "Hinglish," fail most often on the neutral class. They see an emotionally loaded word and fire a polar label even when the post as a whole expresses no opinion. This note proposes changing what a neutral label means. Instead of treating neutral as a third softmax category, we treat it as a failure to detect sentiment, borrowing the logic gamma-ray astronomers use when they report that a dark-matter signal is absent.
The method splits each post into several partial "views": the whole sentence, only the Hindi fragments, only the English fragments, and a spelling-normalized version. Each view reports how much sentiment evidence it sees and how reliable that evidence is. A polar label is issued only when the combined evidence clears a statistical bar, and views that disagree in direction are treated as noise rather than signal. The idea is borrowed from two places. Multi-telescope Cherenkov arrays reject cosmic-ray background because real gamma-ray showers look consistent from every telescope, while background showers look ragged. Dark-matter searches combine many targets into one likelihood and give each target its own uncertainty term.
Core Claim
Combining per-language views of a code-mixed post in a single likelihood, with a separate reliability term for each view, and deciding "neutral" by a hypothesis test, will raise neutral-class F1 more than tuning a decision threshold on a single strong encoder. The gains should be concentrated on posts that actually switch language. The claim is narrow on purpose. If a single encoder with a well-tuned abstention threshold matches the gains, the multi-view machinery adds nothing and the idea should be dropped.
Why the borrowed pieces fit
The three directions play different roles. Code-mixed sentiment is the application, parameter-space exclusion supplies the statistics, and stereoscopic imaging supplies the design principle of requiring agreement across views.
The application and its open problem. Neutral F1 is the consistent weak point. The TueMix logistic-regression system reports neutral F1 of 0.594 against 0.738 positive and 0.736 negative on SentiMix Hinglish (Bear et al.), and the winning kk2018 system also notes large neutral gaps (Liu et al.). Both attribute the failure to reliance on keyword triggers.
Why per-language views should carry information. On Hinglish Twitter, speakers shift into Hindi to express negativity and swearing, with a polarity switch ratio of 10.27 and over 90% of swear words inside Hindi fragments (Rudra et al.). This motivates splitting by language. Where the language switches, and which fragment carries the evaluation, is evidence in its own right. A model that only ever sees the whole sentence mixes that evidence together.
How existing systems use language identity. Language identity already enters current models, but only as an auxiliary training signal. Huang and Wu jointly predict word-level language tags and sentiment, reaching 0.730 F1 against 0.687 for single-task BERT (Huang & Wu). S et al. couple sentiment classification with extraction of the phrase that carries the sentiment, through co-attention (S et al.). Both still make the final decision with a single argmax over three classes. Neither asks whether the evidence for a polar label is consistent across partial readings of the post.
The statistical method. We reuse the MAGIC joint dwarf-galaxy analysis (Abe et al.). It combines four targets in one binned profile likelihood and treats each target's uncertain dark-matter density (the "J-factor") as a nuisance parameter, so that one poorly modeled target cannot dominate the result. In our setting, the analogue of a target is a view, and the analogue of an uncertain J-factor is how trustworthy that view is. Trust varies with language-identification confidence and spelling noise.
The design principle. Stereoscopic arrays resolve the ambiguity of a single camera by intersecting shower axes from separated telescopes (Aleksić et al.; Klepser et al.). They reject hadronic background largely because hadronic showers are irregular and inconsistent between telescopes (Aleksić et al.). Our analogue of a hadronic shower is a lone trigger word that drives one view while the other views stay flat or point the opposite way.
How this differs from adversarial training. The closest NLP move toward "bounding" the decision is adversarial training with $\epsilon=5$ perturbations in embedding space (Liu et al.). That smooths the boundary but leaves neutral as an ordinary class. We change the decision rule itself.
Research Program
Building the views. A shared multilingual encoder, XLM-RoBERTa or MuRIL, reads $K$ views of each post. View 1 is the full text. View 2 is the text with English tokens masked. View 3 is the text with native-language tokens masked. View 4 is a transliteration-normalized version. Language tags come from SentiMix's word-level annotations where available and from an automatic identifier for the Dravidian datasets.
What each view outputs. Each view $v$ has a small head that outputs a signed sentiment strength $\hat s_v$ (negative for negative sentiment, positive for positive) and a log-variance $\log \sigma_v^2$. The head is trained with a Gaussian likelihood against the targets $-1, 0, +1$, so a view learns to report large variance when it is unsure.
Combining the views. The views are combined through a shared latent sentiment $s$ and per-view reliability nuisances $\theta_v \ge 0$. The nuisances have a weak prior centered on the view's language-identification confidence:
$$\ell(s,\theta) = -\sum_{v=1}^{K} \frac{(\hat s_v - s)^2}{2(\sigma_v^2 + \theta_v)} - \frac12\sum_{v}\log(\sigma_v^2+\theta_v) + \log p(\theta).$$
The test statistic for "no sentiment" is the profile likelihood ratio
$$\lambda = 2\big[\max_{s,\theta}\ell(s,\theta) - \max_{\theta}\ell(0,\theta)\big].$$
Profiling out $\theta$ is what makes disagreeing views self-discounting. If one view shouts and the others contradict it, the fit inflates that view's $\theta_v$, and $\lambda$ stays small.
The decision rule. The post is labeled neutral when $\lambda$ falls below a critical value. That value is chosen once on the dev set to target a fixed false-polar rate on gold-neutral posts. Otherwise the post takes the sign of the fitted $\hat s$.
Training. The model is trained end to end, adding the per-view Gaussian losses to a cross-entropy on the final decision. The three-class argmax head is kept only as an ablation.
Analysis. Beyond overall scores, we will measure three things. First, whether the gains concentrate on posts that switch language versus monolingual posts. Second, whether the views that carry the evidence line up with the human-annotated sentiment spans in the Tamil-English span dataset of S et al.. Third, whether the fitted nuisances $\theta_v$ track language-identification errors, which Bear et al. identify as a failure source.
Baselines
Naive reference. Fine-tuned multilingual BERT on SentiMix Hinglish, reported at 65.4% F1, with the task's best system at 75.0% (Patwa et al.). This anchors the range rather than serving as a serious competitor.
Strongest single-model competitor. XLM-R fine-tuned with adversarial embedding perturbations, which Liu et al. report at 0.7349 test F1, up from 0.7177 for plain XLM-R. Their full ensemble reaches 0.7526 macro F1 (Liu et al.). We compare single models against single models, and our own ensemble against the 0.7526 ensemble. Per-class neutral F1 for this system: no reported result found on Lacuna for kk2018 neutral F1. We will therefore re-run it ourselves under the same splits, encoder size, epochs and hyperparameter-search budget as our method, over five seeds.
Other competitors. The multi-task language-tag BERT of Huang & Wu, reported at 0.730, is re-implemented on the same encoder. It is the closest prior use of language identity. The hybrid encoder-decoder model of Balaga et al. (HLM, 74.54 F1) and Gemma-2 9B (75.49 F1) are cited as reference points for scale. We do not re-run them unless compute allows.
Ablations isolating the key ingredient. These are the most important comparisons. The first is the same XLM-R with a dev-tuned neutral abstention threshold on the softmax, after temperature calibration. The second is our multi-view model with an argmax decision instead of the hypothesis test. The third is our test with the nuisances $\theta_v$ fixed at zero.
What counts as beating the baselines. A neutral-F1 gain over the threshold-tuned single-view ablation that exceeds seed variance, together with macro F1 at or above the re-run adversarial XLM-R. Matching overall F1 while improving neutral F1 at a controlled false-polar rate still counts as a useful result.
Why not just…?
Why not just tune a threshold on a calibrated classifier? This is the real threat. Abstaining to neutral when the top polar probability is low may capture most of the benefit with no new machinery. The threshold-tuned ablation is built specifically to settle this. If it comes within seed noise of our method on neutral F1, and specifically on the language-switching subset, we conclude that the hypothesis-test framing is just a threshold and the multi-view likelihood earns nothing.
Why not just translate to English first? Translation erases the language switch, which is itself the strongest negativity cue (Rudra et al.). Translation also breaks on non-standard romanized spelling (Patwa et al.), and native processing preserves nuance that translation loses (FeedForward at SemEval-2024). We will include a translate-then-classify run on Hinglish for completeness. We expect it to lose most on the posts where our view-level evidence concentrates.
Why not just use a bigger LLM? On Hinglish, Gemma-2 9B (75.49) is only about one point above a 1.5–3B hybrid model (74.54), and on English-Malayalam the smaller model wins (Balaga et al.). Scale does not obviously fix the neutral-class problem. Our method sits on top of any encoder, so if compute allows we will apply the decision rule to a larger backbone as well, not only compete against it.
Why not just add soft labels or label smoothing for the 55% annotator agreement? Smoothing addresses noise in the labels, not trigger words in the input. It is complementary, so we will add label smoothing to both the baseline and our method so that it cannot explain the difference.
Proposed Experiments
Datasets. The primary benchmark is SentiMix Hinglish, which has gold word-level language tags. Generalization is tested on SentiMix Spanglish, where the best system reached 80.6% and roughly 80% of test tweets are positive (Patwa et al.). This makes it a stress test for false-neutral errors. We also use the FIRE 2020 Tamil-English and Malayalam-English sets with automatic language tags, where the best reported weighted F1 is 0.65 for Tamil and 0.68 for Malayalam (Mandalam & Sharma). The Tamil-English span dataset (S et al.) serves as a diagnostic.
Metrics. Macro F1 and weighted F1, each reported to match the cited papers' conventions. Per-class F1, with neutral F1 as the headline metric. The false-polar rate on gold-neutral posts. Calibration of $\lambda$, meaning whether the targeted false-polar rate on dev holds on test.
Decisive checks. The proposal succeeds or fails on three questions. First, does the full model beat the threshold-tuned single-view ablation on neutral F1 on the language-switching subset, while showing no gain on monolingual posts? A gain on both would suggest generic regularization rather than the mechanism we claim. Second, does fixing $\theta_v=0$ remove a measurable part of the gain on the Dravidian sets, where automatic language tags are noisy? Third, does the dev-chosen critical value keep the false-polar rate within a few points of target on the test set? If it does not, the "controlled false-sentiment rate" selling point fails even if F1 improves.
References
- TueMix at SemEval-2020 Task 9: Logistic Regression with Linguistic Feature Set
- Kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
- Understanding Language Preference for Expression of Opinion and Sentiment: What do Hindi-English Speakers do on Twitter?
- MeisterMorxrc at SemEval-2020 Task 9: Fine-Tune Bert and Multitask Learning for Sentiment Analysis of Code-Mixed Tweets
- Span Extraction Aided Improved Code-mixed Sentiment Classification
- Combined search in dwarf spheroidal galaxies for branon dark matter annihilation signatures with the MAGIC telescopes
- Observations of the Blazar 3C 66A with the Magic Telescopes in Stereoscopic Mode
- Mapping the extended TeV source HESS J1857+026 down to Fermi-LAT energies with the MAGIC telescopes
- MAGIC long-term study of the distant TeV blazar PKS 1424+240 in a multiwavelength context
- SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
- Power doesn’t reside in size: A Low Parameter Hybrid Language Model (HLM) for Sentiment Analysis in Code-mixed data
- FeedForward at SemEval-2024 Task 10: Trigger and sentext-height enriched emotion analysis in multi-party conversations
- Sentiment Analysis of Dravidian Code Mixed Data