Tutors That Diagnose the Math, Not the Spelling: Language-Invariant Error Recognition for Code-Mixed Socratic Tutoring

A Socratic tutor is a system that guides a student with questions instead of handing over the answer. Its most important step comes before any question is asked. It has to decide whether the student's last move was right, a small slip, or a conceptual mistake. Current language-model tutors are built and tested only on clean English or Chinese. Yet many learners reason aloud in code-mixed text, meaning sentences that switch languages mid-stream, such as Hinglish written in Latin letters with no fixed spelling ("5 me se 2 minus kiya to 3 bacha, sahi hai na?"). We expect such tutors to confuse the language of a student's answer with its correctness. A correct step written in noisy romanized Hindi may be flagged as wrong, and a real arithmetic error may slip past because the model is busy decoding the words. Either failure breaks the thing active learning depends on, which is timely and accurate feedback on the learner's own attempt.

This note proposes to measure that confusion directly and then remove it. We will build matched sets of student turns that carry identical mathematical content but differ in language rendering: English, native-script Hindi, romanized Hindi, and Hinglish with varied spellings. For each tutor we then ask whether its diagnosis stays the same across the set. We will also train a tutor whose hidden diagnosis is pushed to agree across renderings, while its visible reply stays in the student's own language mix. It does this without translating every turn into English.

Core Claim

Our claim is that the gap between languages in tutoring is mostly a gap in diagnosis, not in fluency. We further claim that a consistency objective applied only to the diagnosis decision closes more of it than simply adding code-mixed dialogues to training. We define the diagnosis decision as a short label the tutor emits before replying, from a fixed set: correct step, arithmetic slip, conceptual error, or unclear. Concretely, let $x$ be a student turn written in English and $v(x)$ a code-mixed or transliterated rendering of the same turn. Let $p_\theta(d \mid \cdot)$ be the tutor's probability over diagnosis labels $d$, and let $\mathcal{L}_{\text{dlg}}$ be the ordinary next-token loss on the tutor's reply written in the register of $v(x)$. Training minimizes

$$ \mathcal{L} = \mathcal{L}_{\text{dlg}}\big(v(x)\big) + \lambda \, \mathrm{KL}\big(\,\mathrm{sg}[p_\theta(d \mid x)] \;\|\; p_\theta(d \mid v(x))\,\big), $$

where $\mathrm{sg}$ stops gradients through the English side, so English acts as the teacher, and $\lambda$ weights the consistency term.

The technique is borrowed from xDistill in XCOT, which reduces the KL divergence between low-resource and English chain-of-thought output distributions. We differ in two ways. We distill only the few-token diagnosis rather than a full solution. And the downstream behavior is a guiding question rather than a delivered answer, so we never train the tutor to produce solutions in the target language.

Why This Gap, and What Prior Work Leaves Open

The motivating evidence comes from three places.

First, SocraticLM builds a Dean-Teacher-Student multi-agent pipeline that generates 35K Socratic math dialogues and fine-tunes a 6B model. It reports an Incorrect Answer Recognition Accuracy (IARA) of 0.83 against GPT-4's 0.76, and a pedagogical quality score of 0.62 against GPT-4's 0.50. This shows that error recognition is measurable and separates tutors. All of that work is monolingual.

Second, the code-mixing literature explains why diagnosis should break on such input. Subword tokenizers cut romanized words into arbitrary pieces (Why Transformers Struggle with Code Mixed Social Media Text). Bag-of-words models come close to fine-tuned multilingual BERT on code-mixed sentiment for exactly this reason (Reed at SemEval-2020 Task 9). And models that do well on word-level tagging still degrade on semantic tasks like inference and question answering (GLUECoS; LinCE). Judging whether "3 bacha" is a correct subtraction step is a semantic task of exactly that kind.

Third, prompting larger models does not rescue code-mixed input. Adapting Multilingual Models to Code-Mixed Tasks via Model Merging finds Llama-3.3-70B with in-context examples below small adapted models on code-mixed classification.

Other work constrains the design. Cross-lingual pivot methods get large gains by forcing reasoning through English. XLT is a multi-step prompt that translates, reasons, and translates back. PLUG trains the model to write an English draft before its target-language response. PLUG itself notes the latency and context cost of intermediate generation, which multiplies over the five to ten turns of a tutoring dialogue. XLT is exposed to translation errors on the input. We therefore avoid per-turn textual pivots. English appears only at training time, as the teacher side of the consistency term.

Several methods use code-mixing only as an alignment device, with synthetic dictionary-swapped text and single-turn tasks. COMMIT keeps instruction templates in English and swaps content words using a dictionary. LinguaLIFT trains a language-alignment layer on code-switched text and then tunes on English reasoning data. Neither handles authentic romanized, noisy student language, and neither tutors. We reuse COMMIT's dictionary swapping as one cheap way to generate variants, and we test it against natural code-mixing.

Finally, the "fine-tuning paradox" constrains us. Tuning on target-language or translated data can erode reasoning (Teasing LLMs Adapted to Italian), and SocraticLM already reports catastrophic forgetting when trained on dialogue alone. So we track math-solving accuracy throughout and keep SocraticLM's roughly 1:10 ratio of problem-solving to dialogue data.

Research Program

The first deliverable is a matched-rendering test set built on SocraTeach dialogues. We take student turns grounded in GSM8K and MAWPS problems, each already labeled as correct or containing a specific error. Native Hindi-English speakers rewrite each turn in native-script Hindi, romanized Hindi, and natural Hinglish, with two or three spelling variants per romanized turn (the "achha/acha" kind of drift). These rewrites form the test set. Separately, we collect a smaller set of authentic student turns: bilingual students solve the same problems while thinking aloud in whatever mix they naturally use, and annotators label the diagnoses. This guards against evaluating only on text we wrote ourselves.

The main measurement is flip rate: how often a tutor's diagnosis changes across renderings of the same turn. We split flips into two failure types. False alarms are correct steps flagged as wrong. Misses are real errors accepted as correct.

The second deliverable is the tutor itself. We start from an open 6–8B base model, and include ChatGLM3-6B so the setup matches SocraticLM directly. We first reproduce SocraticLM's training. We then add code-mixed variants of the student turns, produced by a generator model and spot-checked by humans, and paired with tutor replies whose language mix follows the student's. Finally, we apply the consistency term above.

To keep existing skills intact, we will also test merging the tuned model back into its base with task arithmetic or TIES-merging. This follows Adapting Multilingual Models to Code-Mixed Tasks via Model Merging, where merging preserved the base model's monolingual strengths. We treat merging as a hedge against forgetting, not as the contribution.

The third deliverable is an analysis of where rendering-dependence lives. We compare diagnosis flip rates before and after training, broken down by how fragmented the student turn's tokenization is (subwords per word). This tells us whether the objective works by overriding tokenizer damage or merely by learning spelling equivalences.

Baselines

The naive reference point is SocraticLM-style training on the original English and Chinese dialogues, evaluated unchanged on the code-mixed renderings. Its gap from its own English performance shows how large the problem is. On the English SocraTeach setting the reported bar is IARA 0.83 and pedagogical quality 0.62 (SocraticLM). For code-mixed or transliterated tutoring, no reported result found on Lacuna.

The prompting baseline is a large open multilingual model, Llama-3.3-70B, given a Socratic system prompt written in Hinglish plus the same few-shot dialogues our model sees. It is the strongest option that needs no training, and GPT-4's reported IARA of 0.76 in English is its likely ceiling. It receives the same prompt-tuning budget we spend on our own system prompt.

The strongest pivot competitor is translate-then-interact. Each student turn is machine-translated or pivoted into English XLT-style, diagnosed and answered by the English-trained tutor, and the reply is translated back. For context, the best reported pivot gain in the file is XLT raising gpt-3.5-turbo's zero-shot MGSM average from 23.3 to 70.0 (XLT). That is a single-turn solving result, not a tutoring one. We run it with the same underlying tutor weights, so the only difference is the pivot. We report generated tokens and wall-clock time per turn next to accuracy.

The key ablation is augmentation only. It uses identical code-mixed dialogue data, identical steps, and identical ratio of problem-solving to dialogue data, but sets $\lambda = 0$. Any advantage of our model over this ablation is attributable to the consistency term and nothing else. A second ablation replaces natural code-mixed variants with COMMIT-style dictionary swaps (COMMIT), to test whether cheap synthetic mixing suffices.

We count the proposal as beating the field if three things hold together. It must beat augmentation-only on IARA on the authentic student set, with a lower flip rate, by a margin that holds across three seeds. It must match or exceed translate-then-interact on diagnosis while using clearly fewer tokens per turn. And it must lose no more than a small, pre-registered amount of GSM8K and MGSM accuracy relative to the base model.

Why not just…?

Why not just translate the student's turn to English and use an English tutor? This is the most serious competitor, and we treat it as one. Its weaknesses are documented but indirect. COMMIT reports that tuning on machine-translated data underperforms dictionary code-mixing (COMMIT). XLT acknowledges translation errors at the input step. PLUG notes the latency of intermediate generation. None of these measured romanized student math turns. If the translate-then-interact baseline matches our model on false alarms and misses, the diagnosis claim fails. What would remain is a cost argument (tokens per turn) and a register argument: whether the back-translated reply matches the student's language mix, which we measure with a word-level language-identification tagger of the kind GLUECoS evaluates. We would report that result plainly.

Why not just prompt a frontier model to tutor in Hinglish? SocraticLM shows that strong general models default to giving answers and trail a dedicated tutor on error recognition even in English (IARA 0.76 against 0.83). The model-merging study shows a 70B model trailing small adapted models on code-mixed input. Our prompting baseline tests whether both weaknesses compound. We report its answer-leak rate (replies that reveal the final answer) and its flip rate separately, so it is clear which weakness dominates.

Why not just fine-tune on code-mixed Socratic dialogues, with no special objective? This is exactly the augmentation-only ablation, and it may well capture most of the gain. Our bet is that dialogue loss alone rewards fluent replies in the student's register without directly penalizing a wrong diagnosis. The diagnosis is a few tokens inside a long reply, so its signal is diluted. If the consistency term adds nothing on the authentic set, the honest conclusion is that code-mixed data is the whole story, and the benchmark becomes the main contribution.

Decisive Checks

The first check comes before any training. On the matched-rendering set, SocraticLM-style and prompted tutors must show a clear drop in IARA and a clear rise in flip rate on romanized Hinglish compared with English. If they do not, the premise is wrong and we stop.

The second check is the $\lambda$ sweep against augmentation-only on authentic student turns. The consistency term must reduce false alarms and misses, not trade one for the other.

The third check compares diagnosis accuracy per generated token against translate-then-interact across full five-to-ten-turn dialogues. Latency compounds across turns, so this is where the no-pivot design must pay off.

The fourth check is retention. We report GSM8K and MGSM accuracy for Bengali, Telugu, and English before and after training, and with and without merging. This shows whether the tutor still solves the problems it is teaching.

As a transfer probe, we evaluate the Hinglish-trained tutor zero-shot on a small matched set rendered in romanized Bengali-English. This mirrors the English-Hindi to English-Tamil transfer reported for merged models (0.65–0.68 F1 against 0.61–0.63 for standard baselines, Adapting Multilingual Models to Code-Mixed Tasks via Model Merging). It tests whether invariance learned for one language pair carries over to another.