Split Demonstrations for Code-Mixed, Knowledge-Grounded Dialogue

Large language models can learn a new task from a few worked examples placed in the prompt. This is called in-context learning: no weights change, and the examples alone tell the model what to do. For assistant-style dialogue, the task is unusually structured. At each turn the model must track what the user wants, decide whether to share a relevant fact, and reply. For the hundreds of millions of people who mix languages mid-sentence, such as Hindi and English written in Latin script ("Hinglish"), the natural choice is to show the model examples in that same mixed language. Evidence from benchmarks suggests this backfires. Mixed-language examples can make reasoning worse than giving no examples at all. We propose that in dialogue this failure has a recognizable signature. The model tracks the user's goals less accurately, and it falls back to bland, fact-free replies of the "Sure, I can help with that" kind, which the dialogue literature calls the generic response problem.

The fix we test is simple to state. Teach the task in one language and the voice in another. Our "split demonstrations" show the full structured dialogue procedure in clean English. A separate small set of code-mixed exchanges shows only how users and the system talk. The model then answers code-mixed users in kind. We also ask whether the English-taught procedure lives in a compact internal activation that can be transplanted into a code-mixed forward pass, which would save most of the prompt. To make any of this measurable, we build the first knowledge-grounded, code-mixed task-oriented dialogue test set.

Core Claim

Our main claim is that the "code-mixing tax" on in-context dialogue is mostly a failure to pick up the task from the examples, not a failure to understand the language. If the claim is right, three things should be observable. First, prompts whose structured examples are code-mixed should lose goal-tracking accuracy and grounding behavior compared with prompts whose structured examples are English, even when the test user turn is code-mixed in both cases. Second, the loss should appear as more generic replies, specifically fewer decisions to share knowledge and fewer grounded entities. It should not appear only as wrong slot values. Third, split demonstrations should recover most of both losses while keeping the reply in a natural code-mixed register. If code-mixed examples retrieved for their similarity to the test turn perform as well as English ones, the claim is false for dialogue, and we will report that.

Grounding in Prior Work

The output format comes from unified sequence modeling of dialogue. SimpleTOD writes each turn as a single text string: the user turn, then the belief state (the running list of user constraints as slot-value pairs), then database results, then the system action, then the response. UBAR extends this to whole sessions and reports 52.59% joint goal accuracy on MultiWOZ 2.0. Joint goal accuracy is the share of turns where every tracked constraint is exactly right. KETOD adds Wikipedia-grounded chit-chat to the Schema-Guided Dialogue data. It inserts a <chitchat>/<nochitchat> decision token into the string, so its SimpleToDPlus model learns when to ground. We reuse this string format, including the decision token, as the target of in-context learning rather than of fine-tuning. That decision token is what lets us measure blandness as a choice the model makes.

IC-DST shows that in-context learning works for state tracking. It reaches 48.35% joint goal accuracy on MultiWOZ 2.4 with 1% of the training data, using a retriever that picks similar examples. It is English-only and produces no responses. We borrow its retriever and its idea of tracking state changes, and we extend the setting to grounded responses and mixed-language users.

The motivating contradiction comes from two papers. CodeMixBench finds that 1–2 code-mixed examples often hurt code-mixed reasoning. It also reports penalties approaching 15 points on code-mixed MMLU for Marathi-English and Nepali-English. In-context Mixing finds the opposite for transduction tasks. Replacing half the content words of native-language examples with English improves Telugu disfluency correction by 4.8% on mT0-XXL, but the method is weaker on global semantic tasks. Neither paper studies dialogue. Our account reconciles them: English helps when it carries the task procedure, and mixing hurts when it obscures that procedure.

The mechanistic tools constrain how we test the transplant idea. In-Context Learning Creates Task Vectors shows that examples are compressed into a single hidden state at an intermediate layer. Patching that state into an example-free pass keeps 80–90% of few-shot accuracy, versus 10–20% without it. That result covers only single-token outputs, so multi-token dialogue strings are an open question we must test rather than assume. Separating Tongue from Thought shows that multilingual models fix the output language earlier (around layer 12 in Llama 2-7B) than the concept (layers 16–31). This supports the idea that task and language separate inside the model, but only for clean single nouns.

Two code-mixed dialogue papers show the gap we fill. Banerjee et al. built code-mixed DSTC2 in four Indic languages. Their recurrent models reach BLEU-4 of 53.8–62.1 but only 1.0–1.6% per-dialog accuracy, and the data has no external knowledge. BART-PA3 fights blandness in Hinglish chit-chat with personality conditioning, but it has no task tracking and relies on fine-tuning.

Research Program

We first build CM-KETOD, a test set derived from KETOD's test split, in Hindi-English, Tamil-English and Bengali-English in Latin script. Native-speaker annotators will rewrite about 300 dialogues per pair. They rewrite user turns and system responses, including the knowledge-grounded chit-chat turns, into natural code-mixed speech, and morpheme-level mixing such as Tamil suffixes on English stems is allowed. Entity names, slot values and Wikipedia snippets stay as in KETOD, so the gold belief states and grounding decisions carry over unchanged. A second annotator rates naturalness on a sample, and we report agreement. The language pairs differ in how much resource each language has, which lets us test whether the tax grows for lower-resource languages, as CodeMixBench suggests.

The method has two forms. In the prompt form, split demonstrations contain $k$ English KETOD-format examples with full serialized structure, chosen by an IC-DST-style retriever over the English KETOD training set. They are followed by $m$ short code-mixed user/system exchanges that carry no belief states, and then the code-mixed test context. In the activation form, we run the English examples once and read the hidden state at the final delimiter at layer $\ell$. We then add it into a pass that contains only the code-mixed style exchanges and the test context, sweeping $\ell$ across depth. Because outputs here are long strings, we compare patching once at the delimiter against patching at each segment boundary (belief, action, response). This settles whether the single-vector picture survives multi-token output.

The analysis tests whether degradation under code-mixed examples is concentrated in specific parts of the output: the belief state, the grounding decision, or the response. To find out, we swap the language of each segment of the examples independently. We will use open models where internals are visible, a Llama-family 8B and LLaMA3-70B, and run GPT-4o for prompt-only conditions.

Baselines

The naive reference is zero-shot prompting. The model gets an English instruction describing the serialized format and the knowledge snippets, then the code-mixed dialogue. CodeMixBench's finding that few examples can underperform zero examples makes this a real competitor, not a strawman.

The most natural competitor is IC-DST-style retrieval of matched code-mixed examples. Its pool is a machine-code-mixed version of the KETOD training set, produced with GPT-4 Turbo and filtered as in CodeMixBench's pipeline, so the examples resemble the test turns. It is the strongest test of our claim because it gives the model examples in the test language that are also similar in content. Split demonstrations must beat it on joint goal accuracy and on grounding-decision F1 with a confidence interval that excludes zero.

The second competitor is English examples plus an instruction to "reply in the user's language mix." It is identical to split demonstrations except that it has no code-mixed style exchanges. It is the main ablation: it isolates whether the separate voice examples matter beyond the English task examples.

The third is In-context Mixing at 50% applied to all segments of the examples. It is the strongest existing mixed-prompt method.

The fourth is translate-then-operate. The model translates the user turn to English, runs the English in-context system, and translates the reply back into code-mixed text.

The fifth is supervised: SimpleToDPlus retrained on the machine-code-mixed KETOD training set with the multilingual model's backbone. For reference, KETOD reports 28.6% joint goal accuracy and 64.2% factualness for SimpleToDPlus in English, against 23.7% for the task-only model. No result on Lacuna reports any system on code-mixed, knowledge-grounded task-oriented dialogue, so the code-mixed numbers for every baseline will be new.

For fairness, every in-context arm uses the same model, the same total prompt token budget, the same retriever, and the same 200-turn development slice for format tuning. Results are averaged over five example orderings, given the known fragility of example order.

Why not just…?

Why not just translate to English and back? GlobalWoZ reports that translated entity names fail in lookups 60% of the time (GlobalWoZ). However, KETOD entities mostly stay in English inside code-mixed speech, so modern translation by a large model may preserve them. This is therefore a real threat to state tracking. The likelier weakness is the reply. Back-translation tends to produce stilted mixing or English, which we measure with a code-mixing index (the share of tokens in the less frequent language) and with native-speaker naturalness ratings. If this pipeline matches split demonstrations on all metrics, we will say so. The contribution then shifts to the benchmark and the diagnosis.

Why not just give English examples and ask for a code-mixed reply? This baseline is our own method minus the voice examples, and it may already recover task accuracy. If it does, the claim about task pickup still stands. The open question is then only whether the separate code-mixed exchanges improve register without hurting grounding. Our segment-swap analysis and the naturalness ratings answer that directly.

Why not just fine-tune on synthetic code-mixed data? It is included as a baseline. KETOD's annotation shows that grounding needs human-written chit-chat, and machine-mixed training data may teach unnatural mixing. A fine-tuned model also cannot adapt to a new language pair without new data, which in-context methods can.

Proposed Experiments

Metrics are joint goal accuracy, F1 on the grounding decision, KETOD-style factualness judged against the snippet, and blandness. Blandness is measured as distinct-2, grounded-entity coverage, and the rate of replies matching a fixed list of generic phrases, with the list frozen before evaluation. We also report the code-mixing index and human ratings of naturalness and engagingness. Secondary state-tracking results use code-mixed DSTC2 from Banerjee et al.

The decisive check comes from the segment-swap design. If code-mixing the belief-state and action segments of the examples, but not the response segment, produces most of the drop in tracking accuracy and grounding rate, then task pickup is the bottleneck. If instead the drop follows the user-turn segments, then comprehension of the mixed language is the bottleneck, and our claim fails.

The second check is the transplant. A patched English task vector should recover at least the 80% of full-prompt performance that Geva and colleagues found on single tokens, while using far fewer prompt tokens. If patching only works segment by segment, that shows dialogue procedures are not one vector.

The third check is the ordering of languages. If our account is right, the gap between split and matched code-mixed examples should be largest for Tamil-English and smallest for Hindi-English.