Writing Down Partner Conventions: Retrieved Convention Notes for Conversational Adaptation

When two people repeatedly talk about the same objects, they quickly settle on shared names for them. "The blue thing with the ear" turns into "the ear" by the third mention. They also come to sound more alike in word choice, timing and pitch. This is called entrainment, and in task-oriented games it tracks how well the pair performs. Language models do not pick up these partner-specific conventions. If you give them a growing transcript, they usually do not improve over successive rounds, and when they generate a reply they fall back on their own default vocabulary. We propose a simple change. After each stretch of dialogue, the model writes a short explicit note describing the conventions it has seen: which term this partner uses for which referent, who introduced it, and whether the other person adopted it. Later, the model retrieves the notes relevant to the current moment and places them in its prompt as worked examples. The model only has to reason about conventions once, when it writes the note. After that, the notes are cheap to look up and specific to one partner.

To state the idea in two sentences: partner adaptation fails because raw history is a poor representation of what was agreed, so we have the model condense history into written convention notes and retrieve them, instead of rereading everything. We test this on the Columbia Games Corpus and two related benchmarks. The question is whether retrieved notes produce the improvement across rounds that long context does not, and whether they move entrainment toward human levels without overshooting into mimicry.

Core Claim

Our claim is that the bottleneck in conversational adaptation is how the conversation is represented, not how much of it the model can access. Three earlier findings point there. The first is LVLMs are Bad at Overhearing Human Referential Communication, which gave vision-language models transcripts of human director–matcher games and found that for at least 70% of human pairs the models did not improve as rounds accumulated, even with prior rounds in context. The second is Lexical Entrainment for Conversational Systems. It measured an agent entrainment score of 0.798 for humans against 0.558 to 0.719 for dialogue models, and it reported that limiting models to the last one or two turns sharply reduces recall of entrainable terms, so the relevant evidence is spread across the whole dialogue. The third is High Frequency Word Entrainment in Spoken Dialogue, which found on the Columbia Games Corpus that entrainment correlates with task success (r = 0.341 and 0.376) and with shorter turn-taking latencies (r = −0.757). These are correlations, not proof that entrainment causes success.

Taken together, the evidence is there in the context but the model is not using it. Our hypothesis is that an explicit, structured summary of conventions, created by a reasoning step and recalled by retrieval, will produce a positive slope across rounds where plain context accumulation produces none.

Research Program

The system has three parts.

The first is note writing, which is the chain-of-thought part. Chain-of-thought means the model writes out intermediate reasoning before its answer. At the end of each game segment, the model reads that segment and writes convention notes. Each note records a referent, the expression or expressions used for it, which speaker introduced it, whether the partner took it up, and whether it has since been shortened or replaced. This extends Reflect, Not Reflex, which showed that asking GPT-3 an intermediate common-ground question before responding increased high-quality responses by 30%. Its inferences, however, concern situational facts such as persona, prerequisites and emotion. They are made from scratch each turn and never stored. Our notes concern communicative conventions, and they persist.

The second is retrieval, the retrieval-augmented generation part. Retrieval-augmented generation means fetching stored text and adding it to the prompt. At each new turn we fetch the notes whose referents or expressions overlap with the current utterance. We use cheap lexical matching (BM25) over head nouns, with dense matching as a variant. The notes and the index are built after each segment, away from the moment of response. This addresses the latency objection the research file raises against putting retrieval inside the turn-taking window.

The third is delivery through in-context learning, meaning the model adapts from examples placed in its prompt without any change to its weights. Each retrieved note is paired with the original exchange it came from and inserted as a demonstration. A demonstration shows the model how this particular partner talks about this particular object.

The method changes in one respect because of the "uncanny valley" concern raised by Shi and Sen. Maximizing entrainment is not the aim, because a system that copies every user term would seem uncanny. The note-writing step therefore also records whether the partner kept a term or abandoned it. The target is to match the adoption rate humans show in the same position, not to exceed it.

Baselines

The naive reference point is a model given only the current segment. It shows how much any form of history contributes.

The strongest existing practice is to put the full history in context. The model receives every earlier round verbatim. This is the condition LVLMs are Bad at Overhearing Human Referential Communication tested, where proprietary models reached 45% to 76% single-round referent accuracy against 100% for humans and showed no improvement across rounds for at least 70% of pairs. The baseline is strong because the Columbia Games Corpus sessions are short enough to fit in modern context windows, so the model has every piece of evidence our method uses. We will run it with the same base model, the same instructions, and the same budget for tuning prompts. Our method beats it only if the slope of accuracy against round index is reliably positive while this baseline's slope stays near zero, and if the final-round accuracy is higher.

On MULTIWOZ-ENTR, the reference points for generation are the human agent entrainment score of 0.798 and the model range of 0.558 to 0.719 reported by Shi and Sen. For extracting entrainable terms, the end-to-end extractor reaches 57.8 F1, against 30.7 F1 for the pipeline, and our note writer has to match or beat it to show the notes are accurate. For the Columbia Games Corpus itself, we found no reported language-model result on Lacuna, so we will set the reference by running the naive and full-history baselines ourselves.

Two ablations isolate the key ingredient. In raw-turn retrieval, we retrieve the original past turns with no notes, which tests whether writing the notes adds anything beyond retrieval alone. In per-turn reasoning without memory, the model is prompted Reflect-style to reason about conventions at every turn but stores nothing, which tests whether persistence matters.

Why not just…?

Why not just use the full history in a long context? This is the main alternative, and the research file does not settle it. The evidence against it comes from vision-language models on picture-matching transcripts, and newer text models with stronger long-context use might adapt by themselves. The experiment on round-wise slope settles the question directly. If the full-history baseline matches our slope on the Columbia Games Corpus, we would conclude that the earlier failure was about the image modality or the capability of older models, not about how history is represented, and we would report that.

Why not just retrieve past turns without writing notes? It is plausible that retrieving the earlier turns that mention the same object with BM25 gives most of the benefit. That is why raw-turn retrieval is our main ablation. Our expectation is that notes do better on conventions that changed over time ("the ear" replacing an earlier long description), because raw retrieval returns the older and longer forms just as readily. We will report results separately for referents whose expression changed and for those whose expression stayed fixed, which is where the difference should show if it exists.

Why not just prompt for chain-of-thought at every turn? Per-turn reasoning re-derives conventions repeatedly, costs reasoning tokens inside the response window, and, as Reflect found, depends on model scale: BlenderBot-440M needed fine-tuning to benefit. The per-turn ablation compares quality and latency at matched token budgets.

Why not just fine-tune on entrainment data? Fine-tuning learns entrainment in general, not this particular partner's terms, and a session lasting minutes leaves no room for weight updates. A fine-tuned model would serve as a complement inside our method rather than replace it. We will include a model fine-tuned on MULTIWOZ-ENTR as an extra point of comparison.

Proposed Experiments

We define three evaluations on Columbia Games Corpus transcripts.

The first is cloze prediction of referring expressions. We hide a later mention of a referent and ask the model to predict the speaker's head noun. We score exact and head-noun matches, broken down by how many times the referent has already been mentioned.

The second is next-turn generation for the follower role. We score it with Shi and Sen's entrainment metric, which is based on head-noun identity, measured as absolute distance from the human value at the same point. We also collect blind pairwise human judgments of naturalness and of how similar the reply feels to the partner.

The third tests transfer to referent identification. We use the dog-and-basket games from Wang et al., scoring accuracy for each round and fitting a slope across rounds.

MULTIWOZ-ENTR serves as a check that results hold in a second setting with the reported numbers above. We use the Columbia Games Corpus only for transcripts in the main study. The ToBI intonation labels are a secondary probe: we test whether adding pitch-accent marks to notes, which humans use to signal given versus new information, improves cloze prediction at the first reuse of a term.

Two checks decide whether the claim holds. In the cross-partner swap, we replace a dyad's notes with notes from another dyad playing the same game. If the gains come from partner-specific conventions, performance should fall to the naive baseline or below it. If it does not fall, the notes are carrying generic game knowledge. In the counterfactual edit, we change a term in one note, for example "the ear" to "the handle," and measure how often the model's output follows the edited term. This tells us whether the notes are actually being used or are only correlated with the gains. The proposal fails if the full-history baseline matches the round-wise slope, or if raw-turn retrieval matches the notes on referents whose expression changed.