See It, Don't Say It: Copy Control for Tutors Listening to Student Speech

When a student talks to a tutoring system instead of typing, the system receives a transcript with no punctuation, full of "um" and false starts, with some words misheard by the speech recognizer and some that are correct but express a misconception. A good human tutor uses all of this. They notice the hesitation, they reuse the student's own words for things the student got right, and they never repeat the misconception back as if it were true. Current neural tutors have no explicit control over which of the student's words they carry into their reply. They either clean the transcript first, which deletes the hesitation signals, or they generate freely and sometimes echo the error or the misrecognized word back to the student. We propose to make this choice explicit and measurable. The tutor should see every student token but be permitted to copy only some of them.

The mechanism comes from the pointer-generator. This is a decoder that, at each step, mixes ordinary vocabulary generation with directly copying a word from the input, weighted by a learned gate. In summarization the goal is to copy more. In tutoring the goal is to copy selectively. We attach a per-token permission score to the copy distribution. The score is low for low-confidence recognized words, filler words, and spans that state a misconception, and high for correct student terminology and for identifiers in the student's code. We test the idea where the pedagogical evaluation gap is sharpest: Socratic tutoring, scored by pedagogical criteria rather than word overlap, under a deliberate mismatch between models trained on clean typed dialogue and deployed on noisy spoken transcripts.

Core Claim

The decisive variable for a tutor conditioned on student speech is the line between what it reads and what it reproduces. Removing bad tokens from the input fails because the tutor then cannot diagnose the misconception. Leaving everything in fails because the tutor echoes it. A copy distribution masked by token-level pedagogical permission should reduce "error echo" with no loss in Socratic quality. Error echo here means tutor turns that restate a misheard word or a student misconception as if it were accepted. The masking should also beat both input filtering and prompt-level tagging on spoken transcripts.

The core formula reweights the copy term of the standard pointer-generator from Get To The Point:

$$P(w) = p_{\text{gen}} P_{\text{vocab}}(w) + (1 - p_{\text{gen}}) \frac{\sum_{i: w_i = w} m_i a_i^t}{\sum_j m_j a_j^t}$$

The symbols are defined as follows:

  • $p_{\text{gen}}$ is the learned generate-versus-copy gate.
  • $P_{\text{vocab}}$ is the ordinary vocabulary distribution.
  • $a_i^t$ is the attention on source token $i$ at step $t$.
  • $m_i \in [0,1]$ is a permission score predicted for source token $i$.

The encoder still attends to and encodes every token, including fillers and errors. Only the copy path is masked.

Positioning Against Prior Work

The motivating evidence comes from two places. On the copying side, Dr. Summarize adds a generator penalty that forces a pointer-generator to copy patient wording into clinical summaries. Topic-aware Pointer-Generator Networks adds topic-level attention to filter noise in spoken nurse–patient dialogue. Both assume the source is trustworthy and should be copied. In tutoring the student's turn is exactly the untrustworthy source, so we invert their objective.

Pointer-Generator Networks for Low-Resource MT: Don't Copy That! constrains the design. It found that pointers on noisy, subword-heavy input copy seemingly random fragments and give marginal or negative BLEU gains. We read this as evidence that an unconditioned copy gate is the problem, and our permission mask targets that failure directly. Hua and Wang supplies the domain-mismatch rationale: copying salient content transfers across domains better than generating style does. That suggests a copy path controlled separately from the generator could survive the shift from typed to spoken input.

On the speech side, Liao et al. train a transcript cleaner that removes disfluencies and restores punctuation, and they note its risk of over-correction. We use their system as a baseline front end, but our claim is that deletion before the tutor is the wrong place to intervene. The ITSPOKE line motivates both the problem and the metric choice. Litman and Silliman found that semantic interpretation matched perfect transcripts 92% of the time despite 31.2% word error rate. Litman et al. 2005 found that word-level recognition errors did not significantly hurt learning. Rotaru and Litman found that system rejections strongly predict student frustration ($\chi^2 = 409.31$) and that uncertain or incorrect answers co-occur with more recognition errors. So the errors that matter are concentrated precisely in the turns where the student is struggling, and asking the student to repeat is costly. A tutor that can reply on the trustworthy part of a turn, without rejecting it, is what those findings call for.

On evaluation, Al-Hossami et al. build the Socratic Debugging Benchmark. They show that BLEU, ROUGE and BERTScore penalize valid Socratic questions, and that chain-of-thought prompting raises recall but produces "spoiler" hints. TutorBench shows that frontier models stay below 56% on expert rubrics. Both evaluate only clean typed input. Jangra et al. frame good hints as staying consistent with the student's vocabulary while steering toward correct concepts. That requirement is close to a verbal statement of our permission mask.

Research Program

The first deliverable is a spoken version of the Socratic Debugging Benchmark. Volunteers will speak the student turns of all 151 dialogues naturally, prompted by the typed turn but encouraged to paraphrase. The recordings will be run through a standard recognizer that keeps word-level confidences. Following the synthetic recipe of Liao et al., we will also produce a larger text-to-speech plus recognizer version for training.

Annotators will label each student token in the spoken test set with one of four tags:

  • correct grounding (the student's valid terminology, or a code identifier);
  • filler or disfluency;
  • recognizer error;
  • misconception span.

The dataset has only 3,495 utterances, so full annotation is feasible. If the ITSPOKE physics corpus can be obtained, it provides genuine spontaneous student speech as a second test.

The second deliverable is the permission predictor and the masked-copy tutor. The predictor is a small token classifier with three inputs: recognizer confidence, a disfluency tagger's output, and a comparison of the student's claim against the problem's code and reference fix. It outputs $m_i$.

We will instantiate the tutor in two ways. The first is Flan-T5, the open model already used on this benchmark, fine-tuned with an explicit pointer head over the student turn and the code; Liu et al.'s copy-augmented model shows such a head costs very few parameters. The second applies a decoding-time copy penalty to an open instruction-tuned LLM. This penalty down-weights continuations that reproduce low-permission spans, so we can ask whether the effect survives without retraining a seq2seq model.

The third deliverable is an evaluation protocol that adds three measures to the benchmark's manual F1:

  • error-echo rate: tutor turns that adopt or affirm a misheard word or misconception, as judged by annotators;
  • grounding rate: use of the student's correct terms;
  • spoiler rate.

A rubric-based LLM judge in the style of TutorBench will be calibrated against human judgments on a held-out slice. BLEU and BERTScore will be reported only to show where they disagree with the pedagogical measures.

Baselines

The naive reference point is a Flan-T5 tutor fine-tuned on the original typed benchmark and given the raw spoken transcript at test time. It shows how much the typed-to-spoken mismatch costs before any intervention.

The strongest existing pipeline has two stages. The transcript is first cleaned with the readability post-processor of Liao et al., which reports a readability-aware WER of 20.26 against a baseline of 33.52 on RT-03. The cleaned text then goes to the strongest available instruction-tuned LLM with the few-shot Socratic prompt from Al-Hossami et al.. This is what most practitioners would build. Its Socratic quality is the bar to clear. The file reports that GPT-4 roughly doubles GPT-3.5's manual F1 zero-shot, but gives no reported absolute GPT-4 F1 on Lacuna for this benchmark. The only absolute reference is human experts at 57.8% F1, so we will rerun this baseline ourselves on both typed and spoken versions.

The prompt-tagging baseline is the same LLM given the raw transcript with inline markup from our permission predictor (uncertain, filler, likely misconception), plus an instruction not to affirm tagged spans. It has the same information as our method but no mechanism.

The forced-copy baseline is the generator-penalty pointer-generator of Dr. Summarize, trained on the same data. It tests whether simply copying more student wording helps.

The ablations isolate the key ingredient. Input filtering deletes low-permission tokens before the encoder, testing see-but-don't-copy against don't-see. An unmasked pointer-generator sets $m_i = 1$. An oracle mask uses gold annotations, giving the ceiling for the predictor.

All variants share the same training data, synthetic speech augmentation, decoding budget and hyperparameter search size. Prompted LLMs get the same number of prompt-tuning iterations on the development split that the trained models get for hyperparameter search. Beating the strongest baseline means two things on the spoken test set: a lower error-echo rate, with confidence intervals bootstrapped over dialogues, and manual F1 and spoiler rate no worse than the baseline's.

Why not just…?

Why not just clean the transcript and prompt a frontier model? This is the main competitor, and it may win on fluency. Cleaning removes fillers, but it cannot tell a misheard word from a correct one, and it does nothing about misconceptions. Its over-correction also removes the hesitation that Rotaru and Litman link to uncertain answers. The comparison against the pipeline, stratified by whether the turn contained a gold misconception or recognizer error, will show whether it echoes errors more on exactly those turns.

Why not just tag the uncertain words in the prompt and tell the LLM not to repeat them? This is a real threat, since the information content is identical. Instruction following is weaker than a hard constraint on the output distribution, and Al-Hossami et al. show that prompting choices like chain-of-thought trade precision for recall in ways that are hard to steer. Still, this is an empirical question. If prompt tagging matches the masked-copy tutor on error echo and F1, we will report that the contribution is the permission predictor and the spoken benchmark, not the architecture.

Why not just delete the risky tokens? If the misconception span is removed, the tutor cannot ask about it, and diagnosing misconceptions is the core of Socratic tutoring. The input-filtering ablation measures this directly. We expect it to lower error echo but also to lower manual F1 on turns containing misconceptions.

Why not just copy more, as in clinical summarization? Forced copying is correct when the source is authoritative and wrong when the source is a struggling student. The forced-copy baseline should raise error echo, and the noisy-input copy failures reported in Don't Copy That! predict it will also copy recognizer garbage.

Proposed Experiments and Decisive Checks

The main experiment trains every system on typed dialogues, with or without synthetic spoken augmentation, and evaluates on both the typed and the human-spoken test sets. The typed-to-spoken drop in manual F1 measures the domain mismatch for each method. The masked-copy tutor should show the smallest drop and the lowest error echo.

The first decisive check stratifies by turn type. Our method's advantage should appear specifically on turns containing recognizer errors and misconceptions, and should be near zero on clean turns. If the advantage is uniform across turn types, it comes from something other than the mask.

The second check replaces predicted permissions with gold ones. A large oracle gap means the bottleneck is permission prediction, not the copy mechanism.

The third check correlates BLEU and BERTScore with the human pedagogical judgments on the spoken set. This extends the overlap-metric failure documented by Al-Hossami et al. to noisy input.

A smaller secondary study replays low-confidence turns from the spoken set. It compares masked-copy replies against replies that ask the student to repeat, using annotator ratings of likely frustration. This gives a first offline look at whether the rejection trap described in the ITSPOKE work can be avoided. Learning gains are out of scope for this offline study.