Counterfactual Credit for the Copy Switch: Teaching Pointer-Generators When Copying Actually Helps

A pointer-generator is a text decoder with a switch. At every step it either writes a word from its vocabulary or copies a word directly from the input. In principle this suits translation between closely related languages such as Spanish–Catalan or Hindi–Bhojpuri, where names, numbers, and cognates (words that share a root across languages) can often pass through unchanged. In practice the switch has failed there. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! found only marginal gains, no extra benefit for closely related pairs, and slight losses on test subsets with heavy subword overlap. The learned switch copied seemingly arbitrary subword fragments. Its behaviour did not track cognates, numerals, names, or the decoder's attention. The authors' diagnosis is that ordinary token-level training gives the switch no signal about whether a copy improved the translation.

This note proposes giving the switch that signal directly. We use self-critical sequence training, a reinforcement-learning method that rewards a sampled output by how much it beats the model's own greedy output on a sentence-level metric. We add a counterfactual twist. For the few positions where the model's copy decision is most consequential, we re-decode greedily with that one decision flipped. The metric difference between the flipped and unflipped outputs becomes the reward for that decision alone. The same move, "decode again with one information channel switched off and see whether the score drops," extends naturally to images in multimodal translation. There, decoders are known to ignore the picture, or to be harmed by it when it is irrelevant.

Core Claim

Standard self-critical training spreads one scalar reward uniformly over every token. It cannot tell whether a particular copy decision was responsible for a gain. We claim that per-decision counterfactual rewards will make the copy switch learn a policy with three properties. First, it will improve translation quality on related-language low-resource pairs, including the high-overlap subsets where a plain pointer-generator lost. Second, its copies will concentrate on linguistically sensible units: names, numerals, and true cognates. Third, the same estimator applied to a visual gate will make multimodal models use the image when it disambiguates and drop to text-only behaviour when the image is uninformative.

Concretely, we treat the switch value $p_{gen}$ as the probability of a Bernoulli choice $z_t$ (copy or generate) during training, while keeping the soft mixture at inference. The update is

$$\nabla_\theta \mathcal{L} = -\big(r(Y^s)-r(\hat Y)\big)\nabla_\theta \log P_\theta(Y^s) \;-\; \lambda \sum_{k\in K}\big(r(\hat Y)-r(\hat Y^{(k)})\big)\nabla_\theta \log \pi_\theta(\hat z_k)$$

The symbols are defined as follows:

  • $Y^s$ is a sampled translation.
  • $\hat Y$ is the greedy translation.
  • $r$ is a sentence-level metric computed against the reference; we use chrF or BLEU.
  • $K$ is the set of the $k$ greedy positions where the switch was least certain.
  • $\hat Y^{(k)}$ is the greedy translation re-decoded from position $k$ with only the switch decision $\hat z_k$ flipped.
  • $\pi_\theta(\hat z_k)$ is the switch's probability of the decision it actually took.
  • $\lambda$ weights the switch-level credit.

The first term is ordinary self-critical training, as in Self-Critical Sequence Training for Image Captioning. The second term is the new ingredient. It costs $k$ extra greedy decodes per sentence and assigns credit locally. This addresses the missing per-step credit assignment noted for uniform-reward self-critical training.

Research Program

The work builds on three lines of prior research, each in a different role.

The negative result of Don't Copy That! motivates the proposal and defines the test conditions. We reuse its language pairs (es-ca, fr-de, hi-mr, hi-bh) and its data sources (WikiMatrix, Europarl, CVIT-PIB, NLLB). We also reuse its analysis split by shared-subword density, because the high-density subset is where the plain pointer-generator underperformed.

Self-critical training supplies the method. Prior uses have focused on correcting pathological generation, as in PARENTing via Model-Agnostic Reinforcement Learning, and on custom non-differentiable rewards, as in Discriminability Objective for Training Descriptive Captions. None of these isolates a structural decision inside the decoder. That isolation is what we add.

Cross-lingual subword sharing constrains which copies make sense. Subword Mapping and Anchoring across Languages showed that identical strings across languages are sometimes false friends (German die is an article, English die a verb). It also showed that some different strings are true equivalents (and/und), and that fixing the latter matters more. We test one variant in which the copy channel is extended this way. It may emit either the identical subword or its mapped partner from that paper's nearest-neighbour alignment. It is blocked from copying strings that the alignment marks as false friends. This is an arm of the study, not a requirement of the core claim.

The cross-modal setting is the second application. Images and text sit in geometrically separated regions of shared embedding spaces, as shown in Mind the Gap. Translation decoders often ignore visual input (A Visual Attention Grounding Neural Model for Multimodal Machine Translation). Conversely, Multimodal Transformer for Multimodal Machine Translation falls to 36.7 BLEU with random images, below its own 37.8 text-only baseline.

Our extension starts from the Multimodal Transformer. We add a scalar visual gate on its text-to-image attention and apply the counterfactual term to that gate. The counterfactual decode replaces the image with a random one, so the gate is rewarded only when the real image changes the score. We do not claim this closes the geometric gap. The claim is narrower: the decoder learns when to trust a channel that lives across the gap. We will measure whether that changes the gap geometry as a secondary analysis.

Baselines

The naive reference point is a standard Transformer with a joint BPE vocabulary trained by cross-entropy. This system already copies implicitly through shared embeddings.

The strongest existing method on the text side is the off-the-shelf pointer-generator of Don't Copy That!. We reimplement it on the same splits. That paper reports only qualitative "at best marginal" BLEU gains, so no reported numeric result was found on Lacuna for its per-pair BLEU, and we will report our own reproduction.

Two further baselines isolate the key ingredient. The first is the same pointer-generator with plain self-critical training and no counterfactual term. The second is a version with the counterfactual term applied at random positions instead of the least-certain ones. Beating plain self-critical training is the claim that matters.

A fourth baseline is a pointer-generator whose switch is supervised under maximum likelihood with heuristic copy labels. A token is labelled "copy" when an identical or alignment-linked subword appears in the source.

On the multimodal side, the main competitors are as follows:

  • The Multimodal Transformer reports 38.7 BLEU on Multi30K English–German against a 37.8 text-only baseline (Multimodal Transformer).
  • VAG-NMT reports 63.5 BLEU on IKEA English–German against a previous best of 59.9 (VAG-NMT).
  • ZeroMMT reports 65.0 disambiguation accuracy with classifier-free guidance on CoMMuTE English–French against a 50.0 text-only baseline (Towards Zero-Shot Multimodal Machine Translation).

For fairness, every system shares the same tokenizer, data, and maximum-likelihood warm start. All self-critical variants receive the same number of reward evaluations; we give plain self-critical training extra samples to match the cost of the $k$ counterfactual decodes. Each system receives the same hyperparameter search budget.

We count the method as beating a baseline only if it meets three conditions. Its gain must exceed seed variance over three seeds. It must hold on the high-shared-density subset. And the copy-category analysis below must show the switch moving toward names, numerals, and cognates. A BLEU gain alone is not enough.

Why not just…?

Why not just supervise the copy switch with heuristic labels? This is the most serious threat, and the follow-up suggested by Don't Copy That! is close to it. String-match labels, however, inherit exactly the false-friend errors documented in Subword Mapping and Anchoring across Languages. They also say nothing about cases where copying a correct cognate still produces a worse sentence, for example through a wrong inflection. If the supervised-switch baseline matches our method on both BLEU and copy-category precision, the reinforcement-learning machinery is unjustified and we will say so. We expect the gap to appear on hi-bh, where the original paper noted noisy, non-literal references.

Why not just use plain self-critical training on the pointer-generator? Its single reward is shared across all tokens, so it changes the switch only indirectly. The flipped-decision ablation measures exactly what local credit adds. If the two runs are indistinguishable at matched compute, the core claim fails.

Why not drop the pointer and only fix the vocabulary with subword anchoring? Anchoring reports up to 3.0 BLEU on hard English–Russian sentences (Subword Mapping and Anchoring across Languages), and it may capture most of the available gain. We therefore run anchoring alone as a baseline and anchoring plus our trained switch as a separate arm. A pointer is worth keeping only if the combination improves rare names and numerals, which anchoring cannot represent if they are absent from the vocabulary.

Why not just steer image use at inference with classifier-free guidance or by shifting embeddings? Guidance in ZeroMMT trades off disambiguation against fidelity with one global knob. Embedding shifts in Mind the Gap were tested only on classification tasks. Neither decides per sentence whether the image is informative. We test the difference directly by applying guidance to our baseline and comparing it under the random-image condition.

Proposed Experiments

The text experiments come first.

We train every system on the four related-language pairs from Don't Copy That! at two resource scales. We report BLEU and chrF overall and on the high- and low-shared-density subsets.

For copy behaviour, we tag a held-out sample of target tokens as named entity, numeral, cognate, or other. We then measure copy precision per category and the correlation between copy mass and attention entropy. The earlier paper found no such correlation, so a positive correlation here would be a clear change.

To guard against reward gaming, we run human pairwise preference on 300 sentences per pair. We also check that gains under chrF reward hold when scored with BLEU, and the reverse.

The multimodal experiments follow. We fine-tune the Multimodal Transformer with the visual gate on Multi30K English–German and English–French. We evaluate on the test sets, on Ambiguous COCO, and on CoMMuTE, using the IKEA dataset for long technical descriptions full of product names that require copying.

The decisive check is the random-image test. Our model should fall no lower than its text-only counterpart. The original Multimodal Transformer fell from 37.8 to 36.7 BLEU under this test. Our model should also keep or improve its CoMMuTE accuracy relative to the 65.0 figure, which tests whether the gate opens when the image matters.

The proposal is refuted under three conditions:

  • The counterfactual term fails to beat plain self-critical training at matched compute.
  • Heuristic switch supervision matches it on both translation quality and copy precision.
  • The visual gate cannot prevent the random-image degradation.