Seeing Double: Copy-Move Clones as a Test of Attribute Hallucination in Vision-Language Models

When someone copies an object inside a photo and pastes the copy elsewhere, the result contains two regions that are identical pixel for pixel. That identity gives us something hallucination research rarely has: an exact, free ground truth about attributes and relations. If a vision-language model says the two dogs differ in color, pose, or size, it is fabricating, because they are the same dog twice. Current hallucination benchmarks mostly ask whether an object exists anywhere in the image. Duplicated objects let us test a finer failure: the model inventing differences, interactions, or separate identities for things that are provably identical.

The same edit also asks a forensic question. Does the model notice that it is seeing the same evidence twice? We propose to build a controlled clone benchmark on natural images and use it to measure clone-induced hallucination. We will then find where the failure lives: in the vision encoder, which may not separate a clone from a natural pair of look-alikes, or in the language decoder, which may hold that information and ignore it. That diagnosis decides the remedy. If the model's internal states already encode duplication, a cheap training-free steering fix should work. If they do not, an external forensic detector is needed, and we will say so.

Core Claim

Large vision-language models (LVLMs) are image encoders attached to large language models. They treat pixel-identical copies as independent entities. As a result, they fabricate attribute differences and relations between clones at a measurable rate, and they rarely report duplication when asked. Our central hypothesis is testable: a linear probe on intermediate decoder states can separate a clone pair from two matched controls, a natural pair of same-category instances and a spliced pair from different photos, well above the model's own answer accuracy. In other words, the model "knows but doesn't say."

Two outcomes are possible. If the hypothesis holds, a steering vector derived from those states should reduce fabrication and raise duplicate detection without the "timid," clipped answers that aggressive anti-hallucination methods produce. If it fails, the result is still informative: it shows that duplication awareness has to come from outside the model.

Relation to Prior Work

The closest work is Copy-Move Forgery Detection and Question Answering for Remote Sensing Image. It defines copy-move question answering for satellite imagery and solves it with a pipeline: an external U-Net segmenter proposes source and tampered regions, and their relational features are injected into the text stream. It reports that general models such as BLIP-2 treat clones as natural repetition, and that textual forensic hints help more than visual masks. We differ in three ways:

  • What we study. We treat clones as a probe of hallucination on natural images, not as a forensic task to solve.
  • What we ask. We ask whether the LVLM's own representations carry the duplication signal, which that paper leaves open because it always relies on an external segmenter.
  • How we test it. We add controls that separate "this image was edited" from "these two regions are the same."

The hallucination literature motivates the measurement. Object-existence benchmarks such as POPE, from Evaluating Object Hallucination in Large Vision-Language Models, expose a strong "yes" bias and co-occurrence effects, but they say nothing about attributes or relations between instances. Two further findings point toward decoder-side causes. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that long generations drift toward attending to previously generated text rather than the image. Villa et al. show that correct answers can rest on context cues rather than perception. Both suggest that captions of clone pairs will pick up invented distinctions ("an older and a younger dog").

Two works supply our method. SHARP finds that hallucinated and faithful states are linearly separable in middle layers (roughly layers 10–15). It builds steering vectors specific to each cause of hallucination and adds them to hidden states during generation. We reuse that probing-then-steering recipe for a new cause, duplicated evidence. BusterNet introduces all-to-all patch self-correlation with percentile pooling, a way to compare every image patch with every other patch and summarize the result as a fixed-length vector. We reuse it as a training-free readout over the LVLM's own vision tokens.

One method constrains our design. VisER scores grounding by projecting image-patch representations into the vocabulary (the "logit lens"). With clones, both copies vote for the word, so evidence is double-counted. We test this directly rather than assume it.

Research Program

The benchmark. Starting from COCO validation instance masks, we paste a copy of a selected instance into a plausible empty location. The copy is made under a graded set of transformations: identity, horizontal flip, rotation, scaling, and boundary feathering. The synthetic USC-ISI dataset is built from COCO and SUN objects, so its generation style is compatible with ours. Each clone image is matched with two controls, balanced for category, instance size, and spatial separation:

  • Natural-twin control: a real COCO image containing two instances of the same category.
  • Splice control: the same host image with an instance of the same category pasted from a different photo. This image is edited but not duplicated.

The splice control is essential. Without it, any detector, including a probe, could succeed by spotting paste boundaries rather than duplication.

The questions. We ask four kinds of question on the benchmark, all in POPE's templated form to keep variance across prompts low:

  • Duplication: "Are the two dogs exact copies of each other?"
  • Attribute comparison: "Which dog is larger?" or "Do they have different colors?" When the clone is not scaled or recolored, the only correct answer is "no difference."
  • Counting.
  • Open captions.

For captions we define the twin fabrication rate: the fraction of clone images whose caption asserts any attribute difference, interaction, or distinct identity between source and clone that the transformation did not introduce. Claims are extracted by a parser, and a sample is checked by hand. We will not use an LVLM judge, given the visual biases documented in Fooling the LVLM Judges.

For realism we add the 1,313 real copy-move images of CASIA CMFD, with hand-written questions on a subset. Real-RSCM serves as an out-of-domain check against the one prior multimodal copy-move result.

The diagnosis. On LLaVA-1.5 and LLaVA-Next we train linear probes in two places:

  • at every decoder layer, on the hidden state at the final question token;
  • on BusterNet-style self-correlation statistics of the vision encoder's patch tokens.

For the encoder readout we compute correlations at two grid offsets. An arbitrary paste location usually breaks alignment with the patch grid, so a single grid would miss many clones.

Each probe separates clone from natural twin, and clone from splice. The gap between probe AUROC and the model's own answer accuracy tells us where the information is lost:

  • Small encoder signal: the encoder cannot see duplication.
  • Large encoder signal but small decoder signal: the information is lost between encoder and decoder.
  • Large decoder signal but poor answers: the decoder has the information and ignores it.

The remedies. If decoder states carry the signal, we take a difference-of-means steering vector (clone minus the pooled controls) from a training split and add it to middle layers at inference, following SHARP. If only the encoder self-correlation carries it, we turn that score into a one-line textual hint. Huang et al. found textual hints more effective than visual masks. If neither carries it, the remedy is an external detector, and our contribution reduces to the benchmark plus a negative diagnostic.

Baselines

Do nothing. The first reference point is the unmodified LVLM with greedy decoding, asked the benchmark questions directly. A close variant adds an explicit instruction: "check whether any object appears duplicated." This separates missing capability from simply not being asked to look.

External detector plus text hint. This is the strongest competitor and mirrors the CMFPF design of Huang et al. on natural images. A dedicated copy-move detector produces a mask, which is converted into a text hint naming the duplicated region. We will use LBRT, a dual-branch Transformer with global and fine sub-patch attention, or IMNet, which mines inconsistencies against source and target prototypes.

The detector's component accuracy is known. IMNet reports 82.16% F1 on USC-ISI CMFD and 52.47% F1 on CASIA CMFD, and BusterNet reports 45.56% F1 on CASIA. For the whole pipeline, the only reported multimodal copy-move result is CMFPF's 93.89% overall accuracy on RS-CMQA and 92.79% on Real-RSCM, both in remote sensing. No reported result was found on Lacuna for copy-move question answering or twin fabrication on natural images.

To keep the comparison fair, the detector is trained on the same synthetic clone split used to fit our probes and steering vectors. It receives the same held-out transformations and the same hint template. Its runtime is reported alongside ours.

Other hallucination methods. We also run RITUAL, which adds randomly transformed image views during decoding. It is the strongest general training-free method in the file, raising LLaVA-1.5 on POPE to 88.87% from 84.13%. We also run SHARP's original steering vectors for text-prior and vision-text-conflict hallucinations. The aim is to see whether generic anti-hallucination steering already fixes twin fabrication.

Fine-tuning. Low-rank adaptation is a cheap way to fine-tune only a small added set of weights. We fine-tune on the synthetic clone data with a matched budget of examples and compute.

Ablations. Two ablations isolate our key ingredient. The first computes the steering vector against natural twins only, dropping the splice control, to test whether we are merely steering toward "edited." The second uses a single-offset self-correlation readout, to test whether grid alignment matters.

What counts as success. Beating the external pipeline means one of two things. The steering method could match its duplication accuracy and twin fabrication rate within noise at a fraction of the cost, with no separate network. Or it could exceed it on held-out transformations and CASIA, where detector errors propagate. Matching the pipeline only on in-distribution identity clones does not count.

Why not just…?

Why not just run an off-the-shelf copy-move detector and put its output in the prompt? This is the real threat. Huang et al. show the approach works in remote sensing. It could plausibly match us on in-distribution clones, so it is a first-class arm of the experiments, not a strawman. Two pieces of evidence suggest limits. Detectors are weak on realistic images (roughly 45–52% F1 on CASIA). And even with masks, Huang et al. report that their model could not tell whether a clone had been rotated. Our held-out-transformation and CASIA splits measure both issues. If the pipeline wins everywhere, we report that the model lacks intrinsic duplication awareness and that pipelines are the right answer; the benchmark and fabrication measurements still stand.

Why not just use test-time augmentation such as RITUAL? Global flips and crops move source and clone together, so their pixel identity is preserved and nothing exposes the duplication. RITUAL also doubles the number of forward passes, and Perception Magnifier notes that geometric warping distorts object shape. We run RITUAL anyway, because diversified views might reduce invented attribute differences even without detecting clones. That would be a meaningful partial win for it, and we will report it as such.

Why not just fine-tune the model on forged images? Fighting Fake News argues that supervised training on manipulations overfits to specific artifacts. It trains instead on a self-supervised consistency signal learned from camera metadata. Fine-tuning also risks damaging general behavior. We test both concerns directly: the fine-tuned model is evaluated on held-out transformations, on splice controls (to check it has not learned "edited implies duplicate"), and on POPE and MME for regressions.

Why not just use VisER-style grounding scores? Grounding scores check whether patches support a word, and both clones support it equally. We will measure whether VisER's visual-evidence score is indeed indistinguishable across clone and natural-twin mentions. If it is not, that is a cheap signal we would adopt.

Proposed Experiments and Decisive Checks

Fabrication. The first check is whether twin fabrication exists at a meaningful rate in LLaVA-1.5 captions of identity clones, and whether it grows with caption length, as the long-response drift finding predicts. If fabrication is rare, the hallucination half of the proposal collapses and we say so.

Diagnosis. The second check compares, layer by layer, probe AUROC for clone versus natural twin and clone versus splice against the model's answer accuracy. The "knows but doesn't say" hypothesis requires a clear gap in middle layers on both contrasts. A probe that separates clones from twins but not from splices would mean it has learned "edited," not "duplicated."

Steering. The third check asks whether steering lowers twin fabrication and raises duplication accuracy under three conditions:

  • on held-out rotation and feathering clones and on CASIA CMFD;
  • without raising false alarms on natural twins;
  • without harming POPE Adversarial, CHAIR, or MME Perception relative to the unmodified model. SHARP's own figures give a sense of what healthy steering looks like: 52.8→34.8 sentence-level CHAIR and 510.0→613.3 MME Perception on LLaVA-v1.5.

Head-to-head. The final check is the matched comparison against the external-detector pipeline across in-distribution and held-out splits. Its outcome determines whether duplication awareness should live inside the model or beside it.