Querying Confident Shortcuts: Explanation-Aware Active Learning for End-to-End Multimodal Models

Active learning picks which unlabeled examples to send to a human annotator. The standard rule is to pick the examples the model is least sure about. That rule has a blind spot. A model that has learned a shortcut, such as reading a label off a caption keyword or a background texture while ignoring the actual object, is very sure about those examples. Uncertainty sampling never asks about them, so the shortcut is never corrected.

We propose the opposite rule for multimodal models trained end to end. We search for examples where the model is confident and its evidence is concentrated on a few image patches or text tokens. We then check with a cheap deletion test that this evidence really drives the prediction. For the selected examples we buy both a label and a rationale, meaning a highlighted region or span that marks the evidence that should matter. The rationale then trains both the image encoder and the text encoder through a penalty on the model's explanation (its per-patch and per-token importance scores, called attributions from here on).

The claim is testable and has three parts. First, this rule finds shortcut examples that output-uncertainty rules miss. Second, spending the rationale budget on those examples beats spending it at random or on uncertain examples. Third, the benefit depends on the encoders being trainable: with frozen encoders the explanation cannot change, so the advantage should largely disappear.

Core Claim

The evidence that motivates this comes from two places. In graphs, Overcoming Shortcut Learning in Graph Neural Networks through Active Explanation Guidance (XIGL) queried nodes where attribution was most concentrated and supplied human correction masks. This raised shortcut-free accuracy on CPatchMNIST from about 0% to over 30%. Prediction-uncertainty querying failed at the same task because shortcut predictions are confident. In vision-language models, Beyond Answer Correctness showed that 87.1% of BakLLaVA's correct ChartQA-Pro answers rest on fabricated evidence. High accuracy and confidence therefore say little about whether the model is right for the right reason.

Neither result has been carried into multimodal active learning, and the existing multimodal methods do something different:

  • Towards Balanced Active Learning for Multimodal Classification (BMMAL) computes per-modality Shapley contributions, a game-theoretic score of how much each modality adds to a prediction. It uses them as scalar weights on gradient embeddings to rebalance label acquisition. It never queries or supervises explanations, and its cost grows as $2^M$ forward passes for $M$ modalities.
  • Towards Multimodal Active Learning chooses image-text pairs by matching-margin uncertainty, a purely black-box signal.
  • MEGL does back-propagate explanation supervision through a multimodal model, but only passively over a fully annotated dataset.

Our contribution sits in the gap between these. We select where to spend explanation annotation using a verified explanation signal, restricted to the region of confident predictions that uncertainty rules ignore. We then show that this only pays off when the encoders can be updated.

Research Program

Acquisition score. For an unlabeled candidate $x$ with predicted class $\hat y$, define the following quantities:

  • $c(x) = p(\hat y \mid x)$ is the model's confidence.
  • $\kappa(x)$ is the fraction of total absolute attribution mass, pooled across image patches and text tokens, that falls in the top-$k$ features. It measures how concentrated the evidence is.
  • $\delta(x) = c(x) - p(\hat y \mid x_{\setminus k})$ is the confidence drop after masking those top-$k$ features.

The score is

$$ s(x) = c(x)\,\kappa(x)\,\mathbb{1}\!\left[\delta(x)/c(x) > \tau\right], $$

where $\tau$ is a threshold on the relative drop. The indicator is the key ingredient borrowed from the faithfulness literature. It keeps only candidates whose concentrated attribution actually drives the prediction. This filters out the case the Beyond Answer Correctness study warns about, where an attractive heatmap is not what the model uses.

Attributions are gradient-times-input. Each candidate therefore costs one backward pass plus one extra masked forward pass. That is cheaper than Monte Carlo dropout and than BMMAL's exponential Shapley enumeration. To avoid querying near-duplicates, we pick the final batch by k-means++ over the top-scoring candidates' embeddings, following BMMAL's diversity step.

Training. Queried examples contribute a cross-entropy loss and a rationale loss that penalizes attribution mass outside the annotated region or span. We use the XIGL-style $L_e$ penalty, generalized to both modalities. We compare three encoder regimes: frozen encoders with a trainable fusion head, low-rank adapters (LoRA) on both encoders, and full fine-tuning. Frozen is the configuration a compute-constrained reviewer would prefer. The research direction on end-to-end training, for example WSI-VQA, argues that frozen extractors cannot refine their representations. Our design turns that argument into a measurable interaction effect rather than an assumption.

Why the confident region also helps with noise. Maximum-uncertainty querying tends to fill the labeled pool with ambiguous or mislabeled examples. ALFLTP found that median-uncertainty selection converges better than maximum-uncertainty selection for exactly this reason. e-SNLI-VE-2.0 reports that 31% of the original neutral labels were invalid. Our score selects from the confident region, so we expect it to draw fewer examples whose ambiguity is inherent to the data. We measure this directly rather than assume it.

Baselines

Random selection with no explanation annotation. This is the naive reference: labels only, drawn at random, trained under the same encoder regime.

Strongest label-only competitor: BMMAL. It is the strongest existing method on the closely related question of acquiring multimodal examples that correct modality dominance. It is concrete and published. On VGGSound, BMMAL reports reaching 43.5% target accuracy with about 5,000 fewer labels than competitors. We run it with its Shapley-weighted BADGE procedure on the same pool, seed set, batch size, backbone, and number of rounds. Hyperparameters get the same tuning budget, a fixed number of trials on a held-out split. Beating it means reaching BMMAL's final accuracy on the shortcut-free test split with fewer queries, or exceeding its accuracy at equal queries, across at least three seeds.

Uncertainty rules. We also run margin sampling and BADGE, the canonical uncertainty rules, under identical conditions.

Upper reference: passive explanation-guided training on the full dataset. This is MEGL trained with rationales on all data. On Object-ME, MEGL reports 83.17% accuracy for MEGL-ViT, against 78.58% for a plain ViT and 82.33% for LLaVA fine-tuning. We treat 83.17% as the ceiling a budgeted method should approach. There is no reported result on Lacuna for active learning of any kind on Object-ME or Action-ME, so all budgeted numbers there will be new.

Ablations isolating the key ingredient. We run three variants of our own score:

  • removing the deletion gate, which reduces the score to XIGL-style minimum explanation entropy;
  • flipping the score to select diffuse explanations, which approximates maximum explanation entropy;
  • keeping the score but dropping the rationale loss, so only labels are used.

The last one separates the effect of which examples are chosen from the effect of what is annotated.

Why not just…?

Why not just use margin or BADGE? Output-uncertainty rules rank shortcut examples near the bottom because the model is confident on them. XIGL observed exactly this failure in graphs. BMMAL showed that BADGE and entropy sampling are pulled toward the dominant modality in multimodal classification. Our experiments report the fraction of each strategy's queries that contain the planted shortcut, which tests this mechanism directly rather than only through final accuracy.

Why not just use BMMAL's modality Shapley values to pick examples, then add rationales? This is the most serious threat. A per-sample modality imbalance score may already capture "the model is relying on the text alone." Our score differs in two ways: it is spatially fine-grained within a modality (a watermark patch, a single keyword), and it is verified by deletion. Neither property matters if shortcuts are purely modality-level. We will run BMMAL selection with our rationale loss at an identical budget. If it matches our method on in-modality planted shortcuts, we will report that the contribution is the explanation-querying regime rather than our score. The benchmark includes both a modality-level shortcut (a caption keyword) and a within-modality shortcut (an image watermark), so the two cases can be told apart.

Why not just annotate rationales for random examples, as in MEGL? If shortcut examples are common, random rationale annotation will hit them often enough and selection adds nothing. We expect selection to matter when shortcut examples are a small fraction of the pool. We therefore vary the fraction of the pool carrying the shortcut, from rare to common, and predict that the advantage over random selection shrinks as the shortcut becomes common. If the advantage is flat at every rate, the claim fails.

Why not just freeze the encoders and train an adapter? It is cheaper, and if it works as well, the end-to-end part of the proposal is unnecessary. However, when the image encoder is frozen, a rationale loss can only reweight fixed features. It cannot move where the encoder itself attends. The frozen, LoRA, and full comparison settles this. Our prediction is that the gain of our score over BMMAL shrinks substantially under freezing. If it does not, we will recommend the frozen setting.

Proposed Experiments

Datasets. Object-ME and Action-ME from MEGL provide ground-truth saliency and rationales. We simulate the annotator by revealing a queried example's ground-truth rationale. e-SNLI-VE-2.0 provides a visual-entailment check with explanations; the file reports prior accuracies of 72.52% for BUTD and 73% for PAE. VGGSound and Food101 support a label-only comparison against BMMAL; Food101 has no reported active-learning number on Lacuna.

Planted shortcuts. On Object-ME and Food101 we create controlled versions. In the training pool, a label-correlated caption keyword or a small image watermark is injected into a controlled fraction of examples. Evaluation uses a shortcut-free test split, in the spirit of CPatchMNIST's corner artifacts.

Models. We use a ViT image encoder and a text encoder with a fusion head, at the scale MEGL runs, so that full fine-tuning is feasible on academic hardware.

Metrics.

  • Accuracy on the shortcut-free test split, as an area-under-curve over labeling rounds.
  • Saliency agreement with ground-truth masks (mIoU, as in MEGL).
  • Deletion-based faithfulness on held-out data.
  • The shortcut-hit rate of each strategy's queries.

Decisive checks. The proposal succeeds if three things hold. First, our score achieves a higher shortcut-hit rate and higher shortcut-free accuracy than margin, BADGE, and BMMAL at equal budget. Second, it beats both random selection with rationales and BMMAL selection with rationales on within-modality shortcuts. Third, the gap narrows substantially under frozen encoders.

A secondary check addresses noise. We inject 20% label noise into the Object-ME pool and count how many noisy examples each strategy queries. This tests whether confident-region selection avoids the noise problem that ALFLTP and e-SNLI-VE-2.0 document for maximum-uncertainty rules. The ablation without the deletion gate tests whether verifying faithfulness matters, or whether plain explanation concentration, as in XIGL, is enough.