Letting Caption Slots Say No: Tagger Noise and Hallucination in Novel Object Captioning

Image captioners trained on paired image–sentence data know only a small vocabulary; COCO covers 80 object categories. To describe objects outside that set, the standard move is to borrow labels from an external object tagger and force them into the sentence. Some systems do this at decoding time, making beam search keep only sentences that contain the tag. Others generate a sentence template with blank "slots" and then fill each slot with a tagged object name. This works when the tagger is right. When the tagger is wrong, the forcing mechanism guarantees the mistake appears in the caption. The tagger is effectively a distant supervisor, a heuristic source of labels that is never checked against the specific image and sentence, and hard constraints pass its false positives straight through as hallucinated objects.

We propose to make the slot itself the unit where the captioner can decline. After a template is drafted, each slot has three possible fillers: the specific tagged name ("accordion"), a coarser name ("instrument"), or nothing, in which case the phrase is removed and the sentence redrafted. The choice combines two signals. The first is image evidence for the filled phrase in its sentence context. The second is a per-class estimate of how often the tagger is wrong, borrowed from the distant-supervision view that label noise is class-dependent and asymmetric. The claim is that this keeps most of the vocabulary gains of constrained captioning while greatly weakening the direct transfer of tagger errors into hallucinations. The claim can fail in a specific way that we test: simply thresholding the tagger's own confidence may do just as well.

Core Claim

Under hard constraints, a caption's object hallucination rate on novel classes is roughly the product of two quantities, the tagger's false-positive rate $\rho$ and the probability $q$ that a supplied tag ends up in the caption:

$$H_{\text{novel}} \approx q \cdot \rho ,$$

with $q \approx 1$ for constrained beam search by construction. This is simple bookkeeping. If every tag must appear, every wrong tag becomes a wrong object. Filtering the tags before decoding lowers $q$ equally for correct and incorrect tags, because it only sees the tagger's score. Our hypothesis is that evidence gathered after the template exists can lower $q$ much more for false tags than for true ones. That evidence is whether this phrase, in this sentence, is supported by this image region. Pre-decoding filters cannot do this. For each slot with tagged word $w$, detected region $r$, and surrounding sentence context $c$, we select

$$f^\star = \arg\max_{f \in \{w,\ h(w),\ \varnothing\}} \Big[ E(f; r, c) + \log \pi(f) \Big],$$

where $h(w)$ is a coarser name for $w$ (a hypernym from a lexical hierarchy). $E$ is an image–text score between the region crop and the phrase rewritten with filler $f$, contrasted against the same phrase with the tagger's top competing labels. $\pi(f)$ is a prior built from the tagger's estimated class-conditional precision, so that tags from classes the tagger often confuses need more image evidence to be committed. The empty filler receives a fixed score that is tuned once on validation data. The new element is not tag denoising, which prior work already does. It is the decision point: the choice is made at the slot, in context, with a graded fallback to a coarser name.

Relation to Prior Work

Our starting point is the template and constraint line of novel object captioning. Guided Open Vocabulary Image Captioning with Constrained Beam Search forces tagger outputs into beam search through a finite-state machine. It shows this prevents prior-driven substitutions such as "refrigerator" for "suitcase," but it binds output correctness to tagger correctness and has a search space that grows as $2^m$ in the number of constraints. Decoupled Novel Object Captioner generates templates with placeholder tokens and fills them from a detector memory. Cascaded Revision Network for Novel Object Captioning drafts with familiar words and then swaps uncertain words for detector labels. nocaps: novel object captioning at scale evaluates slot-based and constrained models and documents a steep drop out of domain. All four supply our mechanism: the slot is a natural place to intervene. None of them lets the slot refuse a tag. We keep their templates and add that refusal.

The closest competitor handles noise before generation. RCA-NOC: Relative Contrastive Alignment for Novel Object Captioning harvests CLIP tags, treats them as noisy labels, and reweights them with uncertainty-aware selection and relative ranking. VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning learns a visual vocabulary from image–tag pairs before caption fine-tuning. Both decide which concepts matter at the image level, before any sentence exists. Neither evaluates hallucination with an open-vocabulary metric. We differ by scoring the concept within the phrase that asserts it and by offering a coarser fallback instead of a binary keep-or-drop.

The distant-supervision literature constrains how we model the tagger. Stanford-UBC at TAC-KBP shows that co-occurrence-based labels teach spurious shortcuts. This is why our prior is class-conditional rather than a single global threshold. Relational Distant Supervision for Image Captioning without Image-Text Pairs trains captioners on pseudo image–sentence pairs aligned through shared entities and reports large CIDEr gains. It never measures hallucination, so it motivates our final experiment on whether unverified constrained captions, reused as pseudo-labels, amplify errors.

The hallucination literature motivates the problem and supplies the measurements. Object Hallucination in Image Captioning introduces the CHAIR metric, which counts caption objects absent from the image, and shows that optimizing CIDEr raises hallucination. We therefore never judge success by CIDEr alone. ALOHa: A New Measure for Hallucination in Captioning Models extends hallucination measurement beyond COCO classes and releases nocaps-FOIL and HAT. These give us an open-vocabulary metric and a test bed for our phrase scorer. Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning? decodes isolated "unit facts" to stop errors propagating through a sentence. This supports our choice to verify each slot's phrase separately. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts from the image to prior text as generation lengthens, which is another reason to check slots against a region crop rather than trusting the decoder. Evaluating Object Hallucination in Large Vision-Language Models introduces POPE, which asks yes/no existence questions per object, and documents a strong "yes" bias. It constrains one tempting alternative, discussed below. Anchor-Final Self-Supervision Drives Hallucination-Aware Optimization in Large Vision-Language Models reduces hallucination using internal layer signals. It needs white-box access and was tested only on COCO classes. Our method needs only a region crop and an external scorer, so it is complementary.

Research Program

The first component is a controlled measurement of noise transfer. We take a fixed template captioner (a placeholder model in the style of the Decoupled Novel Object Captioner, plus a constrained-beam-search variant over the same base captioner) and a fixed tagger. We then inject false tags at controlled rates. Injected tags come from two pools: random classes, and classes that frequently co-occur with the true objects, mirroring POPE's adversarial sampling. The output is a curve of novel-object hallucination against injected false-positive rate. The prediction is a slope near one for hard constraints and a much lower slope for slot commitment. This curve is the cleanest test of the core claim and does not depend on any particular tagger.

The second component is the slot scorer. Region crops come from detector boxes. Phrases are the slot word plus its syntactic neighbors (the verb and prepositional attachment). $E$ is a contrastive image–text score against the tagger's runner-up labels. We validate $E$ before using it by checking that it separates true from foiled words on nocaps-FOIL and agrees with expert labels on HAT. If it fails there, the method is not worth running.

The third component is the class-conditional prior. We estimate each class's tagger precision on the nocaps validation images, which have verified image-level labels, and shrink small-sample estimates toward the global mean. The coarser-name fallback uses a fixed lexical hierarchy, so no learning happens in that step.

Baselines

The naive reference point is the same base captioner decoding with no tags at all. It defines the vocabulary gain that constraints buy and the hallucination floor that comes from language priors alone.

The classic constrained baseline is constrained beam search with the same tagger, as in Guided Open Vocabulary Image Captioning with Constrained Beam Search. On nocaps, the best such configuration reported by nocaps: novel object captioning at scale (Up-Down + ELMo + CBS) reaches 73.1 CIDEr overall and 66.7 CIDEr out of domain. We rerun it with our tagger so that differences come from the decoding rule and not from the detector.

The strongest existing method is RCA-NOC, which reports 101.1 CIDEr on the nocaps test set and 79.5 F1 on held-out COCO (RCA-NOC). We found no reported hallucination result on Lacuna for RCA-NOC under CHAIR or ALOHa, so we measure it ourselves. If the released model is available, we run it as is. Otherwise we reimplement it, report both our numbers and the published ones, and apply slot commitment on top of RCA-NOC's tags. This tests whether the benefit adds to image-level denoising.

The baseline most likely to match us is detector-confidence thresholding. Each tag is kept or dropped by the tagger's own score, and the threshold is tuned on nocaps validation with the same budget as our $\tau$ and prior. All methods share the tagger, the base captioner, and the validation split, and hyperparameters are searched over the same number of configurations.

Three ablations isolate the key ingredient. The first removes context by scoring the bare tag against the region instead of the phrase. The second removes the class-conditional prior. The third removes the coarser-name option, leaving keep-or-drop only.

Beating a baseline has a precise meaning here. At matched novel-object mention rate (held-out COCO F1 within one point, or matched out-of-domain object recall on nocaps), slot commitment must show lower open-vocabulary hallucination under ALOHa and lower CHAIR on the eight held-out COCO classes. It must do so without a CIDEr drop larger than one or two points on the nocaps out-of-domain split.

Why not just…?

Why not just threshold the tagger's confidence? This is the real threat. If tagger errors are well calibrated per class, a tuned threshold captures most of what our prior does, and the context term adds little. Two points argue against it. First, nocaps's blacklist and overlap heuristics still left constrained models far below humans out of domain (nocaps). Second, RCA-NOC found hard filtering inferior to graded reweighting. Neither result compares against context-aware slot decisions, though. The matched-recall comparison and the "no context" ablation settle the question directly. If thresholding lies on the same precision–recall frontier as our method, we will report that the gain comes from calibration alone.

Why not just ask a large vision-language model whether each tagged object is present? This is cheap and needs no templates. POPE shows such models answer "yes" to absent objects at rates near 99% in some settings, and MiniGPT-4 drops from 78.86 to 71.37 F1 when negatives are co-occurring objects (Evaluating Object Hallucination in Large Vision-Language Models). That is exactly the co-occurrence regime where tagger false positives concentrate. Existence questions also cannot catch a correct noun placed in a wrong relation. We include this verifier as a baseline with the same tags, since a stronger recent model could narrow the gap.

Why not just rerank whole captions with an image–text similarity score? Whole-caption similarity averages over many words, so a single wrong novel noun is diluted. The unit-fact results in Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning? suggest that isolating claims matters. We run whole-caption reranking over the same candidate set as a baseline. If it matches slot-level scoring, the slot structure is unnecessary.

Proposed Experiments

The evaluation uses nocaps (in-, near-, and out-of-domain splits; CIDEr and SPICE) and held-out COCO (F1 on the eight held-out classes, METEOR, and CHAIR, which applies because those classes are COCO categories). Open-vocabulary hallucination on nocaps is measured with ALOHa, and HAT and nocaps-FOIL are used to validate both ALOHa and our scorer.

The first decisive check is the noise-transfer curve. If slot commitment does not flatten the hallucination-versus-injected-noise slope relative to constrained beam search, especially for co-occurring false tags, the core claim is wrong.

The second is the matched-recall comparison against confidence thresholding and RCA-NOC described above.

The third is a small human study on nocaps out-of-domain images. It labels whether each committed phrase's relation or attribute is correct, not just its noun. This tests the distant-supervision-motivated prediction that constrained captions often carry the right noun inside a prior-driven wrong context, an error that object-level metrics cannot see.

The last experiment connects to Relational Distant Supervision for Image Captioning without Image-Text Pairs. We generate pseudo-captions for unlabeled Open Images photos with constrained beam search and, separately, with slot commitment. We train one captioner on each set and compare their hallucination on nocaps. If unverified pseudo-captions yield higher CIDEr but also higher ALOHa hallucination, that would confirm the concern, raised by Object Hallucination in Image Captioning, that distantly supervised captioning gains can partly reflect learned hallucination.