Known, Refuted, or Unknown: Three-Valued Claim Checking for Web Image-Text Curation

Web image captions say three different kinds of things about their pictures. Some statements the picture confirms ("a red awning"). Some the picture refutes ("an empty street" over a photo of a traffic jam). Most of the rest the picture simply cannot settle ("where John bought his wedding cake in 1995"). Current filters for vision-language training data score each image-caption pair on a single similarity axis, so they treat the third kind exactly like the second. That choice has a cost. A refuted claim trains the model to bind visual features to the wrong words. An unverifiable claim is often harmless and frequently carries the proper nouns, places, and written text that synthetic recaptioning is known to strip out. We propose to curate web data with a three-valued test borrowed from the logic of incomplete knowledge bases. Each caption is split into atomic claims, and each claim is checked against a cheap per-image evidence record as known true, known false, or unknown. Pairs are then penalized only for what the image actually refutes.

The logical idea comes from Levesque's treatment of querying an incomplete knowledge base, where a query can return yes, no, or unknown, and where "the knowledge base knows φ" reduces to an ordinary classical query over a finite base (The Interaction with Incomplete Knowledge Bases: A Formal Treatment). We treat each image's evidence record as that finite base. The operational question this raises is when failing to detect an object licenses the conclusion that it is absent. Our answer is a per-concept, calibrated "local closed-world" rule. Whether that rule, rather than any heavier logic, drives downstream gains is the central empirical question.

Core Claim

Separating "refuted" from "unknown" at the level of individual claims, and deciding refutation only for concepts whose detector is shown to be reliable enough that non-detection means absence, will produce better training subsets than single-score filters at the same data budget. The gains should concentrate on the tasks that caption-based filters damage: text in images, named entities, and geography. The decisive ablation is the closed-world variant. It is identical in every respect except that every undetected claim counts as false. If the closed-world variant matches the full method, the epistemic distinction adds nothing and the idea is wrong.

The intersection has not been done in this form, as far as the file shows. Pelican decomposes model outputs into first-order predicates and checks them against a detector-built table. It works only at inference time, uses heavy detectors and LLM-generated code, and has no notion of "undetermined" separate from "false." Generalizing Image Captions for Image-Text Parallel Corpus named the noise categories (circumstantial, extraneous, uncommon) that motivate us. It removed ungrounded words by n-gram compression and could not tell absence from ignorance. SIEVE and Matched Data, Better Models curate at web scale but output one score per pair or match global concept counts, and neither checks individual claims. The papers that use "epistemic" in the vision-language literature, such as FUSE and On Epistemic Uncertainty of Visual Tokens, mean statistical uncertainty in model parameters, not the known/unknown status of a claim.

Research Program

The pipeline has three parts.

Claim extraction. A small, fast text model is distilled to emit ground atoms from each caption. The atom types are object presence, attribute, count, simple relations, quoted or visible text, and a residual "context" atom type for named entities, dates, and narrative. The distillation teacher is an LLM run on a small sample.

Evidence record. For each image we store calibrated per-concept presence probabilities from an open-vocabulary tagger over a fixed vocabulary, together with OCR tokens. The tagger must not be the CLIP model whose scores define the CLIPScore baseline, so that we do not reproduce the circularity SIEVE diagnosed.

Query reduction. In the spirit of Levesque's finite-base reduction, each modal query K(atom), K(¬atom), or neither becomes a classical lookup with two thresholds. High presence probability yields known-true. Low probability yields known-false only if the concept is in a "closed" set: concepts whose detector recall, measured on exhaustively annotated held-out images, exceeds a target. Otherwise the answer is unknown. This mirrors the knowledge-producing sensing actions of The Frame Problem and Knowledge-Producing Actions. An observation settles a fluent only if the sensor would have reported it. Here a detector settles absence only if it would have fired.

Context atoms are always unknown unless OCR confirms them, in which case they become known-true. This is how the method keeps the text in images that captioners drop. For the small fraction of captions with negation, quantifiers, or counts ("no people," "two dogs"), we compile the atom set into a tiny program and evaluate it with a standard answer set solver. The goal is to express "known-not" and "possibly" queries correctly, following the rewriting in Evaluating Epistemic Logic Programs via Answer Set Programming with Quantifiers. The bulk of the data needs only the table lookup, and we will report the cost split honestly.

Each pair's epistemic profile is then turned into a selection score. Known-false atoms are penalized heavily, known-true atoms are rewarded, and unknown atoms are neutral. A secondary variant rewrites rather than filters: it removes refuted spans and keeps unknown ones, which is the epistemic counterpart of the Kuznetsova compression.

Baselines

Naive reference. The do-nothing reference is the DataComp "no filtering" setting at the medium scale. It uses the same fixed CLIP architecture and training recipe as DataComp, so every comparison differs only in which pool samples are kept.

CLIPScore thresholding. This is the standard pointwise filter, with the threshold tuned over the same grid of kept fractions we allow ourselves. The file reports no absolute DataComp-medium number for CLIPScore alone, so we will reproduce it.

SIEVE. This is the most directly competing caption-based method, because it targets the same misalignment through captioner-to-alt-text similarity. SIEVE reports that Sieve combined with CLIPScore beat CLIPScore by 2.6% averaged over the 38 DataComp tasks at medium scale and by 1.7% at large scale. It also reports reaching peak performance with 20% of the data. We will run Sieve with the same kept-fraction grid.

SDM. This is the strongest number in the file on ImageNet. Matched Data, Better Models reports 35.2% zero-shot ImageNet-1K accuracy on DataComp-medium. It is a different kind of competitor because it selects for distribution coverage rather than per-pair truth. We will therefore also test our score as a pre-filter feeding SDM's selection.

Beating the baselines. "Beating" means exceeding Sieve plus CLIPScore on the 38-task average at medium scale under an equal kept-sample budget and equal curation compute, without falling below it on retrieval. It also means a clear gain on the OCR- and knowledge-dependent tasks where SIEVE reports weakness.

Key ablation. The ablation that isolates our ingredient is the closed-world variant: every undetected atom is known-false. A second ablation drops the per-concept recall calibration and applies one global absence threshold.

Why not just…?

Why not just use a caption-level visual entailment model with entailed, contradicted, and neutral outputs? This is the most serious threat, because it captures the three-valued idea without decomposing claims. Our bet is that a single caption-level label is dominated by the caption's most salient clause. As a result, a caption with one refuted object and a long unknown narrative gets labeled neutral, and a caption that is mostly verified but carries one useful proper noun gets labeled contradicted. We will run such a model, using the strongest open VLM we can afford as a judge on the same kept-fraction grid, and also give it the claim-level decomposition. If the caption-level judge matches us at equal compute, the decomposition is unnecessary and we will say so. If the claim-level judge matches us but costs far more, the contribution reduces to making that judgment cheap.

Why not just recaption everything, as BLIP's CapFilt does? BLIP shows that synthetic captions can beat raw alt-text. Two costs remain. Autoregressive captioning over the whole pool is expensive, and captioners discard the text in images and the named context that SIEVE found to matter. Pelican further documents substantial hallucination in vision-language models' own outputs, so recaptioning can substitute one kind of noise for another. Our rewrite variant keeps the original text minus refuted spans. A direct comparison on the knowledge-dependent tasks will show whether retention matters.

Why not just use soft targets from a momentum teacher? Momentum distillation softens partial mismatches inside a batch. A teacher trained on the same uncurated pool inherits its overconfidence, which Improved baselines for vision-language pre-training attributes to objectives with no mechanism for data ambiguity. Soft targets are also orthogonal to selection. We will add them to both the best baseline and our method to check that our gain survives rather than being absorbed.

Proposed Experiments

All main runs use DataComp-medium (128M candidate samples, fixed CLIP training) from CommonPool. We report the 38-task average, zero-shot ImageNet, retrieval, and a pre-declared subset of OCR- and geography-dependent tasks. The best configuration is confirmed at the large scale.

Intrinsic checks. The claim labeler is evaluated in two ways before any training. First, on the 1,000-image SBU evaluation set with gold compressed captions from Generalizing Image Captions for Image-Text Parallel Corpus, we test whether words humans deleted fall in our unknown or refuted atoms. Second, on a newly annotated few-thousand-pair CommonPool sample with three-way claim labels, we measure per-class precision, especially the precision of known-false.

Decisive checks. Three results decide the question. The first is the gap between the full method and the closed-world ablation at matched budget, which tests whether refusing to equate "unseen" with "absent" matters. The second is whether the gain over Sieve concentrates on the knowledge-dependent subset, as the mechanism predicts. The third is how the method compares with the claim-level VLM judge per unit of curation compute.

A null result on the first check, with the other two positive, would mean claim decomposition helps but the epistemic distinction does not. That would be a useful negative finding about whether the logic of incomplete knowledge is worth importing into data curation.