Closed-Loop Fact Checking for Generative Image Compression

Image codecs that use a generative decoder, a network that fills in missing detail with whatever looks plausible, now produce sharp pictures at very low bitrates. The sharpness is partly made up. A forensic study of such codecs, A Taxonomy of Miscompressions, catalogs the failures by hand. A sign reading 2264 comes back as 2254, and a commercial OCR system reads the wrong number as fact. People disappear from park scenes. Traffic arrows become dots, and brake lights switch off. These errors are dangerous because nothing in the output signals that anything went wrong. That paper stops at diagnosis, and no current codec checks its own output for this kind of damage.

We propose that the encoder do the checking, because the encoder already has what the check needs: the original image, and the ability to run the decoder and see exactly what the receiver will see. Specialist checkers each extract one kind of discrete fact from both images. These include text strings, the identity of the depicted entity as linked to an external knowledge base (for example, which Wikipedia landmark or species this is), person counts, and sign shapes. Wherever the facts disagree, the encoder spends extra "repair" bits on that region only, and repeats until the facts agree or a rate cap is reached. The idea fits in two sentences. Semantic damage is a change in discrete facts, not a small drift in pixels or embeddings. A sender that can simulate the receiver can detect those changes per image and pay to fix them only where they occur.

Core Claim

Our central claim is that per-image, closed-loop agreement on discrete facts preserves meaning at a lower bitrate than spending bits on "important-looking" regions in advance. Here is why we expect this. Most text, faces and landmarks survive low-rate generative coding intact, and only a minority flip. A loop that measures flips can leave the survivors cheap, while a fixed region-of-interest rule must over-pay for all of them.

The four research directions play distinct roles. The semantic–pixel fidelity tradeoff is the application regime. External knowledge supplies one of the checkers, an entity linker, and gives us an objective definition of "same meaning": does the decoded image still retrieve the same knowledge-base entity? The cross-modal representation gap is the reason the checks must be discrete. A continuous image–text embedding such as CLIP barely moves when 2264 becomes 2254, so optimizing cosine similarity, as Compression Beyond Pixels does, cannot see the error. Multi-agent task decomposition is the method lens. Each miscompression class from the taxonomy gets its own specialist checker with its own tool, in the spirit of role-separated critics in LEGO. Whether a language-model planner adds anything on top of fixed checkers is an ablation, not an assumption.

The objective can be stated compactly. Let $x$ be the image and $\hat{x}_R$ the decoded image when repair bits are spent on a set of regions $R$. Let $f_k$ be checker $k$, which maps an image to a discrete fact set such as strings, an entity ID or a count. The encoder approximately solves

$$\min_{R} \; r_{\text{base}} + \sum_{j \in R} r_j \quad \text{subject to} \quad f_k(\hat{x}_R) = f_k(x) \;\; \text{for all checkers } k,$$

where $r_{\text{base}}$ is the base-layer rate and $r_j$ is the cost of repairing region $j$, including the cost of signaling its location. We solve it greedily: decode, check, repair the regions that disagree, and decode again.

Research Program

The base layer is an off-the-shelf generative codec in the HiFiC or CDC family, the same codecs the taxonomy studied. The repair layer encodes flagged regions at high fidelity with a pixel-faithful learned codec, plus a short bit-mask of region locations. For text regions there is an optional lossless string, which the decoder renders under a conditioning signal, borrowing the text-adaptive spatial conditioning of Perceptual Image Compression with Cooperative Cross-Modal Side Information.

The checker roster follows the taxonomy's categories. A symbol checker runs OCR on detected text regions and compares strings. An entity checker links the image, or an object crop, to a knowledge-base entity using an entity-centric retriever in the style of EntRAG, and compares top-1 entity IDs. We use an entity-level checker because Encyclopedic VQA shows that entity identification is the bottleneck: PaLM rises from 19.7% to 87.0% with oracle retrieval. INFOSEEK shows the same with a rise from 18.2% to 45.6% given an oracle entity. A codec that silently changes the entity therefore wipes out everything downstream. A presence-and-count checker compares person and object detections. A shape-and-amplitude checker compares classifications of traffic arrows, signal states and light states. Each checker compares the original against the reconstruction, not against ground truth. As a result, a checker's misreading tends to appear on both sides and cancel, and false alarms cost bits rather than fidelity.

Three analyses accompany the system. The first measures the cross-modal gap directly: how well CLIP-similarity loss predicts fact flips across bitrates. The second is a rate accounting that splits repair bits by checker type. The third is an adversarial audit. The decoder is never trained against the checkers, but the repair loop could still settle on outputs that satisfy a particular OCR model while looking wrong to people. Every headline result is therefore scored with held-out checkers, meaning different OCR and retrieval models, and with human raters.

Baselines

The naive reference point is the base generative codec with no repair, run at matched bitrates. Its miscompression rate under our checkers is the problem we are measuring. A pixel-fidelity learned codec at the same rate is a second reference. It shows how far one gets by abandoning generative synthesis altogether, and it gives up perceptual quality to do so.

The strongest existing compression competitor is the cooperative cross-modal side-information codec (Perceptual Image Compression with Cooperative Cross-Modal Side Information). It conditions a generative decoder on BLIP2 or GPT-4 captions and reports an LPIPS improvement on Kodak "from 0.06225 to 0.05991" with five GPT-4 sentences per image. It is the closest published system that tries to inject semantics into a generative decoder. We will run it on the same images with captions counted in the rate. We also run a stronger variant whose side text is the exact OCR strings and entity names rather than generic captions. Beating it means fewer held-out fact flips at equal or lower total rate, without LPIPS worsening by more than the gap between their two reported values.

The strongest heuristic competitor is region-of-interest allocation: detect text, faces and signs and always encode them at high fidelity. It uses the same detectors as our checkers, the same repair codec and the same rate. Beating it means a lower rate at equal fact agreement. This is the decisive comparison.

For the cross-modal codecs, Cross Modal Compression and Scene Graph-Aided Probabilistic Semantic Communication, the file reports compression ratios and CLIP similarity but no fact-level scores. We include them only as extreme-rate reference points on COCO and Visual Genome.

Two ablations isolate the key ingredients. The first replaces discrete checkers with a CLIP-cosine threshold. The second replaces fixed checkers with a language-model planner that decides which checkers to run. The first tests whether discreteness matters. The second tests whether agentic orchestration matters beyond decomposition into specialists.

For downstream knowledge tasks, the uncompressed upper reference is EntRAG's reported accuracy of 44.5 on InfoSeek and 46.1 on E-VQA (EntRAG). We report what fraction of that accuracy each codec retains. For HiFiC and CDC on these knowledge benchmarks there is no reported result found on Lacuna, so those numbers will be produced by us under identical settings.

Why not just…?

Why not just protect text and faces everywhere with region-of-interest bit allocation? This is the real threat. If most protected regions would have survived anyway, region-of-interest allocation wastes bits and the closed loop wins on rate. If most of them flip at the operating bitrate, the two methods converge and the loop adds only decoder-simulation cost. The matched-rate comparison above settles it directly. We will also report the base codec's per-category flip rate, which predicts the outcome before any repair is run. If the flip rate is high everywhere, we will say the loop is unnecessary at that rate.

Why not just send exact OCR strings and entity names as side information? For text this may work well, and we include it as a baseline and as a repair option. The taxonomy's errors are not all symbolic, though: erased people, reshaped profiles and arrow shapes have no compact string form. Conditioning also does not guarantee rendering. A decoder told "2264" can still draw "2254", and only re-checking the output catches that. The entity-name variant of the side-information baseline measures how much of the gain comes from information transmitted versus verification performed.

Why not just train the codec with recognizer losses? Training with an OCR loss, as in the text-logit-loss work of Lai et al. summarized in the file, improves text accuracy on average but provides no per-image check. It also overfits to the recognizer used in training. We will train a recognizer-loss variant of the base codec and compare tail behavior, meaning the worst-case images, under held-out checkers. A closed loop should win specifically on the tail.

Why not just use a single large vision-language model to compare the two images? A single model is a plausible checker. It is also exactly the kind of closed-book system that fails on long-tail entities: PaLI reaches only 13.0% on E-VQA (Encyclopedic VQA). We test it as an alternative entity checker and expect it to miss entity swaps that a retrieval-based linker catches.

Proposed Experiments

The first experiment is knowledge-preserving compression. We compress InfoSeek and E-VQA images at three to four low bitrates, then measure two things: top-1 entity-link agreement between original and decoded images using a held-out retriever, and EntRAG-style question-answering accuracy as a fraction of the uncompressed reference. This turns external knowledge from a decoder aid into an evaluation of whether compression preserves the facts a knowledge base attaches to an image.

The second experiment is symbol and presence preservation on CLIC2020, DIV2K and Kodak, the sets the taxonomy inspected. Held-out OCR measures string agreement, a held-out detector measures person counts, and human raters label flips using the taxonomy's amplitude, geometry, shape and symbol categories. In parallel we report bits per pixel, LPIPS and FID, to confirm that perceptual quality is not traded away.

The third experiment is the cross-modal gap analysis. We compute rank correlation between CLIP-similarity drop and fact flips. A weak correlation justifies discrete checkers. A strong one would let the cheaper cosine ablation replace them.

The decisive check combines these. At matched total bitrate, the closed-loop codec must show fewer held-out and human-rated fact flips than both region-of-interest allocation and exact-string side information, with LPIPS within the range reported for the cooperative codec on Kodak. If region-of-interest allocation matches it, the central claim is wrong and the contribution reduces to the evaluation protocol. If the language-model planner ablation matches fixed checkers, multi-agent orchestration should be dropped from the method.