Colorizing What Can Be Looked Up: Capture-Aware Retrieval for Grayscale Photos

Turning a black-and-white photo into color is an underdetermined problem. The same gray value could have been red, green or blue, so a model has to guess. Current colorizers guess well in the statistical sense: large generative models produce vivid, believable colors. For many regions of real photographs, though, the color is not a matter of taste. A particular bird species, a named landmark, or a documented object has a color that could be looked up if the system knew what it was looking at. Our claim is that for these "knowable" regions, colorization should become a retrieval problem. The model identifies the entity from the grayscale image, fetches colored reference images and color facts for that entity from an external knowledge base (a searchable store such as Wikipedia images and Wikidata), and colors the region from that evidence. Regions with no confident match fall back to ordinary generative guessing.

The obstacle is recognition. Knowledge-based question answering has already found that identifying the entity, not reading the retrieved text, is the main bottleneck. On INFOSEEK, a benchmark of questions about rare visual entities, a retrieval pipeline rises from 18.2% to 45.6% accuracy when given the correct entity INFOSEEK. Grayscale makes recognition harder still, because color is often the feature that tells similar entities apart.

Our central mechanism is to search the knowledge base in the same domain as the query. We convert every reference image to grayscale under several models of how the original film recorded light, and match the input photo against those converted references. Candidates are ranked jointly over entity and film model. Under this scheme, the relative brightness of an object's parts becomes evidence about identity. The pattern of dark wing against pale body, or of a dark-rendered red uniform on old red-blind film, survives in the query and is preserved in the keys. The best-matching film model also tells the colorizer how to reinterpret brightness, replacing the manual mode switch used by current historical-photo colorizers.

Core Claim

Retrieval keyed on capture-matched grayscale images identifies the entities in a grayscale photo substantially better than retrieval keyed on color images or than asking a vision-language model to name them. The gain should concentrate on rare entities. Feeding the retrieved entity's reference colors into a colorizer that predicts full RGB should then reduce per-region color error on knowable regions, without degrading image quality elsewhere.

A "capture model" here means a function that maps a color image to the grayscale a given film would have recorded. We use two. The first is panchromatic, meaning standard modern luminance. The second is orthochromatic, meaning early film insensitive to red, approximated as $g_{\text{ortho}}(I) = (I_B + I_G)/2$ following Beyond Fixed Luminance.

For a detected instance $x$ in the input, the retrieval score of candidate entity $e$ under capture model $m$ is

$$ s(x, e, m) = \max_{r \in R_e} \langle \phi(x), \phi(g_m(r)) \rangle + \lambda\, s_{\text{ent}}(x, e), $$

where $R_e$ is the set of colored reference images for $e$, $g_m$ is capture model $m$, $\phi$ is an image encoder fine-tuned on grayscale inputs, $s_{\text{ent}}$ is an entity-focused relevance term in the style of EnFoRe, and $\lambda$ is a mixing weight.

Normalizing $s$ over entities and capture models yields a posterior. Its maximum over entities acts as a confidence gate on how strongly the colorizer follows retrieved evidence. Its marginal over $m$ is an automatic estimate of the film type.

Relation to Prior Work

The closest work retrieves, but only visually and only in closed pools. ColorFlow uses CLIP similarity to pull colored character patches from a comic-chapter reference pool into a diffusion colorizer, reaching FID 13.37 on ColorFlow-Bench. FID, the Fréchet Inception Distance, measures how far generated images are from real ones in a learned feature space; lower is better. ColorFlow assumes the entity already appears, in color, in a pool drawn from the same domain. It keys retrieval on color images. It never links to an encyclopedic source.

We keep the idea of conditioning on a reference image. We change what is searched: an open multi-source knowledge base in the spirit of REVEAL, which showed that a 1.4B-parameter model with such a store beats far larger closed models on knowledge tasks. We also change how the search is keyed: grayscale under matched capture models.

A second line conditions on language. TIC fuses human-written captions with luminance in a GAN. Color-Turbo in Language-based Image Colorization distills a text-conditioned diffusion colorizer to 0.33 s per image. Both assume a person supplies the color words.

Multimodal Semantic-Aware Automatic Colorization automates the prompt with BLIP captions and EfficientNet classes, reaching FID 9.799 on COCO-Stuff. These are generic recognizers with a fixed vocabulary and no lookup step. We treat these works as supplying the conditioning interface. We replace the prompt source with retrieval.

The third anchor is Beyond Fixed Luminance. It motivates and constrains the design, and it supplies the generator. It shows that colorizers which keep the input brightness fixed, working in Lab color space, cannot restore bright reds that orthochromatic film rendered dark. Its remedy is a LoRA-tuned FLUX.2-klein model that predicts full RGB. LoRA is a low-rank adapter that fine-tunes a small fraction of the weights; this one touches 0.5% of them. The model is switched manually with the prompt "colorize ortho". We adopt its RGB-output generator because exemplar transfer in Lab would inherit the same failure. We replace its manual switch with the capture-model posterior.

The entity-retrieval methods EnFoRe and EntRAG supply our reranking machinery. EntRAG reports 46.1% answer accuracy on Encyclopedic VQA (E-VQA). Neither method has been applied to generating pixels, and neither has faced grayscale queries.

Research Program

The system has four stages.

The first stage finds object instances with Mask R-CNN, as in Instance-Aware Image Colorization.

The second stage builds a knowledge base whose image keys are stored twice, once rendered panchromatically and once orthochromatically. The images come from Wikipedia and Wikidata-linked sources covering the iNaturalist and Google Landmarks entities used by E-VQA. We fine-tune the encoder $\phi$ contrastively so that a grayscale instance crop lands near its own entity's gray-rendered references.

The third stage scores candidates with the equation above. An EntRAG-style reranker adds textual entity context. A calibrated confidence comes out of this stage.

The fourth stage conditions the RGB-output FLUX LoRA colorizer on two inputs. One is the top reference image, injected through in-context attention as in ColorFlow's dual-branch design. The other is a short text of retrieved color attributes, weighted by confidence. When confidence falls below a threshold chosen on validation data, conditioning is dropped and the base generative prior takes over.

The main analysis asks where retrieval helps. We split test entities by how often they occur in the colorizer's pretraining-like data, using Wikipedia page views and image counts as a proxy. We expect the benefit to be near zero on the most common entities and largest in the tail. That is the pattern INFOSEEK reports for question answering, where a model without a knowledge base, PaLI-X, reaches only 10.8%.

A second analysis asks whether the capture-model posterior recovers film type. We test this on synthetic mixtures of the two capture models and on real glass-plate negatives.

Baselines

The naive reference point is automatic colorization with no external knowledge. We use two versions. The first is the multimodal diffusion colorizer of Multimodal Semantic-Aware Automatic Colorization, which reports FID 9.799 on COCO-Stuff against 10.86 for ControlNet. The second is the unconditioned FLUX.2-klein LoRA from Beyond Fixed Luminance, given the correct mode prompt as an oracle; it reports FID 7.91 on ImageNet in the ortho setting against 8.74 or more for DDColor and BigColor. That model is the strongest existing colorizer for our regime, because it shares our generator and already handles orthochromatic input. Any gain we show over it is therefore attributable to retrieved knowledge rather than to generator capacity.

We run it on identical test images at identical resolution and sampling steps. Beating it means lower per-instance color error on knowable regions at equal or better Hue-Invariant FID (HI-FID) from Language-based Image Colorization. HI-FID matches output saturation to ground truth before computing FID, so it measures artifacts such as color spilling across object boundaries while ignoring legitimate hue differences. Neither the FLUX paper nor any other file entry reports numbers on E-VQA-derived colorization, so no reported result was found on Lacuna for colorization on iNaturalist or Landmarks images. All comparisons there are rerun by us.

The prompting baseline asks a strong off-the-shelf vision-language model to name each instance and describe its colors. The resulting text drives Color-Turbo and, separately, our FLUX generator. This is the most natural competitor. It gets the same instance crops and the same prompt-tuning budget of a few hundred validation images.

The color-keyed retrieval baseline is ColorFlow-style retrieval. It searches the same knowledge base with an off-the-shelf CLIP encoder over color keys, and passes the top result to the same conditioning branch. A hybrid baseline has the vision-language model name the entity and then fetches that entity's reference image from our knowledge base. This isolates recognition from the rest of the pipeline.

The ablations follow the same logic. Swapping gray-rendered keys for color keys, with $\phi$ held fixed, isolates the key design. Forcing $m$ to panchromatic isolates capture matching. Removing the gate isolates fallback behavior. An oracle-entity upper bound, analogous to INFOSEEK's oracle, measures how much headroom recognition still leaves.

Why not just…?

Why not just have a large vision-language model name the object and describe its colors? This is the real threat for famous entities, and we expect it to match us on the most common frequency bins. The evidence against it is in the tail. INFOSEEK finds that models without a knowledge base collapse on rare entities (10.8% for PaLI-X). Visual Riddles reports that GPT-4 misses the relevant visual clue in 27% of cases. Grayscale input removes the color cues these models likely rely on for fine-grained species. The frequency-stratified comparison, plus the hybrid baseline, settles it. If the prompting baseline matches us across all frequency bins, the capture-aware retrieval claim fails.

Why not just rely on the generative prior of a large colorizer? Generative priors sample plausible colors; BigColor (BigColor) even samples a noise vector to diversify them. Plausible is not the same as correct for a specific bird or building. FID rewards the former and cannot detect the latter. Our per-instance color error against ground truth, on entities whose colors are fixed by identity, measures exactly this gap.

Why not just retrieve colored exemplars by visual similarity, as ColorFlow does? A grayscale query compared against color keys suffers a domain mismatch. Entity-retrieval work (EnFoRe, EntRAG) also reports that generic CLIP retrieval latches onto background style rather than entity identity. The gray-key ablation tests directly whether matching the domain matters. If gray keys fail to improve top-1 entity recall over color keys with the same encoder, the core mechanism is not supported.

Proposed Experiments

We build a knowable-color benchmark from E-VQA's iNaturalist 2021 and Google Landmarks v2 images, since these carry entity labels linked to Wikipedia. We add CUB-200-2011 species as a fine-grained bird subset. Each image is converted to grayscale under both capture models, with held-out entities never seen during encoder fine-tuning. Knowable regions are the masked instances of labeled entities. On those regions we report hue and chroma error against ground truth in CIE Lab.

Everywhere, we report HI-FID and FID for image quality. For retrieval, we report top-1 entity recall from grayscale input. For the capture-model posterior, we report film-type classification accuracy. COCO-Stuff and ImageNet runs check that the gate does no harm on scenes with few knowable entities; there FID should stay within noise of the FLUX baseline.

The decisive checks are three. The first is whether gray-rendered, capture-matched keys raise top-1 recall over color keys, especially under the orthochromatic rendering. The second is whether the per-instance color-error gain over the prompting baseline grows as entity frequency falls. The third is a 25-person study on the Minnesota Digital Library glass-plate negatives used by Beyond Fixed Luminance, in which the automatically inferred film mode matches or beats the manual "colorize ortho" switch in artifact-free rate. For entities with documented archival colors, the study also measures whether raters judge our outputs more faithful.