Retrieving Metamer Sets: Grounding Vision-Language Models in What a Camera Cannot See

A color camera compresses the full spectrum of light reflected by a surface into three numbers. Many different materials therefore look identical in an ordinary photo. Linoleum looks like wood, resin looks like skin, and one dried herb looks like another. This many-to-one collapse is called metamerism. Vision-language models, the systems that answer questions about images in text, inherit the blindness and answer anyway. SFOOD reports that GPT-4o and Gemini fail or refuse on 93.5% and 98% of physical-attribute queries. On herbal medicine identification, the same benchmark shows RGB-only Top-1 accuracy of 29.28% against 70.98% with 40 spectral bands. The obvious fix is retrieval-augmented generation (RAG): fetch outside evidence and put it in the model's context. Here that would mean fetching the best-matching reference spectrum. The obvious version cannot work, though. If two materials produce the same RGB, no lookup keyed on RGB can tell them apart. A system that returns "the" best match is guessing with extra confidence.

We propose instead to retrieve the metamer set. Given a spectral library, a camera model, and a set of plausible illuminants, the system returns every library material whose spectrum would render to the observed color within a calibrated tolerance. The generator gets that whole set, with the labels and physical attributes attached. It can then do three things: answer when every candidate agrees on the answer, break ties with texture and context when the image allows it, or ask for one targeted non-RGB measurement when the candidates disagree on what the user actually asked. In short, retrieval reports what the photo cannot distinguish, and the generator decides whether that ambiguity matters for the question.

Core Claim

Our central claim is about when to spend an expensive measurement. Physics-consistent retrieval of the ambiguity set predicts which RGB queries need a spectral measurement better than the model's own learned confidence does. The reason is that a model trained on RGB has no signal telling it that two classes are metameric. It learns to be confidently wrong on exactly those pairs. The metamer set is computed from the projection itself, so it can see this.

Define the library $\mathcal{L}=\{(s_j, y_j, a_j)\}$ of reflectance spectra $s_j$ with material labels $y_j$ and attributes $a_j$. Let $P$ be the camera projection (the per-channel spectral sensitivities, integrated over wavelength), $\mathcal{E}$ a set of candidate illuminant spectra, and $c$ the observed, gain-normalized RGB of a segmented region. The metamer set is

$$M_\delta(c)=\{\, j : \min_{e\in\mathcal{E},\,g>0} \| g\,P(s_j\odot e) - c \|_2 \le \delta \,\},$$

where $g$ absorbs unknown exposure and $\delta$ is a mismatch bound. We calibrate $\delta$ on held-out paired data as a quantile of real RGB-versus-rendered residuals. This borrows the bounded-error treatment of library mismatch from Semiblind Hyperspectral Unmixing in the Presence of Spectral Library Mismatches. Retrieval compares candidates in the projected RGB space rather than against a reconstructed spectrum. This sidesteps the ill-posed inversion that Statistics of real-world hyperspectral images characterizes. It also lets illuminant uncertainty widen the set honestly instead of being assumed away.

Position Relative to Prior Work

Two systems come closest, and each retrieves a single point. MatSpectNet reconstructs a hyperspectral cube from RGB under a learned camera model, queries spectraldb with that reconstruction for roughness and specularity, and reaches 88.24% pixel accuracy on LMD. Its reconstruction picks one member of the ambiguity set, and it assumes uniform illumination. MaRI trains DINOv2 dual encoders to retrieve rendering materials from photos (54.0% Top-1 on unseen materials). It works entirely in RGB appearance space, so metameric pairs are invisible to it by construction. Our proposal reuses MatSpectNet's camera-model idea and inverts its use: we enumerate the candidates consistent with an observation rather than committing to one.

On the RAG side, Active Retrieval Augmented Generation and Corrective Retrieval Augmented Generation trigger or repair retrieval based on the generator's own confidence or a judged retrieval quality. We keep their control loop but change the trigger. A measurement is requested when the retrieved candidates disagree on the answer to the question, not when the generator feels unsure. The Agentic AI in Remote Sensing survey motivates this. It documents a "grounding gap" in which RGB-trained foundation models misread multi-band physics, and it frames retrieval as document or tool lookup rather than retrieval of physical signatures.

Other work sets the evidence and the constraints. Material Classification Using Color and NIR Images and Multimodal Material Segmentation show that extra bands do resolve ambiguity: roughly 72.5% to nearly 100% on 51 samples, and 33.7% to 42.86% mIoU on MCubeS. These results supply the "measurement" action our system chooses to invoke. RF-MatID is a cautionary result: large vision backbones did not reliably beat small models on physical signals, and accuracy fell 24% under distance shift. We therefore do not expect a bigger backbone to absorb the physics.

Research Program

We will build three components and test one hypothesis.

The first component is a calibrated renderer. From paired RGB and hyperspectral captures, we fit the camera sensitivities $P$. We assemble $\mathcal{E}$ from measured daylight and indoor illuminants, and we fit $\delta$ from real residuals. Using real residuals is how we handle the lab-versus-field drift that the unmixing work identifies.

The second component is the metamer-set retriever. It precomputes rendered colors for every library entry under every illuminant, stores them in an approximate nearest-neighbor index over RGB, and returns $M_\delta(c)$ for each segmented region, with labels and attributes attached.

The third component is the generator interface. A frozen open vision-language model receives three inputs: the image, the candidate set formatted as structured text (material name, attribute ranges, rendered-color residual), and the question. It must emit one of three outputs: an answer, an answer with stated ambiguity, or a request for a single measurement. The measurement is an NIR or full-spectrum reading of that region, simulated from held-out hyperspectral data. If the model requests a measurement, the measured spectrum shrinks the set by direct spectral matching under the same $\delta$ rule, and the model answers again.

The hypothesis concerns how answer quality trades off against measurement cost. At any fixed fraction of queries that receive a measurement, set-disagreement triggering should give higher accuracy and lower confident-error rates than triggering on model confidence. We also expect the advantage to concentrate on known metameric class pairs.

Baselines

The naive reference point is the frozen vision-language model answering from the RGB image alone, with no retrieval and no measurements. For context, SFOOD reports 93.5% failure for GPT-4o on weight estimation. We will rerun this baseline with our chosen open model on our splits so that comparisons share a model.

The two endpoints bracket the problem. The lower endpoint is a dedicated classifier fine-tuned on RGB. On herbal medicine, SFOOD reports 29.28% Top-1 for RGB. The upper endpoint is the same classifier on 40-band spectra, which SFOOD reports at 70.98% Top-1. That spectral classifier is the "measure everything" ceiling at 100% measurement cost. We report where each method sits between the two at fixed budgets.

The strongest competitor on the actual question, which is deciding when to measure, is a confidence-gated system. An RGB classifier is fine-tuned on the same training split with temperature calibration. It requests a measurement when predictive entropy exceeds a threshold, and it passes its top-$k$ labels to the same generator. This is the natural way practitioners would build selective measurement. To make the comparison fair, it gets the same labeled data used to build our library, the same generator, the same measurement oracle, and a matched hyperparameter search budget, with thresholds swept to trace full budget curves. We beat it only if our accuracy at matched measurement fractions of 10%, 20%, and 30% is higher with confidence intervals that exclude zero. Beating it at a single operating point does not count.

The second strong competitor is reconstruct-then-retrieve. A MatSpectNet-style physically constrained reconstruction network, trained on ARAD_1K and SFOOD pairs, produces a spectrum, and its nearest library neighbors are handed to the generator. This tests whether enumerating the set beats committing to one reconstructed point. For MaRI-style RGB embedding retrieval we found no reported result on Lacuna for any spectral or SFOOD setting, so we will retrain it on our splits as a further competitor.

Two ablations isolate the key ingredient. In the first, the generator receives the metamer set but measurements are disabled. This separates the value of reporting ambiguity from the value of measuring. In the second, the illuminant set $\mathcal{E}$ is collapsed to a single known illuminant and $\delta$ is set to zero. This tests whether physics-calibrated tolerance, rather than plain nearest neighbors, is doing the work.

Why not just…?

Why not just fine-tune a multimodal model end-to-end on paired RGB and spectral data? This works where spectra are available at test time, which is our ceiling baseline. It cannot help when only RGB is available, because the missing information is not in the input. A model fine-tuned on RGB can only learn statistical correlates of texture and context. SFOOD's Swin Transformer drops from about 83% on ImageNet to about 48% on SFOOD, and RF-MatID found large backbones gave no consistent gains on physical signals. The fine-tuned RGB model is still worth running, and it is exactly what the confidence-gated baseline contains, so the experiments settle whether its learned uncertainty already captures metamerism.

Why not just use the classifier's own uncertainty to decide when to measure? This is the real threat. If calibrated entropy finds the metameric queries as well as set disagreement does, our retriever adds nothing beyond a nicely formatted top-$k$. Our reason to expect otherwise is mechanistic. Cross-entropy training on metameric pairs pushes the model toward whichever class texture or context favors, often confidently, and nothing in the labels flags the pair as indistinguishable. The decisive check below tests this directly. If entropy wins, we will report that, along with the narrower conclusion it supports.

Why not just reconstruct the spectrum from RGB and match it? Reconstruction returns one point from an ambiguity set it cannot see. It also inherits illumination assumptions that the color-constancy literature, for example Single Pixel Spectral Color Constancy, shows are fragile. Even MatSpectNet needed outside physical data on top of its reconstruction. The reconstruct-then-retrieve baseline tests this alternative directly. We especially expect it to be confidently wrong under illuminant shift.

Why not just always capture NIR? When the sensor is free, we agree and the problem disappears. The setting we target is one where measurement is costly: a handheld probe, a second capture, a lab assay, or a remote request. There the useful output is the accuracy-per-measurement curve, not accuracy alone. NIR also does not resolve everything. Broadband scattering reaches only 54.13% on white materials in Spectral Subsurface Scattering for Material Classification. The metamer set can say which measurement would split the candidates, rather than assuming one band suffices.

Proposed Experiments and Decisive Checks

The primary testbed is SFOOD. Its paired RGB, 40-band hyperspectral data (400–1020 nm), and Brix and weight measurements support both class questions and physical-attribute questions. We build the library from training-split spectra and query with real RGB from held-out samples. A second condition renders RGB from held-out spectra under unseen illuminants. Because we can compute exactly which test items are true metamers of other classes under the known camera, that condition gives ground truth for the ambiguity itself.

A secondary testbed is MCubeS segmentation. There the "measurement" is revealing the NIR and polarization channels for a selected region, with the reported 42.86% versus 33.7% mIoU as endpoints. The 51-sample color-and-NIR database serves as a small sanity check where the confusions are already known.

Metrics are as follows. The main one is accuracy (or attribute error) as a function of measurement fraction. The second is the confident-error rate, meaning wrong answers given without stated ambiguity. The third is the AUROC for detecting queries that RGB-only methods get wrong. For attributes, we also report whether the stated range covers the true Brix value.

The decisive check is a head-to-head on the ground-truth metameric subset. If the area under the accuracy-versus-measurement curve for set-disagreement triggering does not exceed that of calibrated-entropy triggering there, the core claim fails. A second check is a regression of RGB-only error on both metamer-set size and classifier entropy. Set size must explain variance that entropy does not. A third check sweeps $\delta$ and $\mathcal{E}$. If performance is flat across realistic tolerances, physics calibration is not doing the work, and the method reduces to color-space nearest neighbors.