Where to Put the Next Gray Card: Active Probe Placement for Multi-Illuminant White Balance

Most photographs of indoor scenes are lit by more than one light, for example a warm lamp on one side and cool daylight from a window on the other. Correcting their colors requires estimating a different light color at every pixel, and training a network to do that requires ground truth that is expensive to collect. Today that means photographing each scene with calibration targets (color charts or gray cards) and, in datasets like LSMI, separate captures under each light. We propose to replace dense per-pixel labels with a few probes. A probe is one calibration reading at one chosen spot in one chosen scene, telling the model the true light color there. The model itself decides where the next probe should go.

The core claim is that two kinds of physical structure make this efficient. First, illumination fields are low-dimensional: a few light colors, blended by smooth spatial weights. One well-placed probe therefore constrains a large region of the image. Second, some spots are bad places to measure, such as specular highlights, dark corners and interreflections. These spots look uncertain to a model, but labeling them teaches it little. An acquisition rule that spreads each probe's value over the region sharing the same light mixture, and refuses to spend probes where readings are inherently noisy, should match dense supervision with far fewer measurements than random placement, a fixed grid, or labeling whole images. The motivating use case is concrete: bringing up white balance on a new camera sensor, where a team can afford a few hundred gray-card readings but not a new dense dataset.

What exists and what is missing

Multi-illuminant estimators already exploit the low-dimensional structure. Attentive Illumination Decomposition (AID) uses slot attention, a mechanism in which a fixed set of learned "slots" compete to explain distinct parts of the input. Here each slot is one light, and each outputs a chromaticity plus a per-pixel weight map. The final map is the weighted sum of slot chromaticities. AID reports a mean angular error (MAE) of 2.04 on LSMI. Pixel-wise color constancy via smoothness techniques (PWCC) regresses a log-chrominance map with a U-Net and adds total-variation smoothing. It also argues explicitly that LSMI labels are noisy because charts cannot account for local reflectance and specular effects, and it counters this with Gaussian label smoothing. PWCC reports MAE 2.0 and worst-25% error 3.8 on the Galaxy Note 20 Ultra split. Both papers train passively on dense labels. Their structural prior and their observation about label noise are exactly what our acquisition rule uses.

On the active-learning side, three papers supply the method and one constrains it.

  • Sparse labeling in dense prediction. PixelPick showed that sparse, uncertainty-chosen pixels spread across many images nearly match dense labels: 96% of full supervision on CamVid from 0.06% of pixels. Its uncertainty measure, however, is a classification margin, and its labels are categorical.
  • Active learning for dense regression. USIM-DAL does dense active regression for super-resolution. It selects whole images by predicted aleatoric variance (the noise the model expects in the data itself).
  • Choice of uncertainty estimator. MCDUE found that dropout-based uncertainty beats geometric diversity sampling for regression. Deep Evidential Regression separates epistemic uncertainty (what the model doesn't know yet) from aleatoric uncertainty in a single forward pass.
  • The constraint. ALFLTP found that picking the maximum-uncertainty samples in trajectory regression pulls in unlearnable noisy samples, and that median-uncertainty selection converges faster.

This sits in direct tension with USIM-DAL's choice to acquire by aleatoric variance. Our design takes ALFLTP's side and tests it, because PWCC's evidence says LSMI noise is concentrated at exactly the spots a naive rule would pick.

The research file reports no paper that runs active learning for multi-illuminant estimation. The closest work, USIM-DAL, differs on three counts:

  • it acquires whole images rather than locations within images;
  • it uses no physical mixing structure;
  • it treats aleatoric uncertainty as a reason to query rather than as a reason to avoid.

Research Program

Model. We use an AID-style decomposition network trained from sparse probes. The loss is the angular error between the predicted map, averaged over the probe's patch, and the probe reading, plus PWCC's total-variation term to propagate supervision smoothly. To get uncertainty we train a small deep ensemble of five such models. Ensemble disagreement in predicted chromaticity at pixel $y$, measured as angular variance, gives the epistemic term $\sigma^2_{e}(y)$. A per-pixel heteroscedastic head, trained with a Gaussian likelihood on probe residuals, gives the aleatoric term $\sigma^2_{a}(x)$. A single-pass evidential head following Amini et al. is a second, cheaper uncertainty estimator that we compare against the ensemble. Speed matters little for offline labeling, so the ensemble is the default.

Acquisition rule. A probe at candidate location $x$ is scored by how much unresolved epistemic uncertainty it shares a light mixture with, gated by how trustworthy a reading there would be:

$$a(x) = \mathbb{1}\left[\sigma^2_{a}(x) < \tau\right] \sum_{y} s(x,y)\, \sigma^2_{e}(y).$$

The terms are defined as follows:

  • $s(x,y)$ is the cosine similarity between the predicted slot-weight vectors at $x$ and $y$, so a probe "covers" pixels lit by the same mixture of lights;
  • the sum runs over all pixels of the same image;
  • $\tau$ is set at the median of $\sigma^2_{a}$ over the pool, following ALFLTP's median finding.

Batches are built greedily. After each pick, the epistemic uncertainty of the pixels it covers is down-weighted by $1-s(x,y)$. This prevents the spatial clustering that PixelPick handles with heuristics. Images and locations are selected jointly, so the probe budget naturally flows to scenes whose mixture is ambiguous.

Analyses. Beyond accuracy, we will examine:

  • where probes land: near illuminant transitions, or in flat regions;
  • whether gated-out locations are in fact the specular or dark regions where LSMI labels disagree with their neighbors;
  • whether epistemic uncertainty after training predicts held-out error, which we measure as a rank correlation per image.

Baselines

The naive reference is random probe placement: uniformly random images and locations, with the same probe budget, model, loss, ensemble size and training schedule. Every curve is reported against it.

The strongest simple competitor is a fixed spatial grid of probes per image, for example 3×3, applied to randomly chosen images. Illumination fields are smooth, so a grid may already capture most of the information. Beating random but not the grid would mean the gain comes from coverage, not from uncertainty.

The strongest existing active-learning method is USIM-DAL-style image-level acquisition. Whole images are chosen by mean predicted aleatoric variance and receive full dense LSMI labels. To make costs comparable, we sweep the probe-equivalent cost $c$ of one dense label over a range (we treat $c$ as unknown) and report the crossover cost at which sparse active probing wins.

The full-supervision ceilings are AID (MAE 2.04 on LSMI, as reported by Kim et al.) and PWCC (MAE 2.0 and worst-25% 3.8 on the Galaxy split, as reported by Entok et al.). Splits differ between those papers, so we retrain our own architecture on dense labels for every split we use and treat that retrained number as the ceiling. No reported result was found on Lacuna for any sparse-label or active-learning method on LSMI, so the active-learning curves have no published comparison point.

Three ablations isolate the proposal's ingredients:

  • Mixture coverage. Replace $s(x,y)$ with $\mathbb{1}[x=y]$, which reduces the rule to pure per-pixel epistemic sampling.
  • Aleatoric gate. Remove the gate, giving maximum-uncertainty selection.
  • Gate type. Replace the gate with "query maximum aleatoric," USIM-DAL's choice.

All arms share the same pretraining on Afifi's 150-scene synthetic benchmark, the same initial seed set, five seeds, and an equal hyperparameter-tuning budget on a validation split.

Why not just…?

Why not just place probes on a regular grid? This is the real threat, and we treat it as the primary comparison rather than a straw man. Our expectation that the active rule wins rests on two points. First, errors in multi-illuminant scenes concentrate at light transitions and shadow boundaries. Second, many LSMI scenes are close to single-light and need almost no probes. A grid spends probes uniformly in both respects. If the active rule fails to beat the grid by a clear margin at budgets below roughly one probe per image, we will conclude that the mixture structure alone, plus smoothness, does the work. The contribution would then shrink to "sparse grid probes suffice," which is still a useful negative result for dataset builders.

Why not just label whole images, as USIM-DAL does? Dense labels carry redundant information under the low-dimensional mixture prior, much as PixelPick found for segmentation. Dense labels also absorb the chart noise PWCC documents. The crossover-cost experiment answers this directly. If dense image acquisition wins even at small $c$, sparse probing is not worth it.

Why not just use generative virtual checkers instead of real probes? GCC inpaints virtual color checkers with an adapted diffusion model and can place many of them per scene at zero labeling cost. No reported result was found on Lacuna for GCC's multi-illuminant accuracy on LSMI. GCC is also limited to 512×512 resolution and requires a large GPU. We include GCC pseudo-probes at grid locations as a zero-cost arm, and as a hybrid in which our rule spends real probes only where GCC and the ensemble disagree. If pseudo-probes alone come close to the ceiling, human probes are only worth spending on the residual disagreement.

Why not just pick the most uncertain locations? ALFLTP reports that maximum-uncertainty selection pulls in noisy, unlearnable samples, and PWCC locates LSMI's label noise at specular and reflectance boundaries. The no-gate ablation tests whether this carries over to color constancy. We will also check directly whether ungated picks fall disproportionately on pixels whose labels disagree with their local median.

Why not just use diversity sampling over color histograms? MCDUE found uncertainty sampling up to 20% better than greedy max-min diversity in regression. That evidence comes from other domains, however, so we add a k-center arm over chromaticity-histogram embeddings at the image level, combined with grid placement within each image.

Proposed Experiments

We run two settings. In-sensor active learning uses each LSMI camera split separately. Cross-sensor adaptation starts from a Galaxy-trained model and adapts to Sony and then Nikon with probes only, which matches the camera bring-up use case. Probes are simulated by revealing the LSMI ground truth averaged over a chart-sized patch. A secondary run adds noise matched to the ground-truth variance within each patch, so that the aleatoric gate has realistic material to act on.

Metrics are mean, median and worst-25% angular error as a function of probe count. We also report the number of probes needed to reach within 10% of our dense-supervision ceiling. To test whether the model generalizes beyond LSMI capture conditions, we evaluate on MIIW (where AID reports MAE 2.46) and on Afifi's synthetic benchmark.

The decisive checks, in order of importance, are:

  1. active placement versus the grid at budgets of 0.25–2 probes per image;
  2. the crossover cost against dense image-level acquisition;
  3. whether removing the coverage term or the aleatoric gate erases the gain.

The proposal succeeds if the full rule reaches the dense ceiling within 10% using a small fraction of the information in dense labels, and beats both the grid and random placement across seeds, with each ablation costing measurable accuracy.