Suppose an agent is told to "pick up the dusty rose block, not the salmon one." Today it has two ways to get a training signal for that instruction. The first is a pass/fail reward at the end of the episode, which gives almost no learning signal. The second is a learned similarity between the image and the text, usually from CLIP, a vision-language model trained on web image-caption pairs, which tends to blur fine color distinctions. We propose a third option. Represent both the color phrase and each object's perceived color as probability distributions over the same fixed palette of quantized color bins. The agent's dense reward is then the overlap between those two distributions, given only near the object it should approach.
The motivation comes from a familiar failure in image colorization. A network trained to regress a single color under ambiguity outputs the average of the plausible answers, and the average of vivid alternatives is a dull gray-brown. Treating color as classification over discrete bins fixed this in colorization (Colorful Image Colorization). We argue the same failure appears twice in color-instructed agents. Color words cover broad and sometimes split regions of color space, so mapping "brown" or "pale" to one point lands near a prototype that matches few real objects. An object's true surface color seen under unknown lighting is also ambiguous, so regressing it pulls toward neutral. In both cases a reward built from point estimates penalizes valid targets and rewards near-gray distractors. Making both sides categorical should fix both problems with one mechanism.
Core Claim
A dense reward for color-specified goals works better when it is computed as the Bhattacharyya overlap between two categorical distributions over a shared palette of quantized CIELAB chroma bins. CIELAB is a perceptual color space in which lightness is separated from the two chroma axes, called $ab$. The two distributions are a listener distribution $q(k \mid w)$ over bins $k$ for color phrase $w$, and a surface-color distribution $p(k \mid o)$ for candidate object $o$. The overlap is $\sum_k \sqrt{p(k\mid o)\,q(k\mid w)}$. This should beat both CLIP-similarity rewards and continuous point-distance rewards in Lab space. The advantage should be concentrated where it matters: distractors whose color is perceptually close to the target, color terms that cover wide or split regions of color space, and lighting conditions shifted away from training.
The overlap score is gated by a spatial focus term and given as a change in potential, which avoids the reward hacking that raw shaping invites. Here $\Phi_t = \max_o \text{BC}(p(\cdot\mid o), q(\cdot\mid w))\,g_t(o)$. $\text{BC}$ is the overlap above, and $g_t(o)$ is a Gaussian focus on object $o$, as in COPL's focal reward. The per-step reward is $r_t = \Phi_t - \Phi_{t-1}$, which is the telescoping form used by Kreis et al..
What exists and what is different
The closest work on dense rewards for grounding works in continuous space. GUI-G² replaces hit/miss click rewards with 2D Gaussian point and overlap rewards, using the same Bhattacharyya overlap we use. Its Gaussians live over screen coordinates, though, and it never handles color. COPL turns MineCLIP patch-text alignment into a focal reward. Its grounding targets entity nouns, and the authors say it struggles with attribute compositions, which is exactly the regime we target. CAREL shapes representations with contrastive auxiliary losses in BabyAI. BabyAI colors are symbolic engine IDs, so continuous color ambiguity never arises there. These three papers motivate the proposal: dense signals help a great deal, but none of them represents the attribute being grounded as a distribution.
The method comes from colorization and categorical critics. Colorization supplies the 313-bin $ab$ palette and the argument that cross-entropy over bins keeps multiple modes, from Colorful Image Colorization and CT2. CT2's luminance-dependent pruning of impossible bins also carries over directly to our surface-color head. From Start Classifying we take Gaussian-smoothed categorical targets (HL-Gauss), which replace scalar regression with a softened histogram. There it fixed the asymmetric advantages produced by MSE critics in text-only RL. We use the smoothing so that perceptually adjacent bins give partial credit, which keeps the reward graded rather than all-or-nothing.
Two results constrain the design. Pragmatically Informative Color Generation shows that regressing color modifiers from text embeddings to coordinates generalizes poorly, which supports our listener design. It does so with a point-output pragmatic speaker, however, so it is also the strongest competitor for the listener component. CCC-GRPO reports that pushing predictions toward rare tail values hurts accuracy on common ones. For us this predicts a risk that anti-desaturation measures will hurt genuinely gray or beige targets. For that reason we do not use the class rebalancing from colorization in the reward model, and we evaluate neutral-color targets separately.
Research Program
We would build and test three components.
The listener. This is a small text encoder with a 313-way softmax head. It is trained on the xkcd color survey, with each response projected onto bins using HL-Gauss smoothing, so a phrase yields a histogram rather than a point. Modifiers such as "pale" and "-ish" are handled by feeding the full phrase to the encoder. We evaluate on the seen-pairing, unseen-reference-color, and fully-unseen splits of Pragmatically Informative Color Generation. For comparability with that work, we decode the expected color in that paper's cosine-similarity and ΔE metrics. ΔE is a standard perceptual color distance.
The surface-color perceiver. This takes the pixels inside an object mask and predicts a 313-way distribution over the object's true albedo, its color independent of lighting. It is trained in simulation, where albedo is known, under randomized lighting.
The reward. The two distributions are combined as above, and an agent is trained with the result. Masks come from the simulator at first and later from an open-vocabulary detector-plus-segmenter pipeline like SegDAC, which tests robustness to imperfect segmentation.
The analytical question throughout is where gains come from. We test whether reward quality, measured as how well the reward ranks the true target above distractors, tracks the entropy of the phrase distribution and the ΔE between target and distractor. The hypothesis predicts that point-based rewards degrade fastest as entropy rises, and that CLIP rewards degrade fastest as ΔE falls.
Baselines
Sparse reward. The naive reference point is the sparse success reward alone, using the same actor-critic and step budget. Prior evidence suggests it may barely learn. COPL reports sparse baselines at 0–5% on single-task chicken hunting, against 29.5% with dense grounding.
CLIP-similarity focal reward. The strongest existing method is a COPL-style focal reward computed from patch-text similarity, using CLIP features in place of MineCLIP for the manipulation domain. It is the most direct published answer to "dense language-grounded reward without hand engineering." Its reported figure, 48.1% success on unseen MineDojo hunting targets (COPL), comes from a different domain. No reported result was found on Lacuna for color-specified manipulation with near-ΔE distractors. We would rerun it ourselves with the same masks, focus kernel, potential-difference form, agent, and tuning sweep over bandwidth and reward scale as our method. Beating it means higher success and a lower wrong-object rate in the closest ΔE bins, at equal environment steps, across seeds with non-overlapping confidence intervals.
Continuous point-distance ablation. This is the key ablation and isolates the proposal's main ingredient. The listener and perceiver are given regression heads that output single Lab chroma points, and the reward is a Gaussian of their Lab distance, with an adaptive bandwidth in the style of GUI-G². Everything else is identical. If this matches our method, quantization is not doing the work and the claim fails.
HL-Gauss smoothing ablation. A second ablation removes the HL-Gauss smoothing and uses hard bin targets. This tests whether graded partial credit across neighboring bins matters.
Oracle reward. As an upper bound, an oracle reward uses the ground-truth target pose distance. The share of the gap between CLIP and oracle that our method closes is the headline number.
Listener baseline. For the listener alone, the baseline is the pragmatic $S_2^R$ model, which reports cosine similarity .934 and ΔE 4.79 on seen pairings (Pragmatically Informative Color Generation). Matching it on point metrics while also providing a calibrated distribution is the bar.
Why not just…?
Why not just use CLIP patch-text similarity as the reward, as COPL does? This is the real threat. COPL's 4x gain over global embeddings shows that patch-level CLIP grounding is already a strong dense signal. Our bet is narrower: CLIP features merge perceptually close hues and rare color terms. COPL's own authors report weakness on attribute compositions, but that is not direct evidence about color. The experiment settles it by binning results by target-distractor ΔE. If CLIP rewards match ours in the closest bins, we would drop the perceiver and keep only the quantized listener as a re-ranker on CLIP features.
Why not just regress a point color for the phrase and use Lab distance, which is simpler and continuous? This is our key ablation. The colorization evidence argues against it: L1 still converges to the median, and regression baselines fooled humans on about 18% of Turing-test trials versus 32% for classification (Colorful Image Colorization). So does the color-modifier evidence, where regression fails on unseen colors (Pragmatically Informative Color Generation). The reward version of the claim is untested. For narrow, single-region terms like "red," we expect the point reward to be competitive, and we will report it per term-entropy bin rather than only on average.
Why not just read the object color from simulator IDs, or apply heavy color jitter for robustness? Simulator IDs are the oracle and are not available in the real world or under continuous color specifications. Jitter destroys the property the task depends on. Self-supervised Color Generalization in RL found that standard augmentations made agents unable to tell orange lava from blue rivers. We would still include a jitter-trained CLIP-reward agent in the lighting-shift test to confirm this in our setting.
Proposed Experiments
Environment. We build color-instructed variants of ManiSkill3 pick and push tasks, using the visual-generalization suite's lighting and color perturbations (SegDAC). Each scene has three to five cubes. Target colors are sampled from xkcd term regions, and distractors are placed at controlled ΔE from the target in near, medium, and far bins. One held-out set contains rare and modifier phrases, and another contains held-out lighting. All agents share SegDAC's actor-critic setup and step budget, with five seeds each.
Metrics. We measure success rate, wrong-object rate by ΔE bin, and steps needed to reach half of oracle success. For the reward itself, we report the AUROC of target-versus-distractor ranking and its calibration, measured offline on logged frames. AUROC is the area under the curve of true-positive against false-positive rate. Separately, we report neutral-color targets (grays, beiges) to check the head-tail trade-off flagged by CCC-GRPO.
Decisive checks. The proposal stands if three things hold. Our reward beats the point-distance ablation by more on high-entropy phrases than on low-entropy ones. It beats the CLIP reward mainly in the near-ΔE bin. It does not fall below either baseline on neutral targets. If the gains are uniform across bins, something other than distributional color representation is responsible, and we will say so. If CLIP matches us in the near-ΔE bin, the perceiver adds nothing, and the contribution reduces to the listener.
References
- Colorful Image Colorization
- Interactive Shaping of Granular Media Using Reinforcement Learning
- GUI-G²: Gaussian Reward Modeling for GUI Grounding
- Visual Grounding for Object-Level Generalization in Reinforcement Learning
- CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectives
- CT2: Colorization Transformer via Color Tokens
- Start Classifying: Categorical Critics for LLM Reinforcement Learning
- Pragmatically Informative Color Generation by Grounding Contextual Modifiers
- Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regression
- SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens
- Self-supervised Color Generalization in Reinforcement Learning