TL;DR
Large Language Models (LLMs) can follow directional feedback like "make it more blue," but they often struggle to calibrate the magnitude of that change, leading to overshooting or undershooting. This paper introduces ColorRef, a reference game framework that uses "receiver-calibrated feedback"—matching verbal modifiers like "a little" or "much" to a specific model's observed behavior—to significantly improve the accuracy of iterative color revisions.
What Problem Does This Paper Address?
When you ask a designer to make a color "a little darker," you aren't just giving a direction; you're implying a distance. For AI assistants to be useful in creative tasks, they must understand these vague scalar modifiers.
Current LLMs are surprisingly good at identifying the correct direction in color space (e.g., moving toward "redder" or "lighter"). However, the authors identify a "miscalibrated action" problem: a model might interpret "much darker" as a massive jump that overshoots the target, actually increasing the error. The core challenge is that phrases like "a little" or "somewhat" have no universal numeric meaning; their effect depends entirely on how a specific "receiver" (the model) typically responds to them.
Methods
The researchers study this through ColorRef, a reference game where a "Guesser" (an LLM) tries to reach a hidden target color based on feedback from a "Teacher."
The paper's primary innovation is receiver-calibrated feedback:
- Calibration Phase: They first measure how a specific model (e.g., Qwen3-14B) typically moves when told to make a color "a little," "somewhat," or "much" [direction]. This creates a model-specific "step size" profile.
- Selection Policy: When the teacher knows the target is, say, 10 units away, it doesn't just pick a random modifier. It looks at the calibration table and selects the phrase whose median observed step size is closest to that 10-unit gap.
- Evaluation: They compare this calibrated approach against a "fixed rule" (e.g., always using "a little" for small gaps and "much" for large ones) to see if tailoring the language to the model's "personality" helps.
Datasets
The study uses a massive corpus of 518,830 description–target pairs from colornames.org. These are real-world, crowd-sourced pairings of color names and HEX codes. To analyze performance across different types of language, they categorized the descriptions into four "regimes":
- Explicit grounded: Direct names like "dark lavender shoot."
- Prototype mediated: References to objects like "druidic moss."
- Compound associative: Combinations like "newest green bud."
- Abstract idiosyncratic: Subjective names like "dreamy kiss."
Key Insights and Ideas
- Direction $\neq$ Magnitude: A model can be 100% compliant with the direction of feedback while failing the task because it moves too far. "Strong" modifiers like "much" are particularly dangerous, often inducing excessive changes that increase target error.
- Calibration is Model-Specific: Different models have different "internal scales." For example, Qwen3-14B might move 76 units for "much redder," while Gemma 4 12B might move only 63 units.
- The "Informative Failure": The authors highlight that simply following instructions isn't enough for grounding. A model can "obey" the prompt "make it much darker" but still produce a "harmful revision" if its step size is uncalibrated.
- Abstract Names are Harder to Start, but Easier to Fix: Colors with abstract names (like "celestial mistress") start with much higher initial error, but they also see the largest absolute gains from feedback because there is more "room to improve."
Key Results
- Oracle Success: In the full game, three rounds of basic directional feedback (without magnitude modifiers) reduced the mean target distance from 42.33 to 22.15 $\Delta E_{76}$ (a standard measure of color difference).
- Calibration Wins: On a held-out set of 128 starting colors, using calibrated wording reduced one-step mean error significantly:
- Qwen3-14B: Error dropped from 12.36 to 6.86.
- Gemma 4 12B: Error dropped from 11.06 to 5.89.
- Concentrated Benefit: The advantage of calibration was most pronounced for large corrections (24-unit targets), where it prevented the models from making wild, overshooting guesses.
Limitations and Future Work
- Single-Axis Focus: Most of the detailed calibration experiments focused on moving along one axis at a time (e.g., just getting lighter or just getting redder). Real-world feedback is often multi-dimensional ("a bit darker and more purple").
- Fixed Vocabulary: The study only used three modifiers ("a little," "somewhat," "much"). A richer vocabulary might allow for even finer control.
- Perceptual Uniformity: While CIELAB is designed to be perceptually uniform, the authors acknowledge it isn't perfect. Human judgments of "a little darker" might not map perfectly to Euclidean distance in any mathematical color space.
- Future Directions: The authors suggest moving toward "target-free stopping" (the model deciding on its own when it's close enough) and exploring how these calibration techniques work in multi-turn, multi-axis dialogues.
Summary updated