ColorRef: Calibrating Verbal Feedback for Color Revision

Pablo Loyola, Andres Hoyos-Idrobo, Edison Marrese-Taylor, Colin Rowat, Alfredo SolanoarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) can follow directional feedback like "make it more blue," but they often struggle to calibrate the magnitude of that change, leading to overshooting or undershooting. This paper introduces ColorRef, a reference game framework that uses "receiver-calibrated feedback"—matching verbal modifiers like "a little" or "much" to a specific model's observed behavior—to significantly improve the accuracy of iterative color revisions.

What Problem Does This Paper Address?

When you ask a designer to make a color "a little darker," you aren't just giving a direction; you're implying a distance. For AI assistants to be useful in creative tasks, they must understand these vague scalar modifiers.

Current LLMs are surprisingly good at identifying the correct direction in color space (e.g., moving toward "redder" or "lighter"). However, the authors identify a "miscalibrated action" problem: a model might interpret "much darker" as a massive jump that overshoots the target, actually increasing the error. The core challenge is that phrases like "a little" or "somewhat" have no universal numeric meaning; their effect depends entirely on how a specific "receiver" (the model) typically responds to them.

Methods

The researchers study this through ColorRef, a reference game where a "Guesser" (an LLM) tries to reach a hidden target color based on feedback from a "Teacher."

Figure 1: ColorRef for southern lime green (example 427046). (a) The guesser predicts from the description; the teacher knows the hidden target and gives feedback for revision. (b) Qwen3-14B with an axis oracle: point 0 is the initial prediction, points 1–3 are fixed-round revisions, and the star is the target. The a∗b∗plane shows chromatic movement; the strip shows L∗(target dashed). Lower rows give full ∆E76 and exact feedback. Both panels use the same saved game. This is an illustration, not an aggregate result; plotting details appear in Appendix C.1.

The paper's primary innovation is receiver-calibrated feedback:

  1. Calibration Phase: They first measure how a specific model (e.g., Qwen3-14B) typically moves when told to make a color "a little," "somewhat," or "much" [direction]. This creates a model-specific "step size" profile.
  2. Selection Policy: When the teacher knows the target is, say, 10 units away, it doesn't just pick a random modifier. It looks at the calibration table and selects the phrase whose median observed step size is closest to that 10-unit gap.
  3. Evaluation: They compare this calibrated approach against a "fixed rule" (e.g., always using "a little" for small gaps and "much" for large ones) to see if tailoring the language to the model's "personality" helps.

Datasets

The study uses a massive corpus of 518,830 description–target pairs from colornames.org. These are real-world, crowd-sourced pairings of color names and HEX codes. To analyze performance across different types of language, they categorized the descriptions into four "regimes":

Examples of the four semantic regimes used in the study.
  • Explicit grounded: Direct names like "dark lavender shoot."
  • Prototype mediated: References to objects like "druidic moss."
  • Compound associative: Combinations like "newest green bud."
  • Abstract idiosyncratic: Subjective names like "dreamy kiss."

Key Insights and Ideas

  • Direction $\neq$ Magnitude: A model can be 100% compliant with the direction of feedback while failing the task because it moves too far. "Strong" modifiers like "much" are particularly dangerous, often inducing excessive changes that increase target error.
  • Calibration is Model-Specific: Different models have different "internal scales." For example, Qwen3-14B might move 76 units for "much redder," while Gemma 4 12B might move only 63 units.
  • The "Informative Failure": The authors highlight that simply following instructions isn't enough for grounding. A model can "obey" the prompt "make it much darker" but still produce a "harmful revision" if its step size is uncalibrated.
  • Abstract Names are Harder to Start, but Easier to Fix: Colors with abstract names (like "celestial mistress") start with much higher initial error, but they also see the largest absolute gains from feedback because there is more "room to improve."

Key Results

  • Oracle Success: In the full game, three rounds of basic directional feedback (without magnitude modifiers) reduced the mean target distance from 42.33 to 22.15 $\Delta E_{76}$ (a standard measure of color difference).
  • Calibration Wins: On a held-out set of 128 starting colors, using calibrated wording reduced one-step mean error significantly:
  • Qwen3-14B: Error dropped from 12.36 to 6.86.
  • Gemma 4 12B: Error dropped from 11.06 to 5.89.
  • Concentrated Benefit: The advantage of calibration was most pronounced for large corrections (24-unit targets), where it prevented the models from making wild, overshooting guesses.

Limitations and Future Work

  • Single-Axis Focus: Most of the detailed calibration experiments focused on moving along one axis at a time (e.g., just getting lighter or just getting redder). Real-world feedback is often multi-dimensional ("a bit darker and more purple").
  • Fixed Vocabulary: The study only used three modifiers ("a little," "somewhat," "much"). A richer vocabulary might allow for even finer control.
  • Perceptual Uniformity: While CIELAB is designed to be perceptually uniform, the authors acknowledge it isn't perfect. Human judgments of "a little darker" might not map perfectly to Euclidean distance in any mathematical color space.
  • Future Directions: The authors suggest moving toward "target-free stopping" (the model deciding on its own when it's close enough) and exploring how these calibration techniques work in multi-turn, multi-axis dialogues.

Summary updated

Abstract

People rarely specify an exact color when asking someone to revise a design: they might say \textit{a little darker} or \textit{more blue} after seeing a proposal. A language-model assistant must then infer not only which way to change the color, but how far. We study this distinction with \textsc{ColorRef}, a color reference game in which a model guesses a color from a description and a teacher, knowing a recorded target hidden from the guesser, supplies verbal corrections. On 1{,}000 crowd-sourced description--target pairs, three rounds of deterministic oracle feedback reduce mean target distance from 42.33 to 22.15 $ΔE_{76}$. However, those corrections provide additional information, and improvement alone does not establish well-calibrated feedback following. A controlled Qwen3-14B diagnostic reveals a more specific failure: following a requested direction and ordering graded phrases do not guarantee a suitable step size; strong modifiers can induce excessive changes. We therefore measure each receiver's typical response to \textit{a little}, \textit{somewhat}, and \textit{much} on separate colors and select wording matched to its observed steps. On 128 held-out starting colors, calibration reduces one-step mean error from 12.36 to 6.86 for Qwen3-14B and from 11.06 to 5.89 for Gemma 4 12B against a fixed graded-wording rule. The benefit is concentrated on large corrections, with many unchanged cases and no consistent increase in threshold success. Our findings show that following the direction of feedback is not the same as making an appropriately sized revision.
BibTeX
@misc{loyola2026colorref,
  author = {Pablo Loyola and Andres Hoyos-Idrobo and Edison Marrese-Taylor and Colin Rowat and Alfredo Solano},
  title = {{ColorRef: Calibrating Verbal Feedback for Color Revision}},
  year = {2026},
  eprint = {2611.25653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25653},
}
Figure