Seeing Red, Thinking Bad: Color Bias in Vision Language Models

Kohsuke Ide, Ryousuke Yamada, Yoshihiro Fukuhara, Hirokatsu Kataoka, Yutaka Satoh

TL;DR

Vision Language Models (VLMs) are surprisingly sensitive to the cosmetic styling of text rendered as images. By introducing "Stealth Visual Prompts," this paper demonstrates that subtle changes in text color and contrast can systematically bias a VLM's sentiment analysis and question-answering performance, revealing a critical vulnerability where low-level visual features override textual semantics.

What Problem Does This Paper Address?

As Vision Language Models (VLMs) are increasingly deployed in real-world decision-making systems—such as analyzing resumes, processing screenshots, or reading financial charts—they frequently encounter text rendered as images. While humans naturally separate the semantic meaning of text from its visual formatting (treating color or contrast as mere decoration), VLMs process both modalities simultaneously through their vision encoders.

This paper addresses a critical security and reliability gap: Are VLMs invariant to the visual form of language? The authors investigate whether subtle, semantically preserving changes to text styling—like changing the color of a word or lowering its contrast—can unintentionally manipulate a model's downstream decisions and distort its internal latent representations.

Fig. 1. Subtle visual styling biases VLM outputs. Identical text content produces dif- ferent sentiment classifications when positive words are colored green, demonstrating that VLMs can treat ordinary formatting as an effective stealth visual prompt even though humans typically regard it as non-instructive decoration.

Methods

The authors introduce Stealth Visual Prompts, a framework that applies controlled visual perturbations to text-as-image inputs while keeping the underlying text string completely unchanged. They focus on two primary axes:

  1. Color Prompts: Recoloring specific sentiment-bearing words using six canonical hues (red, green, blue, yellow, cyan, magenta) across three discrete intensity levels (subtle, mild, strong).
  2. Contrast Prompts: Reducing text-to-background contrast to evaluate how readability limitations affect model behavior. This includes Global Contrast (dimming the entire text) and Saliency Competition (rendering a task-irrelevant "decoy" word in high-contrast black while dimming the rest of the text).

To understand why these biases occur, the authors also deploy two diagnostic probes:

  • A CLIP representation probe to measure how word color shifts image embeddings along abstract semantic axes (e.g., projecting a colored word onto a "good vs. bad" valence axis).
  • A VLM-based OCR proxy to calibrate exactly when text becomes unreadable to the model under different contrast levels.

Datasets

The authors constructed the Stealth Prompt Testset, which is divided into three distinct subsets designed to probe different behavioral regimes:

  • Short-sentence Sentiment Set: 100 short sentences with interspersed positive and negative words, used to isolate word-level color bias.
  • Long-sentence Sentiment Set: 100 longer, structured sentences where positive and negative sentiments are segregated into different halves of the text to test if discourse structure overrides visual bias.
  • VQA Stealth Set: 100 question-context pairs sampled from the SQuAD dataset, modified to evaluate contrast and saliency manipulation.
Fig. 2. Examples of the generated visual stimuli. (a) A mixed-sentiment text used in the Color Axis experiment. (b) A structurally separated text. (c) A stimulus from the Saliency Competition (Contrast Axis), where the semantically incorrect decoy (“twenty- miles”) is made visually salient with high contrast.

Key Insights and Ideas

  • Color as an Implicit Control Channel: Changing the color of a word systematically shifts its representation in the vision encoder's latent space along human-interpretable semantic axes. For example, rendering words in green consistently nudges the vision encoder's embedding toward a positive valence, even if the word itself is negative.
  • Context Dictates Heuristics: VLMs switch their processing strategies based on layout complexity. In short sentences, visual color cues dominate the model's output. However, in longer, structured paragraphs, models tend to abandon color cues and rely heavily on positional heuristics like primacy (focusing on the beginning) or recency (focusing on the end).
  • Contrast Reduction Triggers Saliency Shortcuts: When text-background contrast is reduced, the model's ability to perform full-text reading degrades non-linearly. Instead of failing gracefully, the models default to a shortcut strategy, latching onto whatever text span is visually salient (high contrast), even if it is a semantically incorrect decoy.

Key Results

The experiments revealed a stark spectrum of susceptibility across open-source VLMs (including Qwen2-VL-7B, IDEFICS2-8B, LLaVA-Vicuna-7B, and LLaVA-Mistral-7B):

  • Sentiment Bias: Qwen2-VL-7B exhibited the highest sensitivity to color styling, showing a massive sentiment bias range of 0.900. Coloring positive words green or blue shifted predictions significantly upward (+0.42), while coloring negative words red caused a sharp downward shift (-0.48). IDEFICS2-8B followed with a bias range of 0.520, while LLaVA variants proved much more robust to color variations.
Fig. 3. Short-sentence Sentiment Set: average sentiment bias vs. baseline for four VLMs. The y-axis is the change in sentiment score induced by the color prompt.
  • VQA Saliency Vulnerability: In the Saliency Competition setup, reducing the contrast of the correct context text caused the Induced Error Rate (IER) to spike. For IDEFICS2-8B, the rate of copying the incorrect but highly salient decoy word rose from 24% to 36% as contrast dropped. LLaVA-Vicuna-7B saw a similar increase from 19% to 25%. Qwen2-VL-7B, however, remained highly robust here, maintaining a low error rate of 4–6%.

Limitations and Future Work

The authors note several limitations in their current study:

  • Language Scope: The stimuli were strictly limited to English text.
  • Color Space: Intensity levels were defined in RGB space rather than being perfectly matched for human perceptual distance ($\Delta E$).
  • Scope of Formatting: The study focused narrowly on color and contrast, leaving other common document formatting elements unexplored.

Future Directions: The authors advocate for expanding this evaluation framework to broader rendering factors, including font styles, complex document layouts, and multilingual scripts. Crucially, they suggest that practical VLM pipelines must implement safeguards—such as normalizing text styling before inference or cross-checking image-based answers with independent OCR text extraction—to mitigate these visual vulnerabilities.

Summary updated

Abstract

Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm
BibTeX
@inproceedings{ide2026seeing,
  author = {Kohsuke Ide and Ryousuke Yamada and Yoshihiro Fukuhara and Hirokatsu Kataoka and Yutaka Satoh},
  title = {{Seeing Red, Thinking Bad: Color Bias in Vision Language Models}},
  booktitle = {ICPR},
  year = {2026},
  doi = {10.1007/978-3-032-31583-0\_18},
}
Figure