TL;DR
Vision Language Models (VLMs) are surprisingly sensitive to the cosmetic styling of text rendered as images. By introducing "Stealth Visual Prompts," this paper demonstrates that subtle changes in text color and contrast can systematically bias a VLM's sentiment analysis and question-answering performance, revealing a critical vulnerability where low-level visual features override textual semantics.
What Problem Does This Paper Address?
As Vision Language Models (VLMs) are increasingly deployed in real-world decision-making systems—such as analyzing resumes, processing screenshots, or reading financial charts—they frequently encounter text rendered as images. While humans naturally separate the semantic meaning of text from its visual formatting (treating color or contrast as mere decoration), VLMs process both modalities simultaneously through their vision encoders.
This paper addresses a critical security and reliability gap: Are VLMs invariant to the visual form of language? The authors investigate whether subtle, semantically preserving changes to text styling—like changing the color of a word or lowering its contrast—can unintentionally manipulate a model's downstream decisions and distort its internal latent representations.
Methods
The authors introduce Stealth Visual Prompts, a framework that applies controlled visual perturbations to text-as-image inputs while keeping the underlying text string completely unchanged. They focus on two primary axes:
- Color Prompts: Recoloring specific sentiment-bearing words using six canonical hues (red, green, blue, yellow, cyan, magenta) across three discrete intensity levels (subtle, mild, strong).
- Contrast Prompts: Reducing text-to-background contrast to evaluate how readability limitations affect model behavior. This includes Global Contrast (dimming the entire text) and Saliency Competition (rendering a task-irrelevant "decoy" word in high-contrast black while dimming the rest of the text).
To understand why these biases occur, the authors also deploy two diagnostic probes:
- A CLIP representation probe to measure how word color shifts image embeddings along abstract semantic axes (e.g., projecting a colored word onto a "good vs. bad" valence axis).
- A VLM-based OCR proxy to calibrate exactly when text becomes unreadable to the model under different contrast levels.
Datasets
The authors constructed the Stealth Prompt Testset, which is divided into three distinct subsets designed to probe different behavioral regimes:
- Short-sentence Sentiment Set: 100 short sentences with interspersed positive and negative words, used to isolate word-level color bias.
- Long-sentence Sentiment Set: 100 longer, structured sentences where positive and negative sentiments are segregated into different halves of the text to test if discourse structure overrides visual bias.
- VQA Stealth Set: 100 question-context pairs sampled from the SQuAD dataset, modified to evaluate contrast and saliency manipulation.
Key Insights and Ideas
- Color as an Implicit Control Channel: Changing the color of a word systematically shifts its representation in the vision encoder's latent space along human-interpretable semantic axes. For example, rendering words in green consistently nudges the vision encoder's embedding toward a positive valence, even if the word itself is negative.
- Context Dictates Heuristics: VLMs switch their processing strategies based on layout complexity. In short sentences, visual color cues dominate the model's output. However, in longer, structured paragraphs, models tend to abandon color cues and rely heavily on positional heuristics like primacy (focusing on the beginning) or recency (focusing on the end).
- Contrast Reduction Triggers Saliency Shortcuts: When text-background contrast is reduced, the model's ability to perform full-text reading degrades non-linearly. Instead of failing gracefully, the models default to a shortcut strategy, latching onto whatever text span is visually salient (high contrast), even if it is a semantically incorrect decoy.
Key Results
The experiments revealed a stark spectrum of susceptibility across open-source VLMs (including Qwen2-VL-7B, IDEFICS2-8B, LLaVA-Vicuna-7B, and LLaVA-Mistral-7B):
- Sentiment Bias: Qwen2-VL-7B exhibited the highest sensitivity to color styling, showing a massive sentiment bias range of 0.900. Coloring positive words green or blue shifted predictions significantly upward (+0.42), while coloring negative words red caused a sharp downward shift (-0.48). IDEFICS2-8B followed with a bias range of 0.520, while LLaVA variants proved much more robust to color variations.
- VQA Saliency Vulnerability: In the Saliency Competition setup, reducing the contrast of the correct context text caused the Induced Error Rate (IER) to spike. For IDEFICS2-8B, the rate of copying the incorrect but highly salient decoy word rose from 24% to 36% as contrast dropped. LLaVA-Vicuna-7B saw a similar increase from 19% to 25%. Qwen2-VL-7B, however, remained highly robust here, maintaining a low error rate of 4–6%.
Limitations and Future Work
The authors note several limitations in their current study:
- Language Scope: The stimuli were strictly limited to English text.
- Color Space: Intensity levels were defined in RGB space rather than being perfectly matched for human perceptual distance ($\Delta E$).
- Scope of Formatting: The study focused narrowly on color and contrast, leaving other common document formatting elements unexplored.
Future Directions: The authors advocate for expanding this evaluation framework to broader rendering factors, including font styles, complex document layouts, and multilingual scripts. Crucially, they suggest that practical VLM pipelines must implement safeguards—such as normalizing text styling before inference or cross-checking image-based answers with independent OCR text extraction—to mitigate these visual vulnerabilities.
Summary updated