Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit

Zhuangzi Li, Shilv Cai, Jian Jin, Zhiyi Pan, Haihong Xiao, Weisi LinarXiv 2026 · arXiv

TL;DR

Vision-Language Models (VLMs) internally "perceive" image quality much better than their zero-shot answers suggest, but this information is lost during the final "read-out" to text. By surgically editing a single neuron in the model's last MLP layer using a fast, closed-form solution, researchers can recover nearly all of this lost performance without degrading the model's general reasoning abilities.

What Problem Does This Paper Address?

While VLMs are increasingly used for zero-shot Image Quality Assessment (IQA)—asking a model if an image is "good" or "poor"—their performance often lags significantly behind human judgment. Standard zero-shot methods yield Spearman correlations (SRCC) between 0.50 and 0.81, whereas specialized, fine-tuned models exceed 0.90.

The authors ask a fundamental question: Do VLMs fail to see quality, or do they fail to say it?

They discover that the "perception" is actually there. High-quality information is decodable from the vision encoder and internal language layers. The bottleneck is the "read-out" gap—the final step where internal states are mapped to specific tokens like "good" or "poor." This read-out is often biased; for instance, models tend to over-reward colorfulness and under-reward sharpness compared to human observers.

Methods

The researchers employ a two-step approach: Locate and Edit.

  1. Layer-wise Probing: They use linear ridge probes to decode human Mean Opinion Scores (MOS) from the internal states of five different VLMs (including Qwen3.5 and Gemma 4). This confirms that the vision encoder "sees" quality at a level comparable to fine-tuned models (SRCC ~0.90).
  2. Multi-Prompt Edit (ME): Instead of expensive fine-tuning of millions of parameters, they propose a surgical intervention. They identify a single neuron in the last MLP layer that is "cheap" to overwrite (it has a nearly constant activation across different inputs).
  3. Closed-Form Edit: They derive a closed-form solution to rewrite the weights of this single neuron. This edit is fitted to 12 different prompts simultaneously using a small set of labeled images (~2,000). Unlike gradient descent, this is a direct mathematical calculation that takes only a few seconds.
The "read-out gap" and the proposed single-neuron edit.

Datasets

The study utilizes a wide array of IQA benchmarks:

  • Authentic Distortions: KonIQ-10k (primary fitting/testing), LIVE Challenge, BID, SPAQ, and FLIVE.
  • Synthetic Distortions: KADID-10k.
  • Specialized Domains: AGIQA-3K (AI-generated images) and R4-CGQA (computer graphics).
  • General Capabilities: LLVisionQA (to ensure the edit doesn't break general perception).

Key Insights and Ideas

  • The Read-out is the Bottleneck: Figure 3 shows that while internal "vision blocks" contain high-quality information, the SRCC drops sharply at the final "answer" stage. The model "knows" the quality but can't express it through the standard logit-comparison method.
  • Inherent Biases: The authors found that VLMs are "distracted" by aesthetics. They over-reward colorfulness and are relatively indifferent to sharpness, which explains much of the gap with human MOS.
  • Surgical Precision vs. Brute Force: Traditional fine-tuning often leads to "catastrophic forgetting" or degrades the model's ability to answer basic factual questions. This single-neuron edit acts as a "patch" that improves IQA while leaving the rest of the model's "brain" intact.
  • The Gap is Predictable: The improvement gained by the edit is almost perfectly predicted by how much information a linear probe can extract from the last layer. If a probe can't find the information, the edit won't either.
Probing results showing that quality information is present internally but lost at the output.

Key Results

  • Performance Boost: The Multi-Prompt Edit (ME) consistently improves zero-shot IQA. On the KonIQ dataset, it raised SRCC by +0.10 to +0.22 across different models.
  • Information Recovery: The edit recovers 86–100% of the performance gap identified by the linear probes.
  • Preserving General Intelligence: Unlike unconstrained fine-tuning, which can drop LLVisionQA accuracy by 4-44 points, the ME intervention keeps it within 0.3 points of the base model.
  • Efficiency: The edit is computed in roughly 2 seconds once internal states are cached, making it orders of magnitude faster than traditional training.
Comparison of ME vs. full fine-tuning, showing ME preserves general QA accuracy.

Limitations and Future Work

  • Prompt Specificity: The edit is designed for prompts ending in adjectives (e.g., "The quality is [good/poor]"). It does not automatically improve performance on yes/no questions or numerical ratings without refitting.
  • Domain Transfer: An edit fitted on "authentic" photos (like KonIQ) does not transfer well to AI-generated or computer-graphics images. These require their own domain-specific fits.
  • Side Effects: Because the edited neuron acts at every token, it can slightly shift the "good/poor" contrast in unrelated text tasks (e.g., sentiment analysis), though the authors introduced a "text-side constraint" to mitigate this.
  • Future Direction: The authors suggest that locating and fixing these "read-out gaps" could be a general strategy for improving VLM performance on other specialized perceptual tasks beyond just image quality.

Summary updated

Abstract

Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wise probes on five VLMs point to the second: quality is linearly decodable from the vision encoder and still from the last language layer, far better than the answer reveals, and about a hundred labels recover most of it. The loss lies mainly in the final read-out, which partly reads something else: relative to human opinion, it over-rewards colorfulness and under-rewards sharpness on all five models. We test this by intervention: if the read-out is at fault, a closed-form edit of a single last-layer neuron, fitted to many prompts at once, should close the gap and no more. Both hold. Under prompts never used for fitting, the edit recovers 86--100\% of what a ridge head on the last-layer state, fitted under the same prompts, finds beyond the answer on KonIQ. Where that gap is small, it gains little, too little for our transfer criterion on the two largest models; where the head itself is below the answer, it lowers the score; and it works only when written from the middle of the language model on. Unlike unconstrained fine-tuning, the edit leaves perception question answering intact. It does not carry over to yes-or-no or rating questions, and it reverses the good--poor contrast after sentiment-laden sentences. Code will be made available.
BibTeX
@misc{li2026lost,
  author = {Zhuangzi Li and Shilv Cai and Jian Jin and Zhiyi Pan and Haihong Xiao and Weisi Lin},
  title = {{Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit}},
  year = {2026},
  eprint = {2611.22564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22564},
}
Figure