Intriguing Properties of Large Language and Vision Models

Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Yechan Hwang, Ho-Jin ChoiSubmission to ICLR 2025 · Rejected

TL;DR

Despite their impressive capabilities on complex multimodal reasoning tasks, popular Large Language and Vision Models (LLVMs) exhibit striking perceptual anomalies. This paper shows that LLVMs process images globally (showing permutation invariance to patch order), solve visual math questions via commonsense language shortcuts without actually reading the charts, and suffer severe catastrophic forgetting of basic visual recognition skills after multimodal alignment.

What Problem Does This Paper Address?

Most modern LLVMs (such as the LLaVA series) rely on "model stitching": combining a pre-trained Vision Transformer (ViT) and a Large Language Model (LLM) via a simple cross-modal projection layer. While these models excel at complex reasoning benchmarks, they notoriously fail at basic visual perception tasks that are trivial for humans.

This paper investigates how LLVMs actually perceive images: Do they preserve the properties of their underlying vision encoders? How do they handle spatial disruption and occlusion? Are they genuinely grounding their reasoning in visual details, or are they relying on textual and commonsense shortcuts?

Methods

The authors systematically probe LLVM representations and behaviors using several perturbation techniques:

  • Patch Permutation & Spatial Jigsaw Shuffling: Testing permutation invariance by randomly shuffling visual patch token sequences before feeding them into the LLM, as well as applying spatial jigsaw shuffling at varying grid sizes ($2\times2$, $4\times4$, $8\times8$, $14\times14$).
  • Synthetic Regeneration (Information Distortion): Generating descriptions of benchmark images using an LLVM and re-rendering them with SDXL-Lightning to strip exact numerical, chart, or formula layouts while retaining high-level semantics.
  • Occlusion Analysis (PatchDrop): Masking out patches via Random PatchDrop, Salient PatchDrop, and Non-Salient PatchDrop (guided by DINO attention maps).
  • Mutual Nearest-Neighbor (MNN) Alignment: Evaluating whether cross-modal alignment between LLMs and vision representations is preserved across instruction tuning, under the Platonic Representation Hypothesis.
  • Inference-Time "Sharpness" Importance Score: Measuring layer-wise and modality-wise importance by adding bounded uniform noise to specific token positions or layer inputs and measuring downstream performance drops.

Datasets

The authors evaluate across diverse vision-language benchmarks:

  • Multimodal Evaluation Benchmarks: MMVP, Q-Bench, MME, MMStar, MM-Vet, LLaVA-Bench (LLaVA-W), MathVista, ScienceQA-Image (SQA-IMG), ChartQA, AI2D, SEED-Bench, GSM8K (text-only 8-shot CoT).
  • Zero-Shot Classification (Alignment/Forgetting): Caltech101, CIFAR-100, Food101, Oxford-IIIT Pets, Country211, EuroSAT, FGVC-Aircraft.
  • Image-Text / Representation Alignment: DOCCI (long descriptive captions of natural scenes).
  • Captioning Benchmarks: COCO-Captions, NoCaps, TextCaps.

Key Insights and Ideas

  • LLVMs are surprisingly permutation-invariant: Shuffling the sequence of visual patch tokens causes almost no performance drop on perceptual benchmarks (<1% average drop for LLaVA-1.5 across 10 benchmarks). Because backpropagation loss is derived purely from autoregressive text targets, the model processes visual tokens as a global "bag of tokens."
  • Visual math can often be solved "blind": On synthetic MathVista images where charts and formulas were scrambled or removed, models still achieved strong scores. A major reason: many questions ask "What is the smallest value?", and models defaulted to answering 1 via language prior biases rather than reading the plot.
  • Catastrophic forgetting of basic vision: Stitching a vision encoder to an LLM severely degrades raw image classification abilities. Both LLaVA-1.5 and LLaVA-NeXT drop by nearly 30% on zero-shot image classification compared to their standalone CLIP vision encoders. Cross-modal representation alignment to natural scenes (DOCCI) also drops after visual instruction tuning.
  • Lower layers do the visual work: Layer-importance scoring shows that the bottom 20–25% of LLM transformer blocks (< layer 8) are dominated by visual processing and have the largest impact on downstream multimodal accuracy. Higher layers gradually shift focus almost entirely to text generation.
  • Strong center bias: Patch importance analysis reveals that LLVMs place massive weight on central image tokens, while edge patches contribute negligibly to final performance.

Key Results

  • Permutation Invariance (Table 1 & 4):
  • LLaVA-1.5: Average performance across 10 benchmarks dropped by only 0.59% under random patch token shuffling (even slightly improving on MMVP: +1.33%).
  • LLaVA-NeXT: Average drop of 2.71%.
  • LLaVA-OneVision: Showed a larger average drop of 9.40%, driven by severe degradation on text/geometry-rich tasks like ChartQA (▼35.26%) and MathVista (▼8.30%).
  • Visual Math Shortcuts (Table 2 & 6):
  • On synthesized/degraded MathVista images, LLaVA-1.5's accuracy dropped by merely 1.8% (from 24.7% to 22.9%).
  • When given free-form questions on degraded images, LLaVA-1.5 outputted the number "1" 81.0% of the time.
  • Degradation of Perceptual Representations (Table 3):
  • Zero-shot classification average accuracy dropped from 67.27% (standalone CLIP-ViT-L/336px) to 37.64% (LLaVA-1.5), a relative drop of nearly 30% (e.g., EuroSAT dropped from 58.90% to 11.72%).
  • Occlusion Robustness (Figure 6 & 12):
  • LLVMs inherit the strong occlusion robustness of ViTs; on AI2D, models retained high task accuracy even with 50%–70% of patches masked out.

Limitations and Future Work

  • Model Coverage: The primary focus was on the LLaVA family (with preliminary extensions to Qwen2-VL and Fuyu); broader architectures remain to be analyzed.
  • Evaluation Bias: Many benchmarks rely on multiple-choice formats where language shortcuts and selection bias obscure genuine perceptual failures.
  • Future Directions:
  • Build multi-turn, interactive, and strictly vision-grounded evaluation benchmarks with free-form answers to penalize blind guessing.
  • Mitigate catastrophic forgetting via balanced rehearsal or localized-aware projection modules (e.g., convolution-based projectors like HoneyBee).
  • Train lower LLM layers with targeted visual objectives (e.g., masked autoencoding or layer-traversal architectures like TroL) to preserve fine-grained cross-modal grounding.

Summary updated

Abstract

Recently, large language and vision models (LLVMs) have received significant attention and development efforts due to their remarkable generalization performance across a wide range of tasks requiring perception and cognitive abilities. A key factor behind their success is their simple architecture, which consists of a vision encoder, a projector, and a large language model (LLM). Despite their achievements in advanced reasoning tasks, their performance on fundamental perception-related tasks (\eg MMVP) remains surprisingly low. This discrepancy raises the question of how LLVMs truly perceive images and exploit the advantages of the vision encoder. To address this, we systematically investigate this question regarding several aspects: \textit{permutation invariance}, \textit{robustness}, \textit{synthetic data}, \textit{alignment preserving} and \textit{importance}, by evaluating the most common LLVM's families (\ie LLaVA) across 13 evaluation benchmarks. Our extensive experiments reveal several intriguing properties of current LLVMs: (1) they internally process the image in a global manner, even whenthe order of visual patch sequences is randomly permuted; (2) they are sometimes able to solve math problems without fully perceiving detailed numerical information; (3) the cross-modal alignment is overfitted to complex reasoning tasks, thereby, causing them to lose some of the original perceptual capabilities of their vision encoder; (4) the representation space in the lower layers ($<25\%$) plays a crucial role in determining performance and enhancing visual understanding. Lastly, based on the above observations, we suggest potential future directions for building better LLVMs and constructing more challenging evaluation benchmarks.
BibTeX
@misc{lee2024intriguing,
  author = {Young-Jun Lee and Byungsoo Ko and Han-Gyu Kim and Yechan Hwang and Ho-Jin Choi},
  title = {{Intriguing Properties of Large Language and Vision Models}},
  year = {2024},
  eprint = {2410.04751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.04751},
}
Figure