What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu, Yi-Zhe SongarXiv 2026 · arXiv

TL;DR

This paper uses sketches as a "structural probe" to uncover how Large Vision-Language Models (LVLMs) represent objects. The researchers found that LVLMs internalize human drawing conventions (like drawing a cup's body before its handle) within their language models and that part-level grounding emerges naturally in specific, predictable "mid-network bands" of the architecture without any explicit training.

What Problem Does This Paper Address?

Standard benchmarks for LVLMs typically measure output accuracy—did the model correctly identify the object in the photo? However, natural photographs are "noisy" probes; they contain texture, lighting, and background context that models can use as shortcuts. This makes it hard to tell if a model actually understands the underlying structure of an object or is just pattern-matching pixels.

The authors argue that sketches are a superior probe because they strip away everything except the essential structural strokes humans use for recognition. By testing models on sketches, they aim to answer:

  1. What visual conventions (like drawing order) have models retained?
  2. Can models "ground" (locate) specific parts of an object without being explicitly trained to do so?
  3. Where, mechanistically, does this structural knowledge live within the billions of parameters of a transformer network?

Methods

The authors introduce a training-free probing framework with three distinct "readouts":

  1. Temporal Probing: The model is given a sketch and a shuffled list of its parts (e.g., "handle," "body," "rim"). It is asked to reorder them based on the sequence in which a human likely drew them.
  2. Spatial Probing: Instead of using a dedicated segmentation model, the authors extract cross-attention weights from the frozen LVLM to see if the model's internal "gaze" naturally falls on the correct part of the sketch when prompted (e.g., "Where is the handle?").
  3. Mechanistic Probing: They track which specific layers and attention heads carry the grounding signal to see if there is a consistent "locus" of structural knowledge.
The three-perspective probing framework: temporal, spatial, and mechanistic.

Datasets

  • SPG (Sketch-Part-Grouping): A dataset of object sketches with part-level annotations and, crucially, the temporal order of every stroke.
  • FS-COCO: A dataset of scene-level sketches used to test if grounding holds up in complex environments.
  • Pascal-Part & ControlSketch: Used in a pilot study to observe how model performance degrades as images move from natural photos to increasingly abstract sketches.

Key Insights and Ideas

  • Conventions Live in the Language Model: Surprisingly, the model's ability to predict drawing order doesn't depend much on the image. Even when shown a blank canvas, models still predicted the "canonical" human drawing order. This suggests that the "visual" convention is actually stored as linguistic knowledge about human behavior.
  • Grounding is an Emergent Property: Frozen, off-the-shelf LVLMs can locate object parts in abstract sketches better than some models specifically trained for grounding (like CLIPSeg). This "zero-shot" grounding survives even when texture and color are completely removed.
  • The "Mid-Network Band": Grounding isn't scattered randomly. It concentrates in a narrow band of layers. Interestingly, the location of this band depends on the LLM family, not the model size. For Vicuna-based models (like LLaVA), it's at fractional depth 0.35–0.45; for DeepSeek-based models, it's deeper, at 0.50–0.65.
Figure 1: Sketches as a probe for LVLMs. Sketches keep only the strokes humans considered essential, exposing what an LVLM treats as essential. We probe this from three complementary perspectives. Left: temporal part-construction order on two object sketches (early-to-late gradient, predicted ordering beneath) against human consensus. Right: spatial attention-based grounding on one object and one scene sketch, with per-part / per-object overlays beside the human sketch.

Key Results

  • Superior Grounding: The strongest probe, LLaVA-1.5-13B, achieved a mean Accuracy ([email protected]) of 0.73 on object sketches, significantly outperforming trained grounders like GroupViT (0.59).
  • Predictive Layer Bands: By restricting the search for grounding information to the identified "mid-network band," the authors improved grounding accuracy across every model tested. For example, LLaVA-1.5-7B saw an improvement from 0.65 to 0.73 [email protected].
  • Human Alignment: Model agreement with human drawing order correlates strongly with human consensus (Pearson $r = +0.58$). When humans agree on how to draw something (like an airplane), the model agrees too; when human conventions vary (like drawing a pig), the model's predictions become equally variable.

Limitations and Future Work

  • The "Phantom Part" Problem: The current probe assumes the queried part exists. If asked to find the "wings of a coffee cup," the model will still highlight a region (usually the handle) rather than signaling that the part is missing.
  • Architectural Scope: The study focused on two major LLM families (Vicuna and DeepSeek). Future work is needed to see if these "grounding bands" exist in other architectures like Mistral or Llama-3.
  • Static vs. Dynamic: The temporal probe uses the final sketch. A more advanced probe might look at the model's internal state during the construction of a sketch.

Summary updated

Abstract

LVLM benchmarks tell us whether a model answers correctly, but not which visual conventions it has retained, nor where those conventions live. Sketches expose this hidden structure. A sketch removes texture, lighting, and photographic context while preserving the strokes humans judged sufficient for recognition. If an LVLM can operate on that abstraction, the surviving signal is structural rather than merely photometric. We introduce a training-free sketch probe with three readouts: temporal part-construction order, spatial attention-based grounding, and the mechanistic layer/head locus of grounding. We evaluate nine LVLMs on part order and five on the attention readouts, over two part-annotated sketch datasets. Three findings emerge. First, LVLM part order tracks human convention: models reproduce canonical orderings where human drawings converge, and become correspondingly variable where human orderings vary. Controls that blank, withhold, and mismatch the sketch locate this convention in the language model. Second, part-level grounding survives the abstraction: attention heads selected from the frozen model with an existing recipe recover part and object regions on object and scene sketches, with no grounding supervision. Third, the grounding signal concentrates in a narrow mid-network band whose location follows the base language model rather than parameter count in the two families we compare, and remains stable across object and scene sketches. The result is a practical diagnostic for sketch-driven LVLMs: no fine-tuning, no auxiliary segmenter, and a layer band that doubles as a head-selection prior, matching or improving grounding [email protected] in every model and dataset we tested.
BibTeX
@misc{utintu2026what,
  author = {Chaitat Utintu and Ahmed Bourouis and Arkaprabha Basu and Yi-Zhe Song},
  title = {{What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation}},
  year = {2026},
  eprint = {2610.34037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34037},
}
Figure