Would you still call this Dax? Novel Visual References in VLMs and Humans

Ada Defne Tür, Gaurav Kamath, Joyce Chai, Siva Reddy, Benno KrojerarXiv 2026 · arXiv

TL;DR

This paper introduces the Novel Visual References Dataset (NVRD), a benchmark of over 19,000 images designed to evaluate how vision-language models (VLMs) learn and generalize new visual concepts in-context. The study reveals that while VLMs share human-like structural biases (favoring shape over texture or color), they struggle significantly with "mutual exclusivity" when novel labels contradict prior knowledge, and they drastically overgeneralize compared to humans.

What Problem Does This Paper Address?

When human children encounter a new word—like being shown a strange object and told "This is a dax"—they employ powerful cognitive shortcuts to understand what it means. They use the shape bias (assuming other things shaped like it are also "daxes", regardless of color or texture) and mutual exclusivity (assuming an object only has one primary name; if they already know the word "chair," they resist calling it a "dax").

Modern Vision-Language Models (VLMs) routinely face novel objects at inference time. However, we don't yet fully understand how they map these new visual references to language, especially when the new concepts conflict with their massive pre-training data. This paper investigates whether VLMs possess human-like inductive biases when learning new visual concepts in-context, and how their generalization boundaries compare to human judgments.

Methods

The authors evaluate three open-source VLMs (Qwen-2 VL 7B, Idefics-3 8B, Molmo-2 8B) and two closed-source models (GPT-4o Mini, Gemini-2.5 Flash) across three distinct prompting paradigms:

  1. Name Generation from Multi-Image ICL: Models are given an in-context pool of images (the novel object with its new name, plus distractor images) and must fill in the blank for a perturbed version of the object.
  2. Token Probabilities: The open-source models are evaluated on the exact log probabilities they assign to the new "nonce" word versus its familiar "vanilla" name.
  3. Dual-Image Likert-Scale Rating: Models see an original image and a perturbed variant side-by-side and rate their agreement (1 to 7) on whether both can share the same name.

To ground these findings, the authors collected 2,400 human judgments via a crowd-sourced study on Prolific using the exact same dual-image Likert-scale setup.

Figure 1: Task setup overview: On the left-hand side are examples of visual comparisons and nonce references evaluated; on the right-hand side, models and humans rate agreement with a novel label.

Datasets

The authors introduced the Novel Visual References Dataset (NVRD), which consists of 19,176 unique images built around 90 base visual concepts. The dataset categorizes objects into three distinct novelty tiers:

  • Known Entities: Common objects (e.g., a chair) given a completely fake name (e.g., "blomwich") to test if models can override prior knowledge.
  • Composed Entities: Hybrids combining existing objects or textures (e.g., a boar-toaster hybrid or a lion with bird feathers).
  • Fully Novel Entities: Abstract objects generated entirely from scratch using complex procedural text prompts detailing unique silhouettes, materials, and structural rules.

Each base object is subjected to 11 different types of controlled visual perturbations across up to 20 levels of increasing intensity. These include low-level corruptions (Gaussian noise, pixelation, color shifts) and high-level generative edits (shape deformation, texture shifts, background replacement, and part addition/removal).

Key Insights and Ideas

  • Prior Knowledge Blocks New Learning: VLMs exhibit a strong version of human mutual exclusivity. When shown a standard chair and told to call it a "blomwich," models (especially GPT-4o Mini) strongly resist and default back to their pre-trained labels. They acquire new words much more easily when the object is genuinely novel and lacks a competing pre-trained label.
  • VLMs Have a Shape Bias: Just like humans, VLMs treat structural changes (deforming the shape, removing or adding limbs) as highly destructive to an object's identity. Conversely, surface-level changes like changing the color, background, or adding noise barely impact the model's willingness to extend the label.
  • The Overgeneralization Gap: While humans and models agree on what matters (shape over color), they disagree entirely on boundaries. Humans are highly sensitive to degradation; once an object loses its parts or shape, humans reject the label. VLMs, however, are far more permissive, routinely extending novel labels to heavily mutated objects that humans completely reject.

Key Results

  • Human-Model Correlation: Overall, closed-source models aligned remarkably well with human sensitivity trends across perturbation types. GPT-4o Mini achieved the highest overall Spearman correlation with humans ($\rho = 0.915$), followed closely by Gemini 2.5 Flash ($\rho = 0.905$).
  • Structural Sensitivity: For shape-based perturbations (like part removal), human ratings plummeted below 3 (Somewhat Disagree) by perturbation level 10. In contrast, most open-source VLMs remained highly confident, hovering between 4 (Neutral) and 6 (Agree).
  • Adversarial Sycophancy: In an ablation test where models were shown two completely different objects and asked if they shared the same name, Qwen2-VL agreed up to 30% of the time, demonstrating a tendency to over-accept labels when prompted in-context, particularly for fully novel objects.

Limitations and Future Work

The authors note a few key limitations in their benchmark generation pipeline:

  • Generator Artifacts: Because NVRD relies on generative models (Gemini) to create the perturbations, the images inherently contain the structural and material biases of the underlying generator.
  • Noise in Edits: Manual validation revealed that roughly 18% of the high-level generative edits were "noisy"—meaning the model accidentally changed an unintended attribute (e.g., adding a part when it was only asked to change the texture).
  • Future Directions: The authors suggest that future VLM architectures need to move beyond tracking pure visual similarity and instead incorporate an "essentialist" framework of categorization, mimicking how humans treat established categories as bounded by deeper structural properties.

Summary updated

Abstract

Vision-language models (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human-model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.
BibTeX
@misc{tur2026would,
  author = {Ada Defne Tür and Gaurav Kamath and Joyce Chai and Siva Reddy and Benno Krojer},
  title = {{Would you still call this Dax? Novel Visual References in VLMs and Humans}},
  year = {2026},
  eprint = {2606.05409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.05409},
}

Loading related content...

Figure