TL;DR
This paper investigates how well current Vision-Language Large Language Models (VLLMs) can mimic human variability in describing and naming visual objects. While VLLMs show some promise in naming common and novel objects, they significantly struggle with tasks requiring nuanced quantifier usage, suggesting a gap in their high-level reasoning and counting abilities.
What Problem Does This Paper Address?
Humans exhibit a fascinating variability when describing the same object. This isn't random; it's driven by pragmatic constraints and subjective preferences, leading to a distribution of plausible labels rather than a single "correct" one. For instance, while "mammal" is technically correct for a dog, "dog" is pragmatically more likely. This variability is even more pronounced for uncommon or novel objects where a clear label might be missing. This paper asks: can current VLLMs, which are rapidly advancing in visual understanding and generation, capture this crucial aspect of human language use? Specifically, they explore if VLLMs can replicate the distribution of plausible labels humans produce when naming objects and using quantifiers.
Methods
The researchers evaluated three prominent VLLMs (FROMAGE, BLIP-2, and LLaVA 1.5) in a zero-shot setting. To mimic human variability, they used nucleus sampling to generate multiple outputs for each image, simulating different "speakers." The models' outputs were then compared to human responses using metrics like Jensen-Shannon divergence and Pearson's correlation. They focused on three distinct tasks:
- Naming common and novel objects: Analyzing the distribution of adjectives related to color and texture.
- Naming objects with fuzzy labels: Examining the distribution of nouns used to describe objects.
- Quantifier usage: Assessing how models assign quantifiers (e.g., "few," "some," "many") to visual scenes.
Datasets
The study utilized three datasets:
- NOUN (Novel Object and Unusual Name): Contains images of uncommon, multipart, and multicolored objects. Participants were asked to describe them, and the dataset was used to analyze the frequency of color and texture attributes in human descriptions.
- ManyNames: A large dataset where annotators provided names for objects in real-world images. This dataset was used to study the distribution of nouns used to describe objects.
- QUANT: A dataset of abstract visual scenes with varying proportions of animals and artifacts. Human participants answered "How many of the objects are animals?" by selecting from a list of nine quantifiers. This dataset was used to evaluate the models' ability to use quantifiers.
Key Insights and Ideas
- Human variability is pragmatic: The paper emphasizes that human label distributions aren't arbitrary but are shaped by context and subjective preferences, a nuance VLLMs might struggle to grasp.
- Quantifiers are a hard problem: The most striking finding is the consistent failure of all tested VLLMs to accurately mimic human quantifier usage, even when they perform moderately well on naming tasks.
- Counting is likely the bottleneck: The authors hypothesize that the poor performance on quantifiers stems from a fundamental weakness in the models' ability to accurately count objects in an image.
- Model biases exist: The analysis revealed that different models exhibit distinct biases in their quantifier selection, rather than a uniform failure. For example, one model favored "many," another "none"/"all," and a third "some."
Key Results
- Object Naming: VLLMs showed moderate success in mimicking human distributions for object names and color attributes. LLaVA generally outperformed BLIP-2 and FROMAGE in these tasks, achieving correlations around 0.5.
- Texture Attributes: Models struggled significantly with texture attributes, with no statistically significant correlation found for texture saliency.
- Quantifier Usage: All models failed to mimic human distributions for quantifiers, showing very low correlation. This indicates a severe limitation in their ability to perform this task.
- Qualitative Findings: Manual inspection revealed that models often generated plausible-sounding but incorrect descriptions, especially for texture. In the QUANT task, models frequently failed to count animals correctly, often outputting numbers or the quantifier "many" even when the count was different.
Limitations and Future Work
The authors acknowledge several limitations:
- Prompt Complexity: The multiple-choice format for the QUANT task might be more challenging than prompts for other tasks. While they explored prompt variations, the core difficulty likely lies elsewhere.
- Aggregated Human Data: The study aggregates human variability, and it's acknowledged that a single model might not perfectly replicate such rich, multi-speaker variation. The focus is on understanding the extent to which VLLMs can mimic this feature.
- Underestimation of Saliency: For the NOUN dataset, the string-matching approach for color and texture saliency might underestimate the models' performance.
Future work could explore more sophisticated prompting strategies for quantifiers and delve deeper into the "counting" capabilities of VLLMs. The study also highlights the potential of datasets like NOUN and ManyNames for future research on VLLM grounding and human-like language generation.
Summary updated