When a human is asked to find an object or answer a question, additional context usually helps. "Find the coffee mug" is clear, but "Find the blue ceramic coffee mug on the left side of the wooden desk" is even easier to resolve. The extra descriptors narrow down the search space and eliminate ambiguity.
For state-of-the-art machine learning models, the exact opposite is often true.
Across text processing, computer vision, and multimodal retrieval, models frequently exhibit a counterintuitive vulnerability: as natural language queries grow longer, more descriptive, or syntactically complex, model performance plummets. This phenomenon—often termed the descriptive detail penalty or complex query degradation—reveals a fundamental gap between human linguistic processing and the statistical representations used by modern AI.
The Compositionality Wall
To understand why detail degrades performance, we must look at how models handle compositionality—the ability to combine individual concepts (objects, attributes, and relations) into a coherent whole.
When a query expands from a simple noun ("dog") to a detailed description ("a small brown dog sitting to the left of a red fire hydrant"), a model cannot simply recognize the individual elements. It must correctly bind the attribute "small" and "brown" to the "dog", and resolve the spatial relationship "to the left of" relative to the "fire hydrant."
As descriptions scale, models hit what researchers call a "complexity wall." In Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, Thrush et al. demonstrated that vision-language models struggle significantly with basic compositional reasoning, showing a sharp performance decline as linguistic complexity and caption length increase.
This failure mode is systematically exposed by benchmarks like CREPE. As evaluated by Ma et al., when the number of objects, attributes, and relations in a description multiplies, the accuracy of vision-language foundation models drops to near random chance. The models fail to maintain structural integrity, frequently swapping entity roles or misattributing properties because their underlying linguistic encoders struggle to synthesize complex relational information.
Syntactic Noise and Intent Obscuration
The detail penalty is not merely a consequence of models being overwhelmed by new constraints; it occurs even when the additional information is completely answer-preserving.
In Question Modifiers in Visual Question Answering, William Britton uncovered a literal "detail penalty" by augmenting standard visual questions with descriptive modifiers that did not alter the ground-truth answer. Paradoxically, adding these superfluous descriptors caused state-of-the-art models to perform significantly worse. The increased syntactic complexity hindered the models' ability to filter out irrelevant information, distracting them from the core visual features needed to answer the question.
This vulnerability extends to large language models (LLMs). As Liang et al. demonstrate, increasing the lexical diversity and sentence length of a prompt—while keeping the underlying meaning identical—can severely degrade model performance. The intricate sentence structures obscure the core intent of the request, triggering hallucinations and reasoning failures in models that easily succeed when given simpler versions of the same prompt.
Human Processing:
[Simple Query] "Find the cup." ──> Success
[Complex Query] "Find the blue ceramic cup on the desk." ──> Higher Confidence / Easier Success
Neural Model Processing:
[Simple Query] "Find the cup." ──> Success
[Complex Query] "Find the blue ceramic cup on the desk." ──> Representation Collapse / Failure
The Training Distribution Disconnect
Why are neural architectures so linguistically brittle? A primary driver is the distribution of their training data.
Many vision-language and retrieval models are trained on short, clean, template-driven inputs—such as brief image captions or single-word category labels. When deployed in the real world, they encounter a severe distribution shift.
For example, open-vocabulary object detection models excel at identifying standard category names. However, as documented by Schulter et al. in their evaluation of the OmniLabel benchmark, there is a stark negative correlation between description length and object detection accuracy. When a query transitions from a simple noun to a free-form text description involving multiple attributes or actions, the model's mapping between semantic concepts and spatial regions breaks down.
As noted by Horvitz et al. and Xie et al., the statistical patterns that models rely on for short, common inputs simply do not generalize to the intricate dependencies and noise introduced by longer natural language sentences.
Bypassing the Bottleneck
To mitigate the descriptive detail penalty, recent work has shifted away from forcing monolithic models to process complex queries in a single pass. Instead, researchers are designing architectures that mimic human decomposition strategies.
One prominent approach is the "divide-and-conquer" framework. In A Neural Divide-and-Conquer Reasoning Framework for Image Retrieval from Linguistically Complex Text, Hu et al. proposed breaking down compound, multi-constraint narratives into a series of simple, independent propositions. Each proposition is verified separately before a neural-symbolic reasoner aggregates the results.
Similarly, in the context of object detection, Park et al. introduced a modular framework that translates long, descriptive specifications into a sequence of subtasks handled by specialized tools. By decomposing a long-form query into digestible, short-form pieces, these methods bypass the complexity bottleneck entirely—allowing models to benefit from descriptive detail rather than being penalized by it.
Key Foundational Work
Using Machine Learning Techniques to Interpret WH-questions (2001) — contributed 2 concept elements to this direction.
Development and Evaluation of a Broad-Coverage Probabilistic Grammar of English-Language Computer Manuals (1992) — contributed 1 concept elements to this direction.