Most vision-language models are "eager to please." When asked to find a "keyboard" in an image of a forest, a standard model will often hallucinate a mask over a patch of moss or a fallen log. This behavior stems from a fundamental bias in their training: they are almost exclusively shown images paired with descriptions of things that are there. Consequently, the model learns that its job is to find the best match for a query, rather than to verify if the query is actually present.
To move toward truly "honest" models, researchers are shifting the paradigm from simple matching to Negative Constraint Training. By explicitly teaching models to recognize and reject non-existent objects, the field is building vision-language systems that are more robust, less prone to hallucination, and capable of nuanced spatial reasoning.
The Hallucination of Presence
The core problem, as identified by Holy Lovenia and colleagues in the NOPE benchmark, is that modern models fail catastrophically when asked about objects that do not exist in a scene. This is often a result of "lazy" learning. As Liunian Harold Li notes in DesCo, models often rely on simple noun-matching rather than reading the rich descriptive context provided in a query.
This "affirmative bias" is particularly dangerous in specialized fields. In remote sensing, Jiang Wu and Yi Liu found that models would "lie" about features in satellite imagery rather than admit they couldn't see them. Similarly, in document analysis, Francesco Pinto showed that models might "hallucinate" redacted information by relying on memorized training data rather than the actual visual input.
Teaching Models to Say No
The solution is to incorporate "negative samples" into the training process. Instead of only asking "Where is the cat?", researchers now ask "Is there a dog?" in an image containing only cats.
Several distinct mechanisms have emerged to implement this:
- Rejection Tokens: Models like GSVA and LaSagnA introduce a dedicated
[REJ]or rejection token into the model's vocabulary. When a query cannot be fulfilled, the model is trained to output this specific token rather than a coordinate or a mask. - Null-Target Grounding: In the VQD task, Manoj Acharya and Karan Jariwala require models to output an empty set of detections for absent objects, serving as a counter-bias mechanism.
- Negative Mining and Instruction Tuning: Yufei Zhan (in Griffon) and Wentong Li (in Osprey) use negative mining to force the model to verify visual evidence against linguistic prompts. This often involves "deceptive" instructions that challenge the model to identify what isn't there.
Existence Verification as a Prerequisite
The intuition behind these methods is that existence verification should be a prerequisite for localization. A model should first ask, "Does this object exist in the image?" before asking "Where is it?"
Haoxuan You and the Ferret team demonstrated that "spatial negative mining"—asking about objects absent from a specific region—is particularly effective. This forces the model to rely on actual pixel data rather than linguistic priors (e.g., assuming a mouse is present because a keyboard is visible).
This approach has led to massive performance gains. Yongshuo Zong and Zhihua Li showed in Ground-V that training on instructions for absent objects improved accuracy on non-existent targets by over 20%.
The Corrective Lens
Beyond just avoiding errors, negative constraint training acts as a "corrective lens" for Multimodal Large Language Models (MLLMs). Jian Wu and Yang Zhang describe this in PostAlign as a way to penalize the model for following misleading linguistic priors.
By incorporating negative samples, models like RexSeek and HumanRef can handle "open-world" scenarios where the existence of a target is not guaranteed. This shift transforms the model from a passive pattern-matcher into a critical evaluator of visual input, capable of admitting uncertainty and signaling the absence of evidence.