Finding the handle of a mug or the blade of a knife seems like an instantaneous, wordless act for humans. Yet, in the world of robotics and computer vision—a field known as affordance grounding—most current AI models treat this as a slow, deliberative reasoning task. They "think" in text before pointing to pixels.
A new paper from Peking University, AffordJEV, challenges this status quo. By shifting from slow "System Two" reasoning to fast "System One" perceptual judgment, the authors have created a model that is not only more accurate but also significantly more efficient.
TL;DR
AffordJEV introduces a "System One" approach to affordance grounding, replacing slow text-based reasoning with direct perceptual judgment. By using a hierarchical framework and a novel "region-set judge," it outperforms state-of-the-art reasoning models while requiring four times fewer model requests.
What Problem Does This Paper Address?
Affordance grounding is the task of identifying which part of an object performs a specific action (e.g., "Where do I grasp this hammer?").
Currently, the dominant paradigm follows a "System Two" approach (inspired by Daniel Kahneman’s Thinking, Fast and Slow). These models, usually Multimodal Large Language Models (MLLMs), write out a "chain of thought" in text before outputting a bounding box or points. This is problematic for three reasons:
- High Cost: Generating long strings of text for every decision is computationally expensive.
- Lack of Control: The semantic decision (the text) is often disconnected from the actual spatial extent of the object parts.
- Hard to Correct: If the model makes a mistake early in its reasoning, it’s difficult to fix.
The authors argue that affordance is essentially a perceptual judgment. You don't need to describe a handle to see it; you just recognize it.
Methods
AffordJEV replaces the reasoning chain with a Region-set Judge operating within a hierarchical framework.
1. The Hierarchical Framework
Instead of trying to find the affordance in one go, the model works level-by-level: Object $\rightarrow$ Part $\rightarrow$ Sub-part. Crucially, evidence flows in both directions. If the model is unsure about an object but finds a very convincing "handle" part, it can backtrack and confirm the object. This "coarse-to-fine" approach mimics how humans narrow their gaze.
2. The Region-set Judge
The core of the system is a modified Qwen3.5-4B model. Unlike standard models that look at one candidate region at a time, the Region-set Judge looks at all candidates for a level (e.g., all parts of a drill) in a single forward pass.
- Region Tokens: Candidate masks are converted into "tokens" by pooling visual features and adding geometric data (box, area, centroid).
- Set-level Decision: A "set block" allows candidates to attend to one another, helping the model choose the best candidate among competitors.
- Direct Readout: Instead of generating text, the model projects its internal state onto its own pre-trained "Yes" and "No" embeddings to produce a score.
Datasets
- InstructPart: The primary dataset used for training and in-domain evaluation. It contains task-oriented part segmentations.
- UMD & IIT-AFF: Used for zero-shot transfer evaluation to test how well the model generalizes to unseen objects and environments.
- Definition Alignment: The authors noted that different datasets use different words for the same parts (e.g., "grasp" vs. "hold"). They introduced a "frozen definition-aligned protocol" to ensure fair comparisons across benchmarks.
Key Insights and Ideas
- Perception > Reasoning: For grounding tasks, direct visual judgment is more robust than text-based deliberation. The model's "System One" approach is faster and avoids the "hallucination" risks of text generation.
- The Power of Comparison: By judging regions as a set, the model can perform relative comparisons. It’s easier to identify "the most handle-like part" when you can see all the parts at once.
- Leveraging LLM Priors: By using the model's existing "Yes/No" embeddings, the researchers could train the system on just 30,000 requests—a tiny fraction of what's usually needed to train a vision-language head from scratch.
Key Results
AffordJEV sets a new bar for affordance grounding across multiple metrics (gIoU, cIoU, and P50).
- Superior Accuracy: On the InstructPart benchmark, AffordJEV achieved 70.44% gIoU, outperforming the reasoning-based Affordance-R1 (59.49%) and VisionReasoner (47.26%).
- Efficiency: It reduced the number of model requests from 33.0 to just 8.1 per query, making it significantly faster for real-time robotics.
- Zero-Shot Success: It maintained its lead in zero-shot transfer to UMD and IIT-AFF, proving that its perceptual logic generalizes well.
Limitations and Future Work
While AffordJEV is a major step forward, the authors highlight a few remaining hurdles:
- Over-acceptance: In some cases, particularly with large containers, the model accepts too much of the object rather than just the functional part.
- Brittle Thresholds: The "confidence gate" that triggers backtracking is sensitive. If a judge is "too sharp" (overconfident), it might stay closed even when it has made a mistake.
- Abstention: The model still struggles with knowing when to "abstain" or say that none of the proposed regions fit the task.
By proving that "System One" perception is sufficient for complex affordance tasks, AffordJEV opens the door for more responsive and efficient robotic interaction models.
Summary updated