AffordJEV: Affordance Grounding by Confirming Candidate Regions

Jincheng Li, Yutian Zhang, Jianyu Zhang, Xinshun Wang, Mengyuan LiuarXiv 2026 · arXiv

Finding the handle of a mug or the blade of a knife seems like an instantaneous, wordless act for humans. Yet, in the world of robotics and computer vision—a field known as affordance grounding—most current AI models treat this as a slow, deliberative reasoning task. They "think" in text before pointing to pixels.

A new paper from Peking University, AffordJEV, challenges this status quo. By shifting from slow "System Two" reasoning to fast "System One" perceptual judgment, the authors have created a model that is not only more accurate but also significantly more efficient.

TL;DR

AffordJEV introduces a "System One" approach to affordance grounding, replacing slow text-based reasoning with direct perceptual judgment. By using a hierarchical framework and a novel "region-set judge," it outperforms state-of-the-art reasoning models while requiring four times fewer model requests.

What Problem Does This Paper Address?

Affordance grounding is the task of identifying which part of an object performs a specific action (e.g., "Where do I grasp this hammer?").

Currently, the dominant paradigm follows a "System Two" approach (inspired by Daniel Kahneman’s Thinking, Fast and Slow). These models, usually Multimodal Large Language Models (MLLMs), write out a "chain of thought" in text before outputting a bounding box or points. This is problematic for three reasons:

  1. High Cost: Generating long strings of text for every decision is computationally expensive.
  2. Lack of Control: The semantic decision (the text) is often disconnected from the actual spatial extent of the object parts.
  3. Hard to Correct: If the model makes a mistake early in its reasoning, it’s difficult to fix.

The authors argue that affordance is essentially a perceptual judgment. You don't need to describe a handle to see it; you just recognize it.

Methods

AffordJEV replaces the reasoning chain with a Region-set Judge operating within a hierarchical framework.

1. The Hierarchical Framework

Instead of trying to find the affordance in one go, the model works level-by-level: Object $\rightarrow$ Part $\rightarrow$ Sub-part. Crucially, evidence flows in both directions. If the model is unsure about an object but finds a very convincing "handle" part, it can backtrack and confirm the object. This "coarse-to-fine" approach mimics how humans narrow their gaze.

The hierarchical inference process: from identifying the can to finding the pull-tab.

2. The Region-set Judge

The core of the system is a modified Qwen3.5-4B model. Unlike standard models that look at one candidate region at a time, the Region-set Judge looks at all candidates for a level (e.g., all parts of a drill) in a single forward pass.

  • Region Tokens: Candidate masks are converted into "tokens" by pooling visual features and adding geometric data (box, area, centroid).
  • Set-level Decision: A "set block" allows candidates to attend to one another, helping the model choose the best candidate among competitors.
  • Direct Readout: Instead of generating text, the model projects its internal state onto its own pre-trained "Yes" and "No" embeddings to produce a score.

Datasets

  • InstructPart: The primary dataset used for training and in-domain evaluation. It contains task-oriented part segmentations.
  • UMD & IIT-AFF: Used for zero-shot transfer evaluation to test how well the model generalizes to unseen objects and environments.
  • Definition Alignment: The authors noted that different datasets use different words for the same parts (e.g., "grasp" vs. "hold"). They introduced a "frozen definition-aligned protocol" to ensure fair comparisons across benchmarks.

Key Insights and Ideas

  • Perception > Reasoning: For grounding tasks, direct visual judgment is more robust than text-based deliberation. The model's "System One" approach is faster and avoids the "hallucination" risks of text generation.
  • The Power of Comparison: By judging regions as a set, the model can perform relative comparisons. It’s easier to identify "the most handle-like part" when you can see all the parts at once.
  • Leveraging LLM Priors: By using the model's existing "Yes/No" embeddings, the researchers could train the system on just 30,000 requests—a tiny fraction of what's usually needed to train a vision-language head from scratch.

Key Results

AffordJEV sets a new bar for affordance grounding across multiple metrics (gIoU, cIoU, and P50).

  • Superior Accuracy: On the InstructPart benchmark, AffordJEV achieved 70.44% gIoU, outperforming the reasoning-based Affordance-R1 (59.49%) and VisionReasoner (47.26%).
  • Efficiency: It reduced the number of model requests from 33.0 to just 8.1 per query, making it significantly faster for real-time robotics.
  • Zero-Shot Success: It maintained its lead in zero-shot transfer to UMD and IIT-AFF, proving that its perceptual logic generalizes well.
Figure 1. Results on InstructPart, UMD and IIT-AFF (%). Each axis is one metric, scaled to the range of the four methods; the numbers are our scores (Tabs. 1 to 3).

Limitations and Future Work

While AffordJEV is a major step forward, the authors highlight a few remaining hurdles:

  • Over-acceptance: In some cases, particularly with large containers, the model accepts too much of the object rather than just the functional part.
  • Brittle Thresholds: The "confidence gate" that triggers backtracking is sensitive. If a judge is "too sharp" (overconfident), it might stay closed even when it has made a mistake.
  • Abstention: The model still struggles with knowing when to "abstain" or say that none of the proposed regions fit the task.

By proving that "System One" perception is sufficient for complex affordance tasks, AffordJEV opens the door for more responsive and efficient robotic interaction models.

Summary updated

Abstract

Affordance grounding asks which visible part of an object performs a requested operation, a basic capability for embodied manipulation and human--robot interaction. Most existing methods follow a System Two paradigm: a multimodal language model first writes a chain of thought and then turns its conclusion into a box or points for a segmenter. This incurs considerable inference cost, gives the semantic decision little direct control over the extent of the final region, and makes mistakes hard to correct. Yet locating an affordance is essentially a perceptual judgment that need not be argued out in words. We therefore propose, in which frozen segmenters propose candidate regions and a System One model decides directly which of them perform the operation, generating no text and reading each decision from a single forward pass. Our contributions are twofold. (1) An inference framework that confirms candidate regions level by level, from object to part to sub-part, with evidence passed between the levels in both directions so that the levels correct one another. (2) A System One model trained for this framework: existing System One models are built for general-purpose decisions, judge one candidate per request and are ill-suited to the framework's region questions, so ours learns directly from the requests the framework itself produces and judges all candidates of a level at once. Trained only on InstructPart and evaluated with definition-aligned operation words, outperforms chain-of-thought methods in domain and in zero-shot transfer to UMD and IIT-AFF, is better overall than existing System One models run in the same framework, and needs fewer judge requests. To our knowledge, is the first approach to bring System One models to affordance grounding.
BibTeX
@misc{li2026affordjev,
  author = {Jincheng Li and Yutian Zhang and Jianyu Zhang and Xinshun Wang and Mengyuan Liu},
  title = {{AffordJEV: Affordance Grounding by Confirming Candidate Regions}},
  year = {2026},
  eprint = {2611.07896},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07896},
}
Figure