Abstract
Affordance grounding is a critical task in Embodied AI, involving the localization of interaction regions on objects. Traditional methods are often limited by simple action labels, object-centric biases, and a lack of reasoning for functional alternatives. We present WorldAfford, a framework that shifts affordance grounding toward complex, natural language instructions. By leveraging the reasoning capabilities of Large Language Models (LLMs) and the zero-shot capabilities of vision foundation models, WorldAfford identifies multiple interaction regions and suggests alternative tools in diverse environments. Our approach achieves state-of-the-art performance on the AGD20K benchmark and the newly introduced LLMaFF dataset.
Introduction
In the context of Embodied AI, affordance grounding refers to identifying where an agent should interact with an object (e.g., the handle of a mug). Historically, this field has faced three primary challenges:
- Instruction Complexity: Models typically rely on "naïve" labels like "catch" or "write," failing to parse abstract human needs.
- Object-Centric Bias: Datasets often feature single objects in isolation, neglecting the complexity of real-world scenes.
- Rigidity: Traditional models fail when a specific requested object is missing, lacking the common sense to identify functional substitutes.
WorldAfford addresses these issues by integrating LLMs with vision foundation models through a multi-stage reasoning and localization pipeline.
Methodology
The WorldAfford framework consists of three core components designed to bridge the gap between high-level language and pixel-level grounding.
Affordance Reasoning Chain-of-Thought (ARCoT)
Instead of direct inference, WorldAfford employs an ARCoT strategy using GPT-4. This involves:
- Object-Oriented Reasoning: Identifying candidate objects and potential functional alternatives (e.g., suggesting a chair if a bed is requested but unavailable).
- Action-Oriented Reasoning: Decomposing complex instructions into specific sub-actions (predicate-object pairs), making the task more manageable for the grounding module.
Zero-shot Multiple Object Grounding
To handle a wide variety of objects without specific retraining, the system utilizes the Segment Anything Model (SAM) for high-quality mask generation and CLIP to match these masks with the object categories identified during the reasoning phase.
Affordance Region Localization
To pinpoint interaction pixels, the authors introduce a Weighted Context Broadcasting (WCB) module. Integrated into a DINO-ViT backbone, WCB facilitates denser attention maps by broadcasting context across tokens. This allows the model to focus on multiple informative objects simultaneously, which is essential for complex scenes.
Datasets and Evaluation
The framework was evaluated on two primary benchmarks:
- AGD20K: A large-scale dataset of ~20,000 images used to train the localization module.
- LLMaFF (LLM-based Affordance): A new dataset introduced in this work containing 550 images with natural language instructions and manually labeled affordance maps. LLMaFF specifically tests logical reasoning and multi-object grounding.
Results
WorldAfford demonstrates superior performance across both benchmarks. On AGD20K, it achieved a KLD of 1.201 and an NSS of 1.255, outperforming previous state-of-the-art models like Cross-view-AG+. On the more challenging LLMaFF dataset, the performance gap was significant: WorldAfford reached an NSS of 2.819, compared to 1.713 for the previous best model (LOCATE).
Furthermore, the localization module is highly efficient, requiring only 6.5M learnable parameters, a fraction of the 82.27M required by competing architectures.
Conclusion
WorldAfford represents a significant step toward more capable Embodied AI by enabling agents to understand and act upon complex human instructions. By decomposing tasks and reasoning about functional alternatives, the framework exhibits human-like flexibility. Future research will focus on deploying this framework in real-world robotic systems and extending it to handle temporal reasoning in video-based instructions.
Summary updated