Resource-Efficient Affordance Grounding with Complementary Depth and Semantic Prompts

Yizhou Huang, Fan Yang, Guoliang Zhu, Gen Li, Hao Shi, Yukun Zuo, Wenrui Chen, Zhiyong Li, Kailun Yang

Introduction

Multimodal affordance grounding is a critical capability for robotic agents, enabling them to identify functional regions of objects—such as the handle of a hammer or the blade of a knife—based on visual, structural, and semantic cues. While depth and language provide essential context beyond simple RGB imagery, existing multimodal models often rely on separate, heavy encoders for each data type. This "brute-force" fusion approach results in massive parameter counts and high computational overhead, hindering real-time deployment on resource-constrained robotic hardware.

The BiT-Align Framework

To address these efficiency challenges, we introduce BiT-Align (Bypass and Text-guided Attention for Multimodal Affordance Grounding). BiT-Align is a weakly-supervised framework that leverages two novel mechanisms to achieve state-of-the-art performance with minimal parameters.

The BiT-Align architecture featuring the Bypass Prompt Module (BPM) and Text Feature Guidance (TFG).

Bypass Prompt Module (BPM)

Instead of a dedicated depth encoder, BiT-Align utilizes the Bypass Prompt Module. BPM treats auxiliary depth data as a "prompt" for the primary RGB Vision Transformer (DINO-ViT-S). Geometric features from depth maps are progressively injected into the intermediate layers of the RGB encoder, modulating visual features without the need for a parallel heavy backbone.

Text Feature Guidance (TFG)

The Text Feature Guidance module uses semantic embeddings (via CLIP) to dynamically control the model's focus. By matching text features (e.g., "hold" or "cut with") to specific attention head masks within the image encoder, the model suppresses irrelevant background noise and highlights regions pertinent to the specific task.

Experimental Setup

The framework was evaluated on two major benchmarks: AGD20K and HICO-IIF. Since these datasets lack native depth information, high-quality pseudo-depth maps were generated using Depth Anything V2. The model learns from exocentric (third-person) interaction images and successfully transfers this knowledge to egocentric (first-person) robotic views.

Key Results and Discussion

BiT-Align demonstrates that multimodal learning does not require excessive parameter bloat. The model achieved state-of-the-art accuracy, improving the Kullback-Leibler Divergence (KLD) metric on AGD20K by 6.0% compared to previous baselines.

Qualitative affordance grounding results and efficiency comparisons against state-of-the-art (SoTA) methods.

The most significant contribution is the model's efficiency. BiT-Align requires only 7.7M parameters, representing an 88.8% reduction over the previous state-of-the-art. Computational complexity (GFLOPs) was reduced by 72.4%, and inference time dropped to 10.0 ms per image. Real-world validation on a UR5 robotic arm confirmed the model's ability to guide functional grasping and tool use in real-time.

Conclusion

BiT-Align proves that treating auxiliary modalities as prompts and using semantic guidance for attention can drastically reduce the resource requirements of multimodal models. While future work will address sensitivity to depth noise and explore larger backbones, BiT-Align represents a major step toward practical, real-time affordance grounding for robotics.

Summary updated

Abstract

Affordance refers to the functional properties that an agent perceives and utilizes from its environment, and is key perceptual information required for robots to perform actions. This information is rich and multimodal in nature. Existing multimodal affordance methods face limitations in extracting useful information, mainly due to simple structural designs, basic fusion methods, and large model parameters, making it difficult to meet the performance requirements for practical deployment. To address these issues, this paper proposes the BiT-Align image-depth-text affordance mapping framework. The framework includes a Bypass Prompt Module (BPM) and a Text Feature Guidance (TFG) attention selection mechanism. BPM integrates the auxiliary modality depth image directly as a prompt to the primary modality RGB image, embedding it into the primary modality encoder without introducing additional encoders. This reduces the model's parameter count and effectively improves functional region localization accuracy. The TFG mechanism guides the selection and enhancement of attention heads in the image encoder using textual features, improving the understanding of affordance characteristics. Experimental results demonstrate that the proposed method achieves significant performance improvements on public AGD20K and HICO-IIF datasets. On the AGD20K dataset, compared with the current state-of-the-art method, we achieve a 6.0% improvement in the KLD metric, while reducing model parameters by 88.8%, demonstrating practical application values. The source code will be made publicly available at https://github.com/DAWDSE/BiT-Align.
BibTeX
@inproceedings{huang2025resource,
  author = {Yizhou Huang and Fan Yang and Guoliang Zhu and Gen Li and Hao Shi and Yukun Zuo and Wenrui Chen and Zhiyong Li and Kailun Yang},
  title = {{Resource-Efficient Affordance Grounding with Complementary Depth and Semantic Prompts}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245943},
}
Figure