Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language

Guangfu Hao, Haojie Wen, Liangxuna Guo, Yang Chen, Yanchao Bi, Shan YuarXiv 2025 · DOI · arXiv

Abstract

Humans exhibit remarkable cognitive flexibility by selecting tools based on physical and functional properties rather than rigid labels. This paper introduces a parameter-efficient framework that mimics this behavior by aligning visual and linguistic information within a shared 13-dimensional attribute space. By mapping tool images and task descriptions to these key properties, our model (utilizing ~1.5B parameters) achieves 74% accuracy in tool-matching tasks. This performance rivals massive models like GPT-4o while providing a highly interpretable "attribute bridge" for reasoning.

Introduction

When faced with a task like driving a nail without a hammer, humans can identify suitable substitutes—such as a heavy wrench—by evaluating physical attributes like hardness and weight. Current AI systems often lack this flexibility, relying either on brittle direct mappings between tasks and labels or on opaque, resource-intensive Large Multimodal Models (LMMs). We address this gap by proposing a model that extracts functional requirements from language and matches them against the physical properties of objects in a shared, low-dimensional space.

Conceptual overview and dual-pathway architecture for attribute alignment.

Methods

Our approach employs a dual-pathway attribute alignment framework. The Visual Pathway uses ResNet-50 or ViT backbones to extract features from tool images, which are then mapped to a 13-dimensional vector via an attribute prediction head. Simultaneously, the Language Pathway utilizes lightweight LLMs (e.g., DeepSeek-R1-1.5B) to infer required attributes from natural language task descriptions.

The shared 13-dimensional "bridge" includes:

  • Physical Attributes: Elongation, spikiness, size, smoothness, texturedness, hardness.
  • Functional Attributes: Graspability, hand involvement, force requirements, body extension.
  • Psychological Attributes: Threatness, valence, arousal.

Tool selection is performed by calculating the similarity (cosine or Euclidean) between the task's required attribute vector and the available tools' vectors.

Datasets and Attribute Distribution

We introduce ToolNet, a comprehensive suite of datasets including 10,350 tool images across 115 categories, human-rated attribute scores, and natural language scenarios.

Tool distribution in attribute space and visualization of attribute vectors for specific tools.

The attribute space allows for a nuanced representation of tools. For instance, a "broom" and a "hammer" have distinct 13D signatures that reflect their different physical and functional roles. This structured representation enables the model to reason about tool suitability even for novel objects.

Results

The combination of DeepSeek-R1-1.5B and ResNet50 achieved a top performance of 74% accuracy on tool selection tasks, matching GPT-4o (73%) despite having significantly fewer parameters. Key findings include:

  • Attribute Hierarchy: Manipulation-related attributes (graspability, elongation) are the most critical for successful matching.
  • Human Alignment: Model predictions correlated significantly with human preferences ($r = 0.373$), particularly in ambiguous scenarios.
  • Generalization: The model demonstrated zero-shot reasoning capabilities, achieving 22.6% accuracy on a "New Tool" dataset containing 25 novel categories never seen during training.

Discussion

While effective, the current framework is limited to static image evaluation and a simplified 13-dimensional space. Future work will explore dynamic temporal sequences for motor control and expanded attribute sets (e.g., material brittleness). Nonetheless, this research demonstrates that low-dimensional attribute alignment provides a powerful, interpretable, and efficient path toward human-like cognitive flexibility in AI.

Summary updated

Abstract

Flexible tool selection reflects a complex cognitive ability that distinguishes humans from other species, yet computational models that capture this ability remain underdeveloped. We developed a framework using low-dimensional attribute representations to bridge visual tool perception and linguistic task understanding. We constructed a comprehensive dataset (ToolNet) containing 115 common tools labeled with 13 carefully designed attributes spanning physical, functional, and psychological properties, paired with natural language scenarios describing tool usage. Visual encoders (ResNet or ViT) extract attributes from tool images while fine-tuned language models (GPT-2, LLaMA, DeepSeek) derive required attributes from task descriptions. Our approach achieves 74% accuracy in tool selection tasks-significantly outperforming direct tool matching (20%) and smaller multimodal models (21%-58%), while approaching performance of much larger models like GPT-4o (73%) with substantially fewer parameters. Human evaluation studies validate our framework's alignment with human decision-making patterns, and generalization experiments demonstrate effective performance on novel tool categories. Ablation studies revealed that manipulation-related attributes (graspability, elongation, hand-relatedness) consistently prove most critical across modalities. This work provides a parameter-efficient, interpretable solution that mimics human-like tool cognition, advancing both cognitive science understanding and practical applications in tool selection tasks.
BibTeX
@misc{hao2025flexible,
  author = {Guangfu Hao and Haojie Wen and Liangxuna Guo and Yang Chen and Yanchao Bi and Shan Yu},
  title = {{Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language}},
  year = {2025},
  eprint = {2505.22146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.22146},
}
Figure