Abstract
Humans exhibit remarkable cognitive flexibility by selecting tools based on physical and functional properties rather than rigid labels. This paper introduces a parameter-efficient framework that mimics this behavior by aligning visual and linguistic information within a shared 13-dimensional attribute space. By mapping tool images and task descriptions to these key properties, our model (utilizing ~1.5B parameters) achieves 74% accuracy in tool-matching tasks. This performance rivals massive models like GPT-4o while providing a highly interpretable "attribute bridge" for reasoning.
Introduction
When faced with a task like driving a nail without a hammer, humans can identify suitable substitutes—such as a heavy wrench—by evaluating physical attributes like hardness and weight. Current AI systems often lack this flexibility, relying either on brittle direct mappings between tasks and labels or on opaque, resource-intensive Large Multimodal Models (LMMs). We address this gap by proposing a model that extracts functional requirements from language and matches them against the physical properties of objects in a shared, low-dimensional space.
Methods
Our approach employs a dual-pathway attribute alignment framework. The Visual Pathway uses ResNet-50 or ViT backbones to extract features from tool images, which are then mapped to a 13-dimensional vector via an attribute prediction head. Simultaneously, the Language Pathway utilizes lightweight LLMs (e.g., DeepSeek-R1-1.5B) to infer required attributes from natural language task descriptions.
The shared 13-dimensional "bridge" includes:
- Physical Attributes: Elongation, spikiness, size, smoothness, texturedness, hardness.
- Functional Attributes: Graspability, hand involvement, force requirements, body extension.
- Psychological Attributes: Threatness, valence, arousal.
Tool selection is performed by calculating the similarity (cosine or Euclidean) between the task's required attribute vector and the available tools' vectors.
Datasets and Attribute Distribution
We introduce ToolNet, a comprehensive suite of datasets including 10,350 tool images across 115 categories, human-rated attribute scores, and natural language scenarios.
The attribute space allows for a nuanced representation of tools. For instance, a "broom" and a "hammer" have distinct 13D signatures that reflect their different physical and functional roles. This structured representation enables the model to reason about tool suitability even for novel objects.
Results
The combination of DeepSeek-R1-1.5B and ResNet50 achieved a top performance of 74% accuracy on tool selection tasks, matching GPT-4o (73%) despite having significantly fewer parameters. Key findings include:
- Attribute Hierarchy: Manipulation-related attributes (graspability, elongation) are the most critical for successful matching.
- Human Alignment: Model predictions correlated significantly with human preferences ($r = 0.373$), particularly in ambiguous scenarios.
- Generalization: The model demonstrated zero-shot reasoning capabilities, achieving 22.6% accuracy on a "New Tool" dataset containing 25 novel categories never seen during training.
Discussion
While effective, the current framework is limited to static image evaluation and a simplified 13-dimensional space. Future work will explore dynamic temporal sequences for motor control and expanded attribute sets (e.g., material brittleness). Nonetheless, this research demonstrates that low-dimensional attribute alignment provides a powerful, interpretable, and efficient path toward human-like cognitive flexibility in AI.
Summary updated