Modeling and Learning Constraints for Creative Tool Use

Tesca Fitzgerald, Ashok K. Goel, Andrea Thomaz

Introduction

Humans possess a remarkable ability for tool improvisation, such as using a heavy book as a paperweight or a rock as a hammer. In contrast, robots typically struggle with any tool that deviates from their specific training data. A critical gap in "creative" robotics is that while current computer vision systems can identify broad "affordance regions" (e.g., identifying the blade of a knife as the cutting surface), they lack the precision required for actual task execution.

To successfully use a tool, a robot must identify the exact "tooltip"—the specific point and orientation on the tool that must interact with the environment. Because these constraints are often hidden and vary depending on the task (e.g., the part of a hammer used for pounding versus prying), robots require a mechanism to identify and adapt to these constraints dynamically.

Figure 1: Visualization of broad affordance regions across various tools, highlighting the difference between general functional areas and specific task-relevant tooltips.

Methodology: Transfer by Correction

This research proposes a framework for tool improvisation consisting of three stages: identification, evaluation, and adaptation. The core technical contribution is the Transfer by Correction algorithm, which models the tooltip as a task-specific spatial constraint.

Tooltip as Constraint

The tooltip is defined as a specific pose (position and orientation) relative to the robot's end-effector. Rather than requiring a full demonstration for every new tool, the system utilizes "Learning by Correction." A human teacher provides sparse, noisy corrections to the robot’s trajectory as it attempts a task with a novel tool.

Constraint Modeling and Selection

The system fits these human corrections into one of two geometric models using RANSAC to filter out noise:

  1. Linear Model: Captures orientation-heavy constraints, such as keeping a hook at a specific angle.
  2. Rotational Model: Captures center-of-rotation constraints, such as a brush pivoting around a specific point.

The robot automatically selects the model that best explains the human's corrections and applies that transformation to the entire task trajectory.

Figure 2: A detailed view of a hammer tooltip model, illustrating how specific regions are mapped for task execution.

Experimental Evaluation

The study utilized both simulation and physical environments to validate the framework:

  • Simulation (Gazebo): A 7-DOF Kinova Gen3 arm performed an exhaustive exploratory analysis. A dataset of 729 perturbations per tool-task pairing was generated to map how small changes in tool pose affect success.
  • Physical Evaluation: A 7-DOF Jaco2 arm with a Robotiq 85 gripper was tested using various tools, including spatulas, mugs, hammers, and scrub-brushes.
  • Tasks: The evaluation focused on three primary tasks: hooking, lifting, and sweeping.

Key Results and Insights

The researchers found that performance sensitivity is non-linear; the relationship between tool pose and success is often "peaky" or bimodal. This suggests that robots cannot simply average trajectories but must find the specific valid constraint manifold.

Key performance metrics include:

  • High Success Rate: Using the best-fit model selection, the robot achieved $\ge 85\%$ of maximum task performance in 83% of within-task transfer cases.
  • Significant Performance Boost: Learned transformations resulted in average performance scores 5.9x to 6.9x higher than the baseline (using the tool without adaptation).
  • Across-Task Transfer: The models successfully transferred learned tool-transforms to different tasks using the same tool, provided the required tooltip orientations were similar. This led to a 2.25x increase in successful executions compared to the baseline.

Conclusion and Future Work

The "Transfer by Correction" framework demonstrates that modeling tooltips as geometric constraints allows for robust tool improvisation. However, the current model assumes a static grasp and does not account for tools slipping within the gripper. Future work may involve "meta-learning," where the robot learns to predict these transforms purely from visual features—such as object dimensions or concavity—after observing a sufficient number of examples, eventually reducing the dependency on human feedback.

Summary updated

Abstract

Improvisation is a hallmark of human creativity and serves a functional purpose in completing everyday tasks with novel resources. This is particularly exhibited in tool-using tasks: when the expected tool for a task is unavailable, humans often are able to replace the expected tool with an atypical one. As robots become more commonplace in human society, we will also expect them to become more skilled at using tools in order to accommodate unexpected variations of tool-using tasks. In order for robots to creatively adapt their use of tools to task variations in a manner similar to humans, they must identify tools that fulfill a set of task constraints that are essential to completing the task successfully yet are initially unknown to the robot. In this paper, we present a high-level process for tool improvisation (tool identification, evaluation, and adaptation), highlight the importance of tooltips in considering tool-task pairings, and describe a method of learning by correction in which the robot learns the constraints from feedback from a human teacher. We demonstrate the efficacy of the learning by correction method for both within-task and across-task transfer on a physical robot.
BibTeX
@article{fitzgerald2021modeling,
  author = {Tesca Fitzgerald and Ashok K. Goel and Andrea Thomaz},
  title = {{Modeling and Learning Constraints for Creative Tool Use}},
  journal = {Frontiers Robotics AI},
  year = {2021},
  doi = {10.3389/frobt.2021.674292},
}
Figure