Online Tool Selection with Learned Grasp Prediction Models

Khashayar Rohanimanesh, Jake Metzger, William Richards, Aviv Tamar

TL;DR

This paper introduces a method for robots to intelligently decide when to swap tools (end-effectors) during complex bin-picking tasks. By modeling the problem as a Markov Decision Process and using a "Sparse Tree Search" with "void zones," the authors achieved a 50% increase in throughput compared to standard greedy approaches.

What Problem Does This Paper Address?

In industrial bin picking, a single robot often needs to handle a chaotic mix of objects—some large and flat, others small or irregularly shaped. To maximize success, these robots are equipped with tool changers that allow them to swap between different vacuum suctions or grippers.

However, tool changing is a double-edged sword: while the "right" tool increases the chance of a successful pick, the act of swapping tools takes significant time (cycle time). The challenge lies in online planning: the robot must decide the optimal sequence of objects to pick and when to trigger a tool change, all while dealing with the fact that many objects are hidden (occluded) under others. A simple "pick the best thing right now" (greedy) strategy often leads to excessive, time-consuming tool swaps.

Methods

The authors formalize this as the Grasp Tool Selection Problem (GTSP) and treat it as a Markov Decision Process (MDP). To make it solvable in real-time, they employ several key techniques:

  1. Model Predictive Control (MPC): Instead of planning for the entire bin, the robot looks ahead only a few steps (a receding horizon) based on what it can currently see.
  2. Void Zones: This is a clever heuristic to handle uncertainty. When the robot plans a future pick in its look-ahead window, it creates a "void zone" (a geometric radius) around that pick. It assumes that picking one object will disturb or remove nearby objects, so it ignores other grasp proposals in that radius for the remainder of that planning horizon.
  3. Sparse Tree Search (STS): While the problem can be solved exactly using Integer Linear Programming (ILP), ILP is too slow for high-speed production. The authors proposed STS, which only considers the top k grasp proposals for each tool. This provides near-optimal results but is orders of magnitude faster.
  4. $\beta$-TC-Score: The authors argue that "throughput" (picks per hour) is a noisy metric that rewards aggressive, potentially damaging behavior. They introduced a new metric that balances Pick Success Rate (PSR) and Tool Consistency Rate (TCR) using a harmonic mean (similar to an F-beta score in machine learning).

Datasets

  • Synthetic Data: The authors created a simulator that generates random "grasp maps" using Gaussian kernels to mimic object centers. This allowed them to rigorously test the ILP vs. STS solvers across thousands of scenarios.
  • Real-World Evaluation: Testing was performed on a Fanuc LRMate 200iD/7L industrial arm equipped with two different vacuum tools (30mm and 50mm). The "dataset" consisted of a bin filled with an assortment of mixed consumer goods (varying sizes, weights, and shapes).
  • Grasp Models: The underlying grasp success predictors were trained on a proprietary dataset of ~5k grasp attempts per tool using a ResNet-101 backbone.

Key Insights and Ideas

  • The "Look-Ahead" Sweet Spot: Interestingly, the research found that a horizon of $H=2$ or $H=3$ is often sufficient. Planning too far ahead actually hurts performance because the deterministic "void zone" approximation becomes less accurate as more objects are moved.
  • Decoupling Learning and Planning: The method doesn't require training a new "end-to-end" model. It uses existing, pre-trained grasp models for each tool and simply adds a planning layer on top. This makes it highly practical for existing industrial deployments.
  • Throughput vs. Efficiency: A robot that fails 10 times a minute but succeeds twice might have the same throughput as a robot that tries twice and succeeds twice. The authors' $\beta$-TC-score correctly identifies the second robot as superior because it is more efficient and less likely to damage goods.

Key Results

  • Throughput Boost: In real-world trials, the MPC-STS approach achieved 429.75 picks per hour, a massive improvement over the "Naive Greedy" baseline (317 picks/hr) and the "Randomized" baseline (186 picks/hr).
  • Efficiency: The MPC-STS method significantly reduced the number of tool changes (TC) while maintaining a high pick success count, leading to the highest $\beta$-TC-score (0.6885) among all tested algorithms.
  • Computational Speed: The STS solver proved to be "orders of magnitude" faster than the ILP solver, making it the only viable candidate for real-time robotic control where decisions must be made in milliseconds.

Limitations and Future Work

  • Deterministic Assumption: The "void zone" is a simplified way to model the physics of a bin. It doesn't account for objects sliding or tumbling from one side of the bin to the other after a pick.
  • Static Tool Set: The current work assumes a fixed set of tools. Future iterations could explore more complex scenarios, such as tools with multiple suction cups or hybrid gripper-vacuum systems.
  • Visual Dynamics: The authors suggest that incorporating "deep visual predictive models"—AI that can literally "imagine" how the bin will look after a pick—could eventually replace the heuristic void zones, though such models are currently difficult to deploy in high-variability production environments.

Summary updated

BibTeX
@inproceedings{rohanimanesh2023online,
  author = {Khashayar Rohanimanesh and Jake Metzger and William Richards and Aviv Tamar},
  title = {{Online Tool Selection with Learned Grasp Prediction Models}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160952},
}

Loading related content...

Figure