TL;DR
LAGR (Language-Aware Grasp Re-Ranking) is a new framework that masters task-oriented grasping by decoupling physical feasibility from task suitability. By training on a massive new dataset of 2.68 million scored grasps, LAGR outperforms existing methods by over 9% and demonstrates high reliability in real-world robotic handover and tool-use scenarios.
What Problem Does This Paper Address?
In robotics, "grasping" isn't just about not dropping an object; it's about what you plan to do with it next. If you pick up a hammer to use it, you grab the handle. If you pick it up to hand it to someone else, you grab the head so the handle is free for them. This is Task-Oriented Grasping (TOG).
The challenge is two-fold. First, a robot must balance physical feasibility (can I actually hold this without it slipping?) with functional suitability (is this the right spot for the intended task?). Second, there is a massive data gap: existing datasets for TOG are often small, synthetic, or lack the dense, high-quality annotations needed to train robust models. Most current models struggle because they try to solve both feasibility and suitability in one go, often leading to "semantically correct" grasps that are physically impossible to execute.
Methods
The authors introduce a two-stage architecture that explicitly separates the "can I?" from the "should I?"
- Task-Independent Grasp Proposal Generator: This stage uses a multimodal encoder (MultiMAE) to fuse RGB, depth, and semantic information. It generates a set of 96 physically plausible grasp candidates based purely on the geometry and visibility of the object. Each candidate gets a "physical confidence" score.
- Language-Aware Contextual Re-ranker: This is the "brain" of the operation. It takes the grasp proposals and a natural language instruction (e.g., "Pick up the hammer to pound a nail"). Using a transformer-based architecture, it reasons over all proposals jointly to determine which one best fits the task.
By decoupling these steps, the model ensures that the final selection is both physically stable and task-appropriate. The re-ranker doesn't just look at visual features; it incorporates explicit geometric properties of the grasp (like orientation and gripper opening) to make its decision.
Datasets
To train LAGR, the authors constructed a massive new benchmark:
- 2.68 Million Scored Grasps: Spanning 100,911 image-task pairs across 17 object categories.
- Dual-Scoring System: Every grasp is labeled with a physical feasibility score (derived from 3D mesh analysis) and a task-conditioned utility score (based on functional regions like handles).
- Real-World Grounding: Unlike many synthetic datasets, this is built on top of the HANDAL dataset, which features real-world observations of manipulable objects.
Key Insights and Ideas
- Decoupling is a Power-Up: Separating physical stability from task intent prevents the model from getting "distracted." Ablation studies showed that removing the task-conditioning dropped accuracy from ~82% to 53%.
- Joint Reasoning Matters: Instead of scoring each grasp in isolation, the re-ranker looks at the whole set of candidates. This allows it to pick the best relative option for a specific scene.
- Geometry is Essential: The researchers found that including explicit geometric features (center, orientation, etc.) in the re-ranker was significantly more effective than relying on visual features alone.
- Handover vs. Functional Use: The model successfully learns the nuance of "leaving the handle accessible" for handovers, a common failure point in previous systems.
Key Results
LAGR sets a new state-of-the-art for task-oriented grasping:
- 9.3% Improvement: LAGR achieved an 85.9% success rate on the task-conditioned benchmark, outperforming the previous best method (GATER) by a wide margin.
- Efficiency: Despite its complexity, the model runs in about 37ms per sample, making it suitable for real-time robotic applications.
- Real-Robot Validation: In tests with a Franka Research 3 robot, LAGR achieved a 93% overall success rate. It was particularly dominant in handover tasks, where it achieved 100% success on pan handovers compared to 0% for the baseline.
Limitations and Future Work
While impressive, LAGR has a few areas for growth:
- 5-DoF Limitation: The current system uses a 5-DoF (planar) grasp representation. While effective for many tabletop tasks, it doesn't capture the full complexity of 6-DoF grasping in 3D space.
- Gripper Specificity: The model is currently optimized for two-finger parallel-jaw grippers. Extending this to multi-fingered robotic hands remains a challenge for future work.
- Proposal Dependency: The re-ranker is only as good as the initial proposals. If the generator fails to find a good physical grasp, the re-ranker has nothing "good" to choose from.
Future research will focus on more adaptive grasp generation strategies and expanding the framework to handle more complex robotic hardware.
Summary updated