LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping

Manav Kulshrestha, Shreyans Jain, Apoorva Vashisth, Ankush Gupta, Damon Conover, Aniket BeraarXiv 2026 · arXiv

TL;DR

LAGR (Language-Aware Grasp Re-Ranking) is a new framework that masters task-oriented grasping by decoupling physical feasibility from task suitability. By training on a massive new dataset of 2.68 million scored grasps, LAGR outperforms existing methods by over 9% and demonstrates high reliability in real-world robotic handover and tool-use scenarios.

What Problem Does This Paper Address?

In robotics, "grasping" isn't just about not dropping an object; it's about what you plan to do with it next. If you pick up a hammer to use it, you grab the handle. If you pick it up to hand it to someone else, you grab the head so the handle is free for them. This is Task-Oriented Grasping (TOG).

The challenge is two-fold. First, a robot must balance physical feasibility (can I actually hold this without it slipping?) with functional suitability (is this the right spot for the intended task?). Second, there is a massive data gap: existing datasets for TOG are often small, synthetic, or lack the dense, high-quality annotations needed to train robust models. Most current models struggle because they try to solve both feasibility and suitability in one go, often leading to "semantically correct" grasps that are physically impossible to execute.

Examples of LAGR generating distinct task-specific grasps for the same real-world objects.

Methods

The authors introduce a two-stage architecture that explicitly separates the "can I?" from the "should I?"

  1. Task-Independent Grasp Proposal Generator: This stage uses a multimodal encoder (MultiMAE) to fuse RGB, depth, and semantic information. It generates a set of 96 physically plausible grasp candidates based purely on the geometry and visibility of the object. Each candidate gets a "physical confidence" score.
  2. Language-Aware Contextual Re-ranker: This is the "brain" of the operation. It takes the grasp proposals and a natural language instruction (e.g., "Pick up the hammer to pound a nail"). Using a transformer-based architecture, it reasons over all proposals jointly to determine which one best fits the task.

By decoupling these steps, the model ensures that the final selection is both physically stable and task-appropriate. The re-ranker doesn't just look at visual features; it incorporates explicit geometric properties of the grasp (like orientation and gripper opening) to make its decision.

Overview of the LAGR architecture, showing the proposal generation and re-ranking stages.

Datasets

To train LAGR, the authors constructed a massive new benchmark:

  • 2.68 Million Scored Grasps: Spanning 100,911 image-task pairs across 17 object categories.
  • Dual-Scoring System: Every grasp is labeled with a physical feasibility score (derived from 3D mesh analysis) and a task-conditioned utility score (based on functional regions like handles).
  • Real-World Grounding: Unlike many synthetic datasets, this is built on top of the HANDAL dataset, which features real-world observations of manipulable objects.

Key Insights and Ideas

  • Decoupling is a Power-Up: Separating physical stability from task intent prevents the model from getting "distracted." Ablation studies showed that removing the task-conditioning dropped accuracy from ~82% to 53%.
  • Joint Reasoning Matters: Instead of scoring each grasp in isolation, the re-ranker looks at the whole set of candidates. This allows it to pick the best relative option for a specific scene.
  • Geometry is Essential: The researchers found that including explicit geometric features (center, orientation, etc.) in the re-ranker was significantly more effective than relying on visual features alone.
  • Handover vs. Functional Use: The model successfully learns the nuance of "leaving the handle accessible" for handovers, a common failure point in previous systems.

Key Results

LAGR sets a new state-of-the-art for task-oriented grasping:

  • 9.3% Improvement: LAGR achieved an 85.9% success rate on the task-conditioned benchmark, outperforming the previous best method (GATER) by a wide margin.
  • Efficiency: Despite its complexity, the model runs in about 37ms per sample, making it suitable for real-time robotic applications.
  • Real-Robot Validation: In tests with a Franka Research 3 robot, LAGR achieved a 93% overall success rate. It was particularly dominant in handover tasks, where it achieved 100% success on pan handovers compared to 0% for the baseline.

Limitations and Future Work

While impressive, LAGR has a few areas for growth:

  • 5-DoF Limitation: The current system uses a 5-DoF (planar) grasp representation. While effective for many tabletop tasks, it doesn't capture the full complexity of 6-DoF grasping in 3D space.
  • Gripper Specificity: The model is currently optimized for two-finger parallel-jaw grippers. Extending this to multi-fingered robotic hands remains a challenge for future work.
  • Proposal Dependency: The re-ranker is only as good as the initial proposals. If the generator fails to find a good physical grasp, the re-ranker has nothing "good" to choose from.

Future research will focus on more adaptive grasp generation strategies and expanding the framework to handle more complex robotic hardware.

Summary updated

Abstract

Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from $17$ object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.
BibTeX
@misc{kulshrestha2026lagr,
  author = {Manav Kulshrestha and Shreyans Jain and Apoorva Vashisth and Ankush Gupta and Damon Conover and Aniket Bera},
  title = {{LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping}},
  year = {2026},
  eprint = {2611.23858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23858},
}
Figure