Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement

Ashish Malik, Caleb Lowe, Aayam Shrestha, Stefan Lee, Fuxin Li, Alan FernarXiv 2026 · arXiv

TL;DR

The paper introduces RAMP-3D (Reactive Action Mask Planner), a novel framework that tackles long-horizon 3D box rearrangement tasks directly from natural-language goals and raw visual inputs. By bypassing traditional, brittle symbolic layers and error-prone 2D vision-language models, RAMP-3D formulates planning as a sequential prediction of paired 3D segmentation masks—one for the object to pick and one for the target region to place it.

What Problem Does This Paper Address?

Imagine telling a warehouse robot: "Move all of the larger boxes on the floor to the back shelves, but don't stack them more than two high." To execute this, the robot must reason over long horizons, understand complex 3D geometry, handle occlusions, and satisfy implicit semantic constraints.

Historically, roboticists have solved this using two main paradigms, both of which have major drawbacks:

  1. Symbolic Planners (e.g., PDDL): These require an engineered symbolic layer to translate raw perception into discrete predicates. Building and maintaining these layers is notoriously difficult, especially when dealing with noisy sensors and continuous 3D spatial relations.
  2. 2D Vision-Language Models (VLMs): While great at generating text or 2D coordinates, 2D VLMs struggle with depth, volumetric relationships, and complex 3D clutter.

This paper addresses the gap by asking: Can we use 3D vision-language grounding models to plan actions directly in 3D space without any explicit symbolic abstraction?

Methods

RAMP-3D builds upon UniVLG, a state-of-the-art 3D vision-language model designed to ground language queries into 3D point cloud segmentation masks. The authors adapt and extend this architecture for sequential robotic planning through two key modifications:

  • Pair-Contrastive Features: A valid pick-and-place action requires a highly coordinated pair of decisions (what to pick and where to put it). To enforce this, RAMP-3D introduces a pair-contrastive module. It projects decoder queries into distinct pickup and putdown embeddings, computing pairwise compatibility scores to ensure the selected object and target region match the task constraints.
  • Last-Action Predictor: A lightweight "done" head (a single-hidden-layer MLP) is added to evaluate all decoder queries and output a termination probability, signaling when the long-horizon task is complete.

Instead of outputting text or coordinates, RAMP-3D operates entirely in mask space. At each step, it outputs a M_pick mask and a M_target mask. These 3D masks inherently encode geometric feasibility and occupancy, acting as a direct bridge between high-level semantics and low-level geometric execution.

Figure 2: Overview of RAMP-3D, built on UniVLG. The model takes posed multi-view RGB-D observations and a natural- language goal as inputs, encodes visual features with a transformer backbone and voxel-based 3D fusion, and encodes the goal with a frozen text encoder. A transformer decoder with learnable queries attends jointly to visual and text tokens and are augmented with pair-contrastive pickup–putdown embeddings and a binary “done” head. At each planning step, the model outputs a pair of 3D masks indicating the pickup target and target region and a termination probability. The predicted masks are projected back to instance IDs to yield box–region actions. The planner is run iteratively for long-horizon box rearrangement.

Datasets

Because existing 3D grounding datasets do not feature long-horizon task trajectories, the authors built a custom dataset using the IsaacSim simulator:

  • Environment: A 12m × 12m warehouse populated with up to 30 boxes (red, blue, yellow), pallets of varying sizes, multi-layer shelves, and visual distractors (barrels and traffic cones).
  • Task Variants: 11 distinct task classes capturing diverse commonsense constraints (e.g., homogeneous stacks, box accessibility, shelf prioritization).
  • Data Volume: Approximately 9.5k unique scene snapshots generated via an oracle planner, with 2.2k held out for testing.
  • Language Augmentation: To ensure linguistic robustness, template-based goals were paraphrased into multiple semantically equivalent variants using an LLM (gpt-oss-120b).

Key Insights and Ideas

  • The Power of 3D Mask Space: Operating directly on 3D point clouds allows the model to implicitly respect physical boundaries and occupancy constraints. The model doesn't need to be explicitly taught what "on top of" means in a symbolic language; it learns it geometrically.
  • Why 2D Baselines Fail: The authors compared RAMP-3D against a strong 2D baseline ("2D-pointer" using GPT-4 and Molmo-72B). The 2D baseline frequently failed because empty placement cells on a pallet lack distinct, object-like visual features. While 2D models default to pointing at highly salient objects (like existing boxes), RAMP-3D successfully localizes empty 3D spatial regions.
  • Reactive vs. Deliberative Planning: RAMP-3D is strictly reactive—it decides the next best step based on the current state. This works remarkably well for localized or ordered tasks, but struggles when actions have fragile, long-term global consequences.

Key Results

The model was evaluated under two execution modes: snap-to-target (where the controller places the box perfectly on the predicted target structure) and free-form (where the placement is derived purely from the center of the predicted mask).

  • One-Step Action Validity: RAMP-3D achieved a 96.8% success rate in predicting valid single-step actions, drastically outperforming the 2D-pointer baseline, which sat at 41.4%.
  • Multi-Step Plan Success: In full rollouts, RAMP-3D achieved an overall success rate of 79.5% in snap-to-target mode and 66.5% in free-form mode.
  • Impact of Clutter: Performance scales gracefully but degrades in highly cluttered environments. For scenes with 1–10 boxes, the multi-step success rate was 87.8%, dropping to 66.7% for dense scenes with 21–30 boxes.
Heatmap showing the joint pickup-putdown correctness across varying scene complexities and IoU thresholds.

Limitations and Future Work

While RAMP-3D represents a major step forward for end-to-end 3D planning, the authors note several limitations:

  1. Lack of Lookahead: Because the policy is purely reactive, it can get trapped in deadlocks on tasks requiring complex sequencing (like Box accessibility, where a robot must avoid trapping certain colors). Integrating lookahead search or combinatorial solvers (like bin-packing algorithms) is a key next step.
  2. Simplified Geometry: The dataset relies entirely on uniform boxes. Future iterations will need to scale to more complex, non-parametric object shapes.
  3. Assumption of Executability: The high-level planner assumes any chosen pick-and-place pair is physically reachable and collision-free for the robot arm. Embedding robot-specific kinematic feasibility directly into the mask generation loop remains an open challenge.

Summary updated

Abstract

We study long-horizon planning in 3D environments from under-specified natural-language goals using only visual observations, focusing on multi-step 3D box rearrangement tasks. Existing approaches typically rely on symbolic planners with brittle relational grounding of states and goals, or on direct action-sequence generation from 2D vision-language models (VLMs). Both approaches struggle with reasoning over many objects, rich 3D geometry, and implicit semantic constraints. Recent advances in 3D VLMs demonstrate strong grounding of natural-language referents to 3D segmentation masks, suggesting the potential for more general planning capabilities. We extend existing 3D grounding models and propose Reactive Action Mask Planner (RAMP-3D), which formulates long-horizon planning as sequential reactive prediction of paired 3D masks: a "which-object" mask indicating what to pick and a "which-target-region" mask specifying where to place it. The resulting system processes RGB-D observations and natural-language task specifications to reactively generate multi-step pick-and-place actions for 3D box rearrangement. We conduct experiments across 11 task variants in warehouse-style environments with 1-30 boxes and diverse natural-language constraints. RAMP-3D achieves 79.5% success rate on long-horizon rearrangement tasks and significantly outperforms 2D VLM-based baselines, establishing mask-based reactive policies as a promising alternative to symbolic pipelines for long-horizon planning.
BibTeX
@misc{malik2026grounding,
  author = {Ashish Malik and Caleb Lowe and Aayam Shrestha and Stefan Lee and Fuxin Li and Alan Fern},
  title = {{Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement}},
  year = {2026},
  eprint = {2603.23676},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.23676},
}
Figure