Where2Act: From Pixels to Actions for Articulated 3D Objects

Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, Shubham Tulsiani

TL;DR

Where2Act bridges the gap between seeing and doing by teaching robots to predict exactly where and how to interact with articulated objects like drawers and doors. By learning through trial-and-error in simulation, the system identifies "actionable" pixels and proposes successful manipulation trajectories that generalize to entirely novel object categories and real-world scans.

What Problem Does This Paper Address?

Imagine a robot in a kitchen. Traditional computer vision might help it label a "drawer" or a "handle," but that doesn't tell the robot how to open it. Does it pull the handle? Push the corner? In what direction?

Most existing approaches fall into two camps: passive perception (labeling parts) or complex robotics pipelines that require perfect 3D models and state estimation. The authors of Where2Act argue for a middle ground: actionable perception. They want a system that looks at an object and immediately understands the "affordances"—the potential for interaction—at a pixel level. The challenge is predicting not just where to touch, but the specific 6-DoF (degree of freedom) gripper pose and the likelihood of success for different actions like pushing or pulling.

Methods

The researchers developed a framework that learns by interacting with objects in a physics-rich simulator called SAPIEN.

The Architecture

The core is an encoder-decoder network that takes either a 2D RGB image or a 3D partial point cloud as input. As shown in Figure 2, the network branches into three specialized "decoding heads":

  1. Actionability Scoring ($D_a$): Predicts a score for every pixel, indicating how likely a successful interaction is at that spot.
  2. Action Proposal ($D_r$): A conditional generative model that proposes multiple potential gripper orientations (trajectories) for a given pixel.
  3. Action Scoring ($D_s$): Evaluates each proposal to estimate its specific likelihood of success, accounting for factors like potential collisions.
Figure 2. Network Architecture. Our network takes an 2D image or a 3D partial scan as input and extract per-pixel feature fp using (a) Unet [39] for 2D images and (b) PointNet++ [34] for 3D point clouds. To decode the per-pixel actionable information, we propose three decoding heads: (c) an actionability scoring module Da that predicts a score ap ∈[0,1]; (d) an action proposal module Dr that proposes multiple gripper orientations Rz|p ∈SO(3) sampled from a uniform Gaussian random noise z; (e) an action scoring module Ds that rates the confidence sR|p ∈[0,1] for each proposal.

The "Secret Sauce": Online Adaptive Sampling

Training a robot to pull a handle is hard because most random attempts fail—only about 1% of random "pull" actions on a drawer actually succeed. To solve this data imbalance, the authors used Online Adaptive Sampling. Instead of just trying things at random, the agent uses its current (imperfect) model to bias its exploration toward regions it thinks might work. This creates a virtuous cycle where the agent spends more time practicing difficult but successful interactions, like grasping a small handle.

Datasets

The study utilized the PartNet-Mobility dataset within the SAPIEN simulator.

  • Scale: 2,346 3D CAD models across 46 categories.
  • Training/Testing: They used 10 categories for training (e.g., cabinets, doors, faucets) and reserved 5 categories strictly for testing generalization (e.g., washing machines, buckets).
  • Real-World Validation: The model was also tested on scans from the Replica and Google Scanned Objects datasets, as well as real-world 2D images.

Key Insights and Ideas

  • Affordances are Localized: Successful interactions are often tied to tiny geometric features like handles or edges. The model learns to ignore flat, non-functional surfaces for pulling while highlighting them for pushing.
  • Generalization via Geometry: Because the network learns to associate local geometric shapes (like a protruding cylinder) with actions (like pulling), it can successfully interact with a washing machine handle even if it was only trained on cabinet drawers.
  • Interaction as Supervision: The system doesn't need humans to label "handles." It discovers them itself by seeing which pixels, when pulled, result in the object moving.

Key Results

The authors evaluated their model using the Sample-Success-Rate (SSR), which measures how often the network's top-rated proposals actually work in the simulator.

  • 3D Superiority: The version using 3D point clouds (3D-ours) significantly outperformed 2D versions and geometric baselines. For the "pushing" task, it achieved a success rate of ~28.5% on seen categories and ~14.7% on entirely unseen categories, far exceeding random baselines.
  • Visualizing Success: As seen in Figure 1, the model correctly identifies that the handle of a drawer is the primary spot for pulling, while the entire face of the drawer is suitable for pushing.
Figure 1. The Proposed Where2Act Task. Given as input an ar- ticulated 3D object, we learn to propose the actionable information for different robotic manipulation primitives (e.g. pushing, pulling): (a) the predicted actionability scores over pixels; (b) the proposed interaction trajectories, along with (c) their success likelihoods, for a selected pixel highlighted in red. We show two high-rated proposals (left) and two with lower scores (right) due to interaction orientations and potential robot-object collisions.
  • Real-World Transfer: Despite being trained entirely in simulation, the model produced highly sensible actionability maps when applied to real-world images of toasters, boxes, and nightstands (see Figure 6).
Figure 6. We visualize our action scoring predictions given certain gripper orientations over three real-world 3D scans from the Replica dataset [42] and Google Scanned Objects [38, 35], as well as on two 2D real images [2]. Results are shown over all pixels because of no access to the articulated part masks. Though there is no guarantee for the predictions over pixels outside the articulated parts, the results make sense if we allow motion for the entire objects.

Limitations and Future Work

While impressive, Where2Act has a few hurdles to clear:

  • Single-View Ambiguity: Looking at an object from one angle might hide the joints. A robot might try to pull a door from the hinge side because it can't see where the hinges are.
  • Fixed Primitives: The current model is limited to six pre-defined "atomic" actions (push, pull, and directional variations). Future work could involve learning free-form, complex manipulation trajectories.
  • Static Snapshots: The model doesn't explicitly account for the axis of motion. Integrating part segmentation and motion axis prediction could make the interactions even more robust.

Summary updated

Abstract

One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. Check the website for code and data release: https://cs.stanford.edu/~kaichun/where2act/
BibTeX
@misc{mo2021where,
  author = {Kaichun Mo and Leonidas J. Guibas and Mustafa Mukadam and Abhinav Gupta and Shubham Tulsiani},
  title = {{Where2Act: From Pixels to Actions for Articulated 3D Objects}},
  year = {2021},
  eprint = {2101.02692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.02692},
}
Figure