PACA: Prespective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement

Shutong Jin, Ruiyu Wang, Kuangyi Chen, Florian T. PokornyWACV 2025

TL;DR

This paper introduces PACA, a novel zero-shot pipeline for robotic scene rearrangement that integrates generation, segmentation, and feature encoding into a single step. By leveraging Stable Diffusion and introducing perspective control, PACA enables 6-DoF scene rearrangement, significantly outperforming previous multi-step methods and achieving high success rates in real-world robot experiments.

What Problem Does This Paper Address?

Robotic scene rearrangement, like tidying a table, is a complex task. A key challenge is predicting what the "ideal" goal arrangement should look like, as this can vary greatly depending on context. While large-scale generative models like Stable Diffusion can generate goal images, existing methods for using these models in robotics face several issues:

  • Multi-step error accumulation: Current pipelines involve separate models for generation, segmentation, and feature encoding. Errors in one step can cascade and amplify in subsequent steps, leading to low success rates.
  • Limited perspective control: Most methods are restricted to 3-DoF (top-down) views, making it difficult to match real-world 6-DoF scenarios.
  • Distortions and unintended objects: Generative models can produce distorted images or include objects not specified in the prompt, further complicating segmentation and matching.

This paper aims to overcome these limitations by creating a more robust and flexible system for zero-shot scene rearrangement.

Methods

PACA is a training-free pipeline that integrates several key innovations:

  • Single-step Generation, Segmentation, and Feature Encoding: Instead of separate modules, PACA uses the inherent properties of diffusion models' denoising process. Specifically, it leverages cross-attention maps at different denoising timesteps. These maps implicitly capture both object regions (segmentation) and object features (encoding) as the image is generated. This is achieved by analyzing the correlation between textual prompts and image features during the denoising process.
  • Perspective Control for 6-DoF: To enable 6-DoF rearrangement, PACA introduces a perspective control mechanism. It uses Hough Line Transform on the real scene image to extract line information, which is then used to guide Stable Diffusion (via ControlNet) to generate goal images with a desired perspective. This allows for matching with real-world camera views beyond simple top-down perspectives.
  • Joint Cross-Attention Representation: The core of PACA's representation is a joint cross-attention representation that combines "conceptual embeddings" (object regions, captured at mid-denoising stages) and "perceptual embeddings" (object features, captured at later stages). This combined representation is designed to be robust to distortions and to effectively bridge the gap between real and generated scenes.
  • Object Matching and Robot Execution: The generated goal representations are matched with representations extracted from the real scene using an iterative process. The resulting object transformations are then used to guide the robot arm for execution.

Datasets

The paper utilizes web-scale generative models (Stable Diffusion) trained on large, diverse datasets. For evaluation, they use three custom scenes:

  • Dining Scene: Features knives, forks, spoons, and plates.
  • Fruit Scene: Features apples, oranges, and plates.
  • Office Scene: Features keyboards, mice, and mugs.

These scenes are used to create matching pairs for evaluating the system's performance. No new datasets are introduced.

Key Insights and Ideas

  • Diffusion Models as a Unified Representation Engine: The paper demonstrates that the denoising process of diffusion models, particularly the cross-attention maps generated at different timesteps, can implicitly encode object segmentation and features. This insight is crucial for unifying generation, segmentation, and encoding into a single step.
  • Perspective Control is Key for 6-DoF: The difficulty in controlling the viewpoint of generated images has been a major bottleneck for 6-DoF scene rearrangement. PACA's use of real-scene line information to guide generation effectively addresses this.
  • Categorical Prompts Outperform Specific Ones: User studies and quantitative evaluations suggest that using general object categories (e.g., "apple") in prompts leads to better image quality and prompt alignment compared to highly specific descriptions (e.g., "one apple"). This is attributed to the training data of large language models.
  • Joint Representation Mitigates Multi-step Errors: By integrating multiple steps, PACA avoids the error accumulation inherent in traditional multi-step pipelines, leading to significantly higher matching accuracy.

Key Results

  • Matching Accuracy: PACA achieves an average matching accuracy of 86.72% across three scenes, a substantial improvement over DALL-E-Bot's 51.35%.
  • Execution Success Rate: The system demonstrates an average execution success rate of 68.73% in 3-DoF settings and 65.56% in 6-DoF settings across various tasks.
  • User Satisfaction: PACA's generated goals are rated as satisfactory by users, comparable to human-generated goals.
  • Zero-Shot Performance: The pipeline operates in a zero-shot manner, meaning it does not require task-specific training for new objects or scenes.

Limitations and Future Work

  • User Personalization: While categorical prompts improve generation, they offer limited control over precise spatial relationships or the exact number of objects. Future work could explore energy-based models for finer control over generation.
  • 6-DoF Depth Estimation: The current 6-DoF implementation relies on external depth estimation (Marigold) for generated goals, which can introduce errors. Future work could integrate depth estimation directly into the representation construction.
  • Grasping Critical Information: For effective grasping, more critical information like the object's center of mass is needed, which is not fully provided by the current method.

Summary updated

BibTeX
@inproceedings{jin2025paca,
  author = {Shutong Jin and Ruiyu Wang and Kuangyi Chen and Florian T. Pokorny},
  title = {{PACA: Prespective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure