Abstract
Robotic household rearrangement is a complex task requiring both objective common sense and subjective user preference. This paper introduces a novel framework that mines these preferences directly from the existing state of a home. By leveraging Large Multimodal Models (LMMs) like GPT-4V to transform standard scene graphs into Affordance-Enhanced Graphs (AEGs), the system identifies misplaced objects and determines ideal locations without explicit human intervention or extensive training examples.
Introduction
Robotic "tidying up" is deceptively complex. It requires two distinct types of knowledge: objective common sense (e.g., "milk belongs in the fridge") and subjective user preference (e.g., "I prefer to use this trunk as a coffee table, not for storage").
Existing methods typically fall into two traps: they either rely on rigid, pre-defined rules that ignore personal taste, or they require users to provide numerous "exemplar" placements to teach the robot their preferences. The authors argue that a home’s existing layout already encapsulates these preferences. The challenge lies in "mining" this implicit functionality and context from a 3D scan of a room so a robot can make intelligent decisions in a zero-shot manner.
Methodology
The proposed multi-stage pipeline moves from raw visual data to actionable rearrangement plans:
- Scene Graph Extraction: The robot performs an RGB-D scan to reconstruct the scene and extracts a "vanilla" scene graph where nodes are objects and edges represent spatial relationships.
- Context-Induced Affordance Analysis: Using GPT-4V, the system performs local analysis (assigning fine-grained categories like "Entryway Organizing Console") and global analysis (aggregating information across hierarchies to understand non-local functional relationships).
- Affordance-Enhanced Graph (AEG): The vanilla graph is updated with high-level semantic descriptions and new "functional edges."
- Misplacement Detection: An LLM-based scorer evaluates each object's position against the AEG’s affordances. Objects scoring below a threshold are flagged for relocation.
- Rearrangement Planning (RAG-inspired): To find a new home for an object, the system uses a Retrieval-Augmented Generation (RAG) approach, querying a database of potential receptacles and using an LLM to make the final placement decision based on retrieved context.
Evaluation and Results
The researchers evaluated their method using two primary benchmarks:
- TidyBot Benchmark: In 96 scenarios across four room types, the method achieved a 91.4% average success rate, slightly outperforming the original TidyBot (90.7%) despite not being trained on specific user examples.
- HSSD-200 (Context-Oriented Benchmark): A new benchmark featuring 8 multi-room scenes and over 4,000 "messy" configurations. In misplacement detection, the method achieved an F1 score of 0.894, significantly higher than the baseline Housekeep (0.764).
Key Insights and Discussion
- The Scene is the Prompt: Instead of direct user queries, the model looks at object placement and room layout to infer functionality (e.g., identifying a specific table as an "entryway console").
- Non-Local Relationships: The AEG discovers "functional" edges—recognizing that the relationship between a remote control and a TV is vital for placement, even if they are physically distant.
- Efficiency via Retrieval: Using a RAG-style retriever to filter for the top-3 or top-4 receptacle candidates significantly improves the LLM’s reasoning accuracy by preventing information flooding.
Limitations and Future Work
The system is dependent on the quality of the initial 3D reconstruction and scene graph. Future work aims to explore joint optimization, where the LLM can actively request the robot to re-scan areas if context is insufficient, and end-to-end active learning to refine scene graph construction.
Summary updated