TL;DR
When assisting humans in their homes, embodied AI agents shouldn’t need tedious step-by-step instructions for everyday tasks—they should infer personal habits from just a few observations. This paper introduces the Preference-based Planning (PBP) benchmark and demonstrates that framing personalized planning as a two-stage pipeline (first predicting an explicit preference representation, then planning conditioned on it) dramatically outperforms end-to-end multimodal planning.
What Problem Does This Paper Address?
Natural language commands like "Help me prepare an apple" are inherently underspecified. Does the user like their apple peeled? Sliced? Washed in the sink or a bowl? Placed on a specific plate?
Current embodied AI systems typically either assume universal, "one-size-fits-all" commonsense behaviors or require exhaustive, repetitive back-and-forth prompting. In reality, human preferences are personalized, often subconscious, and rarely articulated verbally. However, humans can easily infer preferences from watching just one or two demonstrations.
This paper tackles few-shot preference learning and preference-guided embodied planning: enabling an agent to watch a handful of user demonstrations, infer the underlying personal preference, and immediately adapt its action plans in novel environments and setups.
Methods
The authors formulate the task as few-shot learning from demonstration and systematically compare two paradigms:
- End-to-End Planning: Feeding few-shot demonstration trajectories (egocentric video or symbolic action logs) directly into a foundation model alongside the current state, prompting it to generate the full personalized action plan in one pass.
- Two-Stage Learning-and-Planning:
- Stage 1 (Preference Inference): The agent abstracts the user’s implicit behavior into an explicit preference representation ($p = f(O)$).
- Stage 2 (Conditioned Planning): The agent uses the inferred preference token alongside the current scene state to generate a customized sequence of primitive actions ($a \sim g(s, p)$).
The authors evaluate both vision-based models (ViViT, LLaVA-NeXT, EILEV, GPT-4V) and symbol-based models that receive symbolic action logs (DAG-Opt / NOTEARS, Llama-3-8B, DeepSeek-R1, GPT-4.1).
Datasets
The authors introduce the Preference-based Planning (PBP) benchmark:
- Simulation Environment: Built atop NVIDIA Omniverse and OmniGibson across 50 detailed indoor scenes.
- Preference Hierarchy: 290 unique preferences structured into three tiers:
- Action Level (75 preferences): Fine-grained execution parameters (e.g., water fill amount, cooking doneness).
- Option Level (135 preferences): Alternative approaches to sub-tasks (e.g., storing fruit in the fridge vs. on the table).
- Sequence Level (80 preferences): High-level task prioritization and order constraints (e.g., cleaning before cooking vs. after).
- Scale: A reusable pool of 15,000 demonstration recordings (egocentric video, third-person views, bird's-eye maps, and frame-level annotations across 17 action primitives) with a standardized 5,000-instance few-shot test set.
Key Insights and Ideas
- End-to-End Planning Fails Dramatically: State-of-the-art vision-language models struggle to map demonstration videos straight to execution plans; their action sequence errors (Levenshtein distances) hover close to the length of the ground-truth sequences themselves. Models memorize isolated low-level actions rather than discovering cohesive goals.
- Preferences Serve as a Crucial Abstraction Layer: Decomposing the problem into explicit preference classification followed by preference-conditioned planning yields massive gains across every model family.
- Vision-Language Models Have an Abstraction Bottleneck: Vision models like GPT-4V and LLaVA-NeXT struggle to infer preferences from video observations alone (GPT-4V achieves only 42.99% overall accuracy), yet they plan remarkably well once an explicit preference label is provided.
- Visual Overfitting vs. Symbolic Invariance: In generalization tests across different rooms and objects, vision-based models suffer significant performance drops because they overfit to superficial visual layouts. In contrast, symbol-based LLMs generalize robustly.
Key Results
- Planning Accuracy (Levenshtein Distance):
- End-to-end: Video-based models averaged Levenshtein distances of 22.99–25.35 tokens (against ground-truth lengths of ~15.8 at option level and ~35.9 at sequence level).
- Two-stage: Decoupling inference and planning lowered overall edit distance significantly (GPT-4V dropped from 24.69 to 17.94; GPT-4.1 dropped from 16.95 to 8.22). When supplied with ground-truth preferences, GPT-4.1 achieved a near-perfect distance of 5.23.
- Preference Prediction Accuracy:
- LLMs operating over symbolic action logs led the pack (GPT-4.1: 79.60%, DeepSeek-R1: 78.62%, Llama-3-8B: 70.08%).
- Multimodal video models lagged behind (GPT-4V: 42.99%, EILEV: 35.51%, LLaVA-NeXT: 30.86%, ViViT: 6.81%).
- Few-Shot Scaling: Increasing demonstration quantity from 1 to 3 steadily improved preference identification and downstream execution, though performance saturated around 5 demonstrations.
Limitations and Future Work
- Synthetic Domain Gap: All demonstrations and scenes are rendered synthetically in Omniverse/OmniGibson, which lacks real-world messiness, sensor noise, and physics variability.
- Predefined Categorical Labels: Real human preferences are continuous, compound, and often context-dependent; the current benchmark relies on a discrete set of 290 predefined preference templates.
- Future Directions: The authors plan to collect natural, real-world few-shot demonstrations using egocentric wearable hardware (e.g., smart glasses) to bridge the sim-to-real gap.
Summary updated