DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement

Benjamin A. Newman, Pranay Gupta, Kris Kitani, Yonatan Bisk, Henny Admoni, Christopher Jason PaxtonarXiv 2024 · OpenReview

Abstract

De gustibus non est disputandum ("there is no accounting for others' tastes") is a common Latin maxim describing how many solutions in life are determined by people's personal preferences. Many household tasks, in particular, can only be considered fully successful when they account for personal preferences such as the visual aesthetic of the scene. For example, setting a table could be optimized by arranging utensils according to traditional rules of Western table setting decorum, without considering the color, shape, or material of each object, but this may not be a completely satisfying solution for a given person. Toward this end, we present DegustaBot, an algorithm for visual preference learning that solves household multi-object rearrangement tasks according to personal preference. To do this, we use internet-scale pre-trained vision-and-language foundation models (VLMs) with novel zero-shot visual prompting techniques. To evaluate our method, we collect a large dataset of naturalistic personal preferences in a simulated table-setting task, and conduct a user study in order to develop two novel metrics for determining success based on personal preference. This is a challenging problem and we find that 50% of our model's predictions are likely to be found acceptable by at least 20% of people.

Introduction

In the field of assistive robotics, defining "success" has traditionally been tied to objective task completion. However, for many domestic chores, success is inherently subjective. A robot might successfully place a fork and knife on a table, but if the placement violates a user's specific aesthetic preference—such as a particular orientation or color coordination—the task is not fully successful. These preferences are often difficult to articulate in language and vary significantly across individuals, making them a major hurdle for generalization in robotics.

Previous approaches to personalized rearrangement relied on extensive data collection to train models on specific user behaviors. Such methods often struggle to generalize to new users or novel objects because the space of human preference is effectively unbounded. DegustaBot addresses this by leveraging the latent "common sense" and spatial reasoning capabilities of large-scale Vision-Language Models (VLMs) like GPT-4o. By treating preference estimation as an in-context learning problem, DegustaBot can infer a user's style from just a few visual examples in a zero-shot manner.

Methodology: Visual Prompting and DegustaBot

The core innovation of DegustaBot is its use of Visual Prompting to bridge the gap between abstract preferences and spatial coordinates. The framework "lifts" non-visual information into the image domain, allowing the VLM to process the entire context visually. The authors explored four primary configurations to determine the most effective way to communicate spatial data to the model:

  • LOUMA (Language Objects, Unmarked Arrangements): Objects are described via JSON metadata; the table is presented as a plain image.
  • LOGMA (Language Objects, Grid-Marked Arrangements): Objects are described in JSON; the table includes a spatial grid overlay to assist the VLM in identifying coordinates.
  • MOUMA (Marked Objects, Unmarked Arrangements): Objects are shown in a visual grid with reference markers; the table is a plain image.
  • MOGMA (Marked Objects, Grid-Marked Arrangements): Both the objects and the table utilize visual grids and markers.
Visual prompting configurations including language-based object descriptions (OaL) and grid-marked arrangements (GMA).

By overlaying a grid (GMA), the researchers transformed a difficult continuous regression problem (predicting exact coordinates) into a discrete multiple-choice selection task. To evaluate the accuracy of predicted arrangements, the system employs the Kabsch algorithm to calculate the Root Mean Squared Deviation (RMSD). This metric accounts for global shifts or rotations, focusing on the internal geometric consistency of the arrangement relative to the user's preference.

Experimental Setup and Datasets

To rigorously test DegustaBot, the researchers utilized two distinct datasets:

  1. Simulated Preferences: A controlled set of 36 arrangements across two table types, encompassing standard, semi-standard, and non-standard settings with varying levels of positional and rotational noise.
  2. Naturalistic Tabletop Arrangement Dataset: A major contribution of this work, this dataset contains 995 arrangements from 199 human participants. Collected via an online study, it captures the vast diversity of real-world human preferences. Participants performed "drag-and-drop" tasks to set tables and then rated "jittered" versions of their own settings to establish a threshold for "acceptability."

Results and Key Insights

The experiments revealed several critical findings regarding the intersection of VLMs and robotics. GPT-4o emerged as the superior model, significantly outperforming Claude-3 Haiku and Gemini 1.5 Pro in both object selection and geometric precision.

The DegustaBot pipeline: inputting a sequence of preferred arrangements to predict a personalized final state.

Key insights include:

  • Visual Grounding is Essential: The "MOGMA" approach, which relies entirely on visual markers rather than text-based JSON descriptions, consistently yielded the best results. This suggests that VLMs reason more effectively about spatial tasks when the input is purely visual.
  • Sensitivity to Precision: The user study highlighted that humans are extremely sensitive to small placement errors. A minor "jitter" in a utensil's position often rendered a scene unacceptable to the user, setting a high bar for robotic precision.
  • Zero-Shot Generalization: On the naturalistic dataset, the GPT-4o MOGMA model produced arrangements that 50% of participants would likely find acceptable, demonstrating a remarkable ability to generalize to new users based on a minimal history of 2-4 example images.

Conclusion

DegustaBot demonstrates that internet-scale foundation models possess latent aesthetic priors that can be successfully harnessed for personalized robotics. While currently limited to simulated environments, the framework provides a scalable path toward robots that can adapt to the unique "tastes" of their human partners without requiring massive, task-specific retraining. Future work will focus on deploying these models on physical hardware and fine-tuning VLMs specifically for fine-grained spatial rearrangement tasks.

Summary updated

Abstract

De gustibus non est disputandum ("there is no accounting for others' tastes") is a common Latin maxim describing how many solutions in life are determined by people's personal preferences. Many household tasks, in particular, can only be considered fully successful when they account for personal preferences such as the visual aesthetic of the scene. For example, setting a table could be optimized by arranging utensils according to traditional rules of Western table setting decorum, without considering the color, shape, or material of each object, but this may not be a completely satisfying solution for a given person. Toward this end, we present DegustaBot, an algorithm for visual preference learning that solves household multi-object rearrangement tasks according to personal preference. To do this, we use internet-scale pre-trained vision-and-language foundation models (VLMs) with novel zero-shot visual prompting techniques. To evaluate our method, we collect a large dataset of naturalistic personal preferences in a simulated table-setting task, and conduct a user study in order to develop two novel metrics for determining success based on personal preference. This is a challenging problem and we find that 50% of our model's predictions are likely to be found acceptable by at least 20% of people.
BibTeX
@misc{newman2024degustabot,
  author = {Benjamin A. Newman and Pranay Gupta and Kris Kitani and Yonatan Bisk and Henny Admoni and Christopher Jason Paxton},
  title = {{DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement}},
  year = {2024},
  eprint = {2407.08876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.08876},
}
Figure