Robot-Assisted Feeding: Generalizing Skewering Strategies across Food Items on a Realistic Plate

Ryan Feng, Youngsun Kim, Gilwoo Lee, Ethan K. Gordon, Matt Schmittle, Shivaum Kumar, Tapomayukh Bhattacharjee, Siddhartha S. Srinivasa

TL;DR

This paper tackles the challenge of robot-assisted feeding by developing a system that can generalize skewering strategies to new food items. By analyzing how food properties and environment affect success, they created a network (SPANet) that predicts optimal bite acquisition actions, significantly improving the robot's ability to feed autonomously.

What Problem Does This Paper Address?

Feeding assistance for individuals who have lost the ability to feed themselves is a critical area for improving quality of life and reducing caregiver burden. A key hurdle in automating this task is the sheer diversity of food items and their physical properties. Unlike rigid objects, food is deformable, comes in countless shapes and textures, and its behavior can change during the acquisition process. This variability makes it difficult for robots to learn a fixed set of manipulation strategies that work for all foods. Existing methods often rely on classifying food items and manually defining strategies, which fails to generalize to unseen foods. The authors aim to bridge this gap by enabling a robot to learn and adapt its feeding strategies to novel food items without explicit prior knowledge.

Methods

The core of the proposed solution is the Skewering-Position-Action Network (SPANet). SPANet is a deep learning model designed to predict the success probability of different bite acquisition actions for a given food item.

Here's a breakdown of their approach:

  • Data-Driven Insight: The central idea is that food items with similar physical properties will have similar success rates for a given action. This allows for generalization.
  • Action Space Simplification: Instead of a vast number of discrete actions, they define a reduced set of six "macro actions" based on fork pitch (vertical skewer, tilted vertical skewer, tilted angled skewer) and fork roll (0° and 90°). This simplification is crucial for generalization.
  • Environment Features: They identify three key environmental contexts for a food item: "Isolated" (alone on the plate), "Wall" (near the edge of the plate or another item), and "Stack" (on top of another item). These features are extracted using a computer vision pipeline involving depth and RGB data.
  • SPANet Architecture: SPANet takes a cropped RGB image of a food item and its environment classification as input. It uses convolutional layers to extract visual features and then concatenates these with the environment encoding. The output is a vector predicting the success rate for each of the six macro actions and the two endpoints of the food item's major axis (for determining the skewering direction).
  • Bite Acquisition Framework: The system integrates SPANet with a robotic arm and a perception pipeline. RetinaNet is used for initial food item detection and segmentation. SPANet then predicts the best action and target for each item. A motion planner generates the trajectory, and a closed-loop controller with force-torque feedback executes the action.

Datasets

  • Newly Introduced Dataset: The authors collected a novel dataset of 2450 robot bite acquisition trials involving 16 different food items across various environmental conditions. This dataset is crucial for training and evaluating SPANet. It captures success rates for specific actions and food item/environment combinations. The dataset is made publicly available.

Key Insights and Ideas

  • Physical Properties as a Generalization Key: The fundamental insight is that physical similarity between food items is a strong predictor of action success, enabling generalization beyond explicitly trained items.
  • Environment Matters: The surrounding environment of a food item significantly impacts the success of a bite acquisition attempt. This highlights the need for context-aware strategies.
  • Action Space Abstraction: Reducing the action space to a few macro actions, rather than highly specific poses, is key to achieving generalization to unseen food items.
  • Data-Driven Success Prediction: Moving away from human-labeled strategies to empirically derived success rates from robot trials provides a more accurate and robust basis for learning.

Key Results

  • Generalization Performance: SPANet demonstrated good generalization to unseen food items, achieving a 18% drop in success rate for Out-Of-Class items compared to In-Class items. For many food categories, the performance was very close to In-Class predictions.
  • Action Prediction Accuracy: SPANet's predicted action success rates closely matched empirical results, correctly identifying the best actions for most food-environment combinations. The proposed action had only a 2.6% lower success rate than the ground-truth best action for In-Class items and a 20% lower rate for Out-Of-Class items.
  • Full System Integration: The integrated system achieved high success rates on cluttered plates, acquiring 91% of Out-Of-Class items in experiments.
  • Environmental Impact: Statistical analysis confirmed that the "stacked" environment significantly impacted bite acquisition success (p-value of 0.0005).
  • Action Impact: The "tilted-angled" (TA) action showed statistically significant differences in success rates compared to vertical actions (p-value of 0.0005). Fork roll also showed a preference for certain item shapes (e.g., 90° for long, flat items).

Limitations and Future Work

  • Slipperiness and Deformability: The authors acknowledge that highly slippery or deformable foods (like bananas) pose a challenge, as their action distributions differ significantly and are hard to capture with visual data alone. This is identified as a key area for future work.
  • Action Space Sufficiency: While the reduced action space aids generalization, it's questioned whether these six actions are sufficient for a wider range of food items. Future work could explore adding actions like scooping or twirling.
  • Robustness to Anomalies: The system's robustness can be improved to handle planning and perception anomalies, such as partially observable objects or cluttered environments that require pre-skewer manipulation.
  • Fork Stability: The variability in fork shape and position due to physical interactions is a limitation. Stabilizing the fork or improving its design is suggested.

Summary updated

Abstract

A robot-assisted feeding system must successfully acquire many different food items. A key challenge is the wide variation in the physical properties of food, demanding diverse acquisition strategies that are also capable of adapting to previously unseen items. Our key insight is that items with similar physical properties will exhibit similar success rates across an action space, allowing the robot to generalize its actions to previously unseen items. To better understand which skewering strategy works best for each food item, we collected a dataset of 2450 robot bite acquisition trials for 16 food items with varying properties. Analyzing the dataset provided insights into how the food items' surrounding environment, fork pitch, and fork roll angles affect bite acquisition success. We then developed a bite acquisition framework that takes the image of a full plate as an input, segments it into food items, and then applies our Skewering-Position-Action network (SPANet) to choose a target food item and a corresponding action so that the bite acquisition success rate is maximized. SPANet also uses the surrounding environment features of food items to predict action success rates. We used this framework to perform multiple experiments on uncluttered and cluttered plates. Results indicate that our integrated system can successfully generalize skewering strategies to many previously unseen food items.
BibTeX
@inproceedings{feng2019robot,
  author = {Ryan Feng and Youngsun Kim and Gilwoo Lee and Ethan K. Gordon and Matt Schmittle and Shivaum Kumar and Tapomayukh Bhattacharjee and Siddhartha S. Srinivasa},
  title = {{Robot-Assisted Feeding: Generalizing Skewering Strategies across Food Items on a Realistic Plate}},
  booktitle = {ISRR},
  year = {2019},
  url = {https://openreview.net/forum?id=SxbKOhNG7q},
}
Figure