Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items on a Plate

Ryan Feng, Youngsun Kim, Gilwoo Lee, Ethan K. Gordon, Matt Schmittle, Shivaum Kumar, Tapomayukh Bhattacharjee, Siddhartha S. Srinivasa

Abstract

Assisting individuals with physical disabilities during mealtime is a critical application for assistive robotics. However, the vast diversity of food items—varying in shape, texture, and compliance—presents a significant challenge for traditional, item-specific robotic strategies. This paper introduces SPANet (Skewering-Position-Action Network), a data-driven framework designed to generalize skewering strategies to novel food items. By mapping visual features and environmental context to a discrete set of high-level actions, the system achieves robust performance on both seen and unseen food items in realistic, cluttered plate settings.

Introduction

Feeding is a fundamental activity of daily living (ADL), yet over one million adults in the United States require assistance to eat. Robotic systems offer a path toward greater independence, but they must navigate the "universe of food," which is effectively infinite in its variety. Previous approaches often relied on rigid, object-specific rules (e.g., "if carrot, use carrot-strategy"), which fail when encountering unfamiliar items like a parsnip or a uniquely cut fruit.

The robot-assisted feeding system mounted on a wheelchair, featuring the plate setup and a sequence of a successful skewering action.

To address this, we propose a shift from identity-based manipulation to physics-based generalization. The goal is to enable a robot to look at an unfamiliar object and intelligently predict the most effective skewering motion based on its physical appearance and its immediate surroundings.

Methodology

Discrete Action Space

To facilitate generalization, the continuous space of possible robot poses is reduced to 6 discrete actions based on three macro-strategies:

  • Vertical Skewer (VS): The fork handle is held vertically, and the robot moves straight down.
  • Tilted Vertical (TV): The fork handle is tilted such that the tines themselves enter the food vertically.
  • Tilted Angled (TA): The fork approaches diagonally with horizontal tines, a strategy particularly effective for soft foods like bananas.

Each macro-action is evaluated at two different fork rolls ($0^\circ$ and $90^\circ$) relative to the food item's major axis, providing a manageable yet expressive action library.

The SPANet Pipeline

The system employs a multi-stage computer vision and machine learning pipeline to select the optimal bite:

  1. Detection: A RetinaNet-based detector identifies individual food items on the plate.
  2. Environment Classification: Using Hough transforms and height maps, the system categorizes the surroundings of each item into three states: Isolated (ISO), Wall (near the plate edge or another item), or Stacked.
  3. Action Prediction (SPANet): The core model takes a cropped RGB image of the target food and its environment encoding. It outputs the coordinates for the skewering location and predicted success probabilities for each of the six actions.
The SPANet pipeline: from raw plate image to bounding boxes, environment classification, and the final action score map.

Empirical Data and Training

Unlike previous datasets that relied on human intuition for labeling, this work utilizes the Robot Bite Acquisition Dataset. This dataset contains 2,450 trials across 16 different food items, recording actual empirical success or failure. This ensures the model learns from the physical reality of robot-food interaction rather than human assumptions. For final integration, the dataset was augmented to approximately 10,000 images to improve robustness.

Key Insights and Results

Physics over Identity

A central finding is that the robot does not need to identify the food (e.g., "broccoli") to succeed. Instead, it learns that items with similar visual and physical properties—such as "leafy" textures or "long and slender" shapes—respond similarly to specific actions.

The Importance of Context

The environment significantly influences success rates. For instance, the Tilted Angled (TA) strategy, which tends to push food away, becomes highly effective when the food is against a "Wall" (the plate edge), as the edge provides the necessary counter-force for the tines to penetrate.

Performance Metrics

In real-world tests on cluttered plates, the integrated system achieved:

  • 83% success rate for "In-Class" (previously seen) items.
  • 91% success rate for "Out-Of-Class" (unseen) items.

While the higher success on unseen items was partly due to the specific test items being physically easier to skewer (e.g., avoiding slippery grapes), the results demonstrate that SPANet effectively generalizes its learned strategies to novel objects.

Discussion and Future Work

While SPANet represents a significant step forward, challenges remain. The system occasionally struggles with "deceptive" textures, such as items with a hard core but soft skin, which are difficult to assess via vision alone. Future iterations could incorporate haptic feedback during the approach to adjust strategies in real-time. Additionally, expanding the action library to include scooping or twirling would allow the system to handle a wider variety of food types, such as peas or pasta, further enhancing the autonomy and utility of assistive feeding robots.

Summary updated

BibTeX
@inproceedings{feng2019robot,
  author = {Ryan Feng and Youngsun Kim and Gilwoo Lee and Ethan K. Gordon and Matt Schmittle and Shivaum Kumar and Tapomayukh Bhattacharjee and Siddhartha S. Srinivasa},
  title = {{Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items on a Plate}},
  booktitle = {ISRR},
  year = {2019},
  url = {https://openreview.net/forum?id=SxbKOhNG7q},
}
Figure