Abstract
Assisting individuals with physical disabilities during mealtime is a critical application for assistive robotics. However, the vast diversity of food items—varying in shape, texture, and compliance—presents a significant challenge for traditional, item-specific robotic strategies. This paper introduces SPANet (Skewering-Position-Action Network), a data-driven framework designed to generalize skewering strategies to novel food items. By mapping visual features and environmental context to a discrete set of high-level actions, the system achieves robust performance on both seen and unseen food items in realistic, cluttered plate settings.
Introduction
Feeding is a fundamental activity of daily living (ADL), yet over one million adults in the United States require assistance to eat. Robotic systems offer a path toward greater independence, but they must navigate the "universe of food," which is effectively infinite in its variety. Previous approaches often relied on rigid, object-specific rules (e.g., "if carrot, use carrot-strategy"), which fail when encountering unfamiliar items like a parsnip or a uniquely cut fruit.
To address this, we propose a shift from identity-based manipulation to physics-based generalization. The goal is to enable a robot to look at an unfamiliar object and intelligently predict the most effective skewering motion based on its physical appearance and its immediate surroundings.
Methodology
Discrete Action Space
To facilitate generalization, the continuous space of possible robot poses is reduced to 6 discrete actions based on three macro-strategies:
- Vertical Skewer (VS): The fork handle is held vertically, and the robot moves straight down.
- Tilted Vertical (TV): The fork handle is tilted such that the tines themselves enter the food vertically.
- Tilted Angled (TA): The fork approaches diagonally with horizontal tines, a strategy particularly effective for soft foods like bananas.
Each macro-action is evaluated at two different fork rolls ($0^\circ$ and $90^\circ$) relative to the food item's major axis, providing a manageable yet expressive action library.
The SPANet Pipeline
The system employs a multi-stage computer vision and machine learning pipeline to select the optimal bite:
- Detection: A RetinaNet-based detector identifies individual food items on the plate.
- Environment Classification: Using Hough transforms and height maps, the system categorizes the surroundings of each item into three states: Isolated (ISO), Wall (near the plate edge or another item), or Stacked.
- Action Prediction (SPANet): The core model takes a cropped RGB image of the target food and its environment encoding. It outputs the coordinates for the skewering location and predicted success probabilities for each of the six actions.
Empirical Data and Training
Unlike previous datasets that relied on human intuition for labeling, this work utilizes the Robot Bite Acquisition Dataset. This dataset contains 2,450 trials across 16 different food items, recording actual empirical success or failure. This ensures the model learns from the physical reality of robot-food interaction rather than human assumptions. For final integration, the dataset was augmented to approximately 10,000 images to improve robustness.
Key Insights and Results
Physics over Identity
A central finding is that the robot does not need to identify the food (e.g., "broccoli") to succeed. Instead, it learns that items with similar visual and physical properties—such as "leafy" textures or "long and slender" shapes—respond similarly to specific actions.
The Importance of Context
The environment significantly influences success rates. For instance, the Tilted Angled (TA) strategy, which tends to push food away, becomes highly effective when the food is against a "Wall" (the plate edge), as the edge provides the necessary counter-force for the tines to penetrate.
Performance Metrics
In real-world tests on cluttered plates, the integrated system achieved:
- 83% success rate for "In-Class" (previously seen) items.
- 91% success rate for "Out-Of-Class" (unseen) items.
While the higher success on unseen items was partly due to the specific test items being physically easier to skewer (e.g., avoiding slippery grapes), the results demonstrate that SPANet effectively generalizes its learned strategies to novel objects.
Discussion and Future Work
While SPANet represents a significant step forward, challenges remain. The system occasionally struggles with "deceptive" textures, such as items with a hard core but soft skin, which are difficult to assess via vision alone. Future iterations could incorporate haptic feedback during the approach to adjust strategies in real-time. Additionally, expanding the action library to include scooping or twirling would allow the system to handle a wider variety of food types, such as peas or pasta, further enhancing the autonomy and utility of assistive feeding robots.
Summary updated