Abstract
Autonomous robot-assisted feeding requires the ability to acquire a wide variety of food items. However, it is impossible for such a system to be trained on all types of food in existence. Therefore, a key challenge is choosing a manipulation strategy for a previously unseen food item. Previous work showed that the problem can be represented as a linear bandit with visual context. However, food has a wide variety of multi-modal properties relevant to manipulation that can be hard to distinguish visually. Our key insight is that we can leverage the haptic context we collect during and after manipulation (i.e., "post hoc") to learn some of these properties and more quickly adapt our visual model to previously unseen food. In general, we propose a modified linear contextual bandit framework augmented with post hoc context observed after action selection to empirically increase learning speed and reduce cumulative regret. Experiments on synthetic data demonstrate that this effect is more pronounced when the dimensionality of the context is large relative to the post hoc context or when the post hoc context model is particularly easy to learn. Finally, we apply this framework to the bite acquisition problem and demonstrate the acquisition of 8 previously unseen types of food with 21% fewer failures across 64 attempts.
Introduction
In the domain of assistive robotics, "bite acquisition"—the successful skewering of food—presents a significant challenge due to the immense variety of food textures and physical properties. A ripe banana and an unripe one may appear identical to a camera but require vastly different physical strategies. Traditionally, this is modeled as a contextual bandit problem: the robot observes visual context, selects an action (skewering strategy), and receives a reward (success or failure).
The primary bottleneck in this standard approach is visual ambiguity. The robot must often fail repeatedly to learn the mapping between high-dimensional visual pixels and successful outcomes, leading to high cumulative regret. This paper introduces the concept of post hoc context—information gained after an action is taken—to accelerate the learning process.
Methods: The Post Hoc Augmented Bandit
The core innovation is a modified linear contextual bandit framework that utilizes multi-modal feedback collected during and after an action to improve the predictive visual model.
The pipeline operates as follows:
- Visual Context: The robot observes the food via an RGBD camera, processed through a featurizer (SPANet) to generate a visual context vector.
- Action Selection & Execution: The robot selects a strategy from a discrete set (e.g., vertical skewer). During execution, a force sensor collects haptic data (force and torque).
- Haptic Classification: The haptic data is passed through HapticNet, which classifies physical properties such as "soft" or "hard-skin."
- Joint Optimization: The researchers modified the LinUCB algorithm to include a hard constraint: the visual model and the haptic model must agree on the expected outcome. By aligning the high-dimensional visual model with the lower-dimensional, easier-to-learn haptic model, the system converges on the optimal strategy much faster.
Experiments and Datasets
The framework was validated through three distinct experimental setups:
- Synthetic Dimensionality Test: Controlled simulations varying context vector sizes to analyze the impact of post hoc information on complex problems.
- MNIST Simulation: Using handwritten digits where the image serves as visual context and the digit label acts as "perfect" post hoc context.
- Real-World Robot Trials:
- Offline Phase: 115 samples of skewering common fruits (grapes, strawberries, bananas) for hyperparameter tuning.
- Online Phase: 64 attempts on 8 previously unseen food types, including kiwi, celery, and cantaloupe.
Key Results and Insights
The experiments demonstrated that haptic data serves as a powerful "shortcut" for learning. It is significantly easier for a model to learn that "soft objects require tilted skewers" than to map complex visual patterns to that same conclusion.
- Performance Gains: In real-world tests, the post hoc augmented learner reduced failures by 21% compared to a vision-only baseline (averaging 22 failures vs. 27.67).
- Rapid Convergence: In MNIST simulations, the augmented model reached optimal error rates in ~500 samples, whereas the baseline failed to plateau even after 2,000 samples.
- Dimensionality Advantage: The benefit of post hoc context is most significant when the primary context (vision) is high-dimensional relative to the post hoc context (haptic categories).
Conclusion
This work demonstrates that leveraging information available only after an action can significantly improve learning efficiency in bandit settings. While currently limited to discrete action spaces and assuming action-independent post hoc context, future work aims to extend this to continuous actions and latent haptic spaces. This approach brings us closer to versatile, adaptive assistive robots capable of handling the unpredictable variety of daily life.
Citation: Gordon, E. K., et al. (2021). "Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding." arXiv:2011.02604v2.
Summary updated