Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects

Junjia Liu, Yiting Chen, Zhipeng Dong, Shixiong Wang, Sylvain Calinon, Miao Li, Fei ChenIEEE RA-L 2022

Introduction

Developing a robotic chef capable of stir-frying represents a "grand challenge" in robotics. Unlike moving rigid objects, stir-frying involves the manipulation of "semi-fluid" materials—mixtures of solids and liquids—that are notoriously difficult to model. The task is inherently bimanual, requiring two arms to work in precise synchronization, and non-prehensile, as the food is tossed and rolled rather than grasped. Furthermore, the highly dynamic nature of these rhythmic movements makes traditional step-by-step planning ineffective.

Bimanual robot system performing non-prehensile stir-fry manipulation of semi-fluid food objects.

Existing coordination methods often rely on rigid kinematic coupling, which fails in this context because the tools (wok and spatula) interact only indirectly through the food medium. This research proposes a framework that treats bimanual coordination as a spatio-temporal relationship modeling problem.

The Leader-Follower Framework

To address the complexity of bimanual coordination, the researchers developed a decoupled "Leader-Follower" framework. This approach simplifies the learning process by assigning distinct roles to each arm:

  • The Leader (Wok): The left arm controls the wok using a canonical rhythmic pattern (pushing, rotating, and pulling). This is modeled via Dynamic Movement Primitives (DMPs), which allow the robot to learn a trajectory from human demonstrations and adjust its speed or spatial scale while maintaining the motion's fundamental shape.
  • The Follower (Spatula): The right arm's motion is more variable and must be coordinated with the wok. This is handled by a novel Structured-Transformer that treats coordination as a sequence transduction problem—effectively "translating" the leader's motion sequence into the appropriate follower sequence over time.

System Architecture and Training

The system integrates human demonstration data with a robust inference pipeline. Human chefs performed stir-fry motions while wearing Xsens motion capture suits to provide ground truth data.

The system pipeline: from human demonstration and DMP modeling to Structured-Transformer inference and visual feedback control.

As shown in the architecture, the Structured-Transformer utilizes a Graph Convolutional Layer to explicitly account for the robot's hardware structure (the spatial relationships between joints). Additionally, a top-down camera provides a visual feedback loop; if food displacement is suboptimal, a PD controller adjusts the leader arm's DMP parameters in real-time to improve tossing effectiveness.

Results and Generalization

The framework was validated both in a PyBullet simulation and on a real-world dual Franka Panda robot system. Key findings include:

  • High Fidelity: In simulation, the Structured-Transformer achieved a normalized Dynamic Time Warping (DTW) value of 0.003, indicating spatula motions nearly identical to human experts.
  • Real-World Success: The physical robot successfully adjusted its tossing motion based on visual feedback to achieve desired food displacement.
  • Generalization: The combination of DMPs and the Transformer's attention mechanism allowed the model to generalize to motion speeds and scales not present in the original training data.

Conclusion and Future Directions

While the Leader-Follower framework successfully demonstrates complex bimanual manipulation, several avenues for improvement remain. Current models rely on pose data and lack the tactile force sensing critical for professional-grade stir-frying. Furthermore, the vision system currently uses 2D segmentation, which does not capture the 3D "loft" of food during a toss. Future work will focus on integrating force feedback, improving 3D state estimation, and implementing spillage prevention mechanisms to handle more diverse and sticky ingredients.

Summary updated

BibTeX
@article{liu2022robot,
  author = {Junjia Liu and Yiting Chen and Zhipeng Dong and Shixiong Wang and Sylvain Calinon and Miao Li and Fei Chen},
  title = {{Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
}
Figure