Encoding Multiple Sensor Data for Robotic Learning Skills From Multimodal Demonstration

Chao Zeng, Chenguang Yang, Junpei Zhong, Jianwei ZhangIEEE Access 2019 · DOI

Introduction

Industrial robots excel at kinematic precision in free space but often struggle with contact-rich tasks—such as pushing objects, scrubbing surfaces, or pressing buttons—where managing interaction forces is critical. Traditional programming for these tasks is labor-intensive and rigid. While Teaching-by-Demonstration (TbD) allows non-experts to guide robots, most systems capture only the trajectory, ignoring the dynamic "human touch"—the adaptive stiffness and force profiles required for stable interaction.

This research introduces a multimodal TbD framework that enables robots to learn tasks requiring both precise motion and force control. By encoding human muscle activity (EMG) and interaction forces into a Hidden semi-Markov Model (HSMM), the researchers enabled a Baxter robot to mimic the nuanced, adaptive stiffness of a human tutor.

Methodology

The proposed architecture is divided into two primary phases: Learning and Reproduction.

Learning Phase

During human demonstration, the system records three distinct data streams:

  1. Motion: The trajectory and velocity of the robot's endpoint.
  2. Force: Data from a wrist-mounted force/torque sensor.
  3. Stiffness: Estimated in real-time from the tutor's muscle activity using a MYO electromyography (EMG) armband.

Reproduction Phase

To perform the task, the robot uses Gaussian Mixture Regression (GMR) to retrieve expected control variables based on its current position. These variables are fed into a joint-space variable impedance controller, allowing the robot to dynamically adjust its "squishiness" and pushing force.

System architecture for multimodal learning and reproduction.

Multimodal Encoding

A key innovation of this work is the use of parallel encoding. The researchers utilize a Hidden semi-Markov Model (HSMM) to capture the temporal duration of task stages more effectively than standard HMMs. Three parallel models are trained to correlate the robot's position ($x$) with its velocity ($\dot{x}$), stiffness ($k$), and force ($F$).

Probabilistic encoding of multimodal sensor data indexed to position.

This approach allows for the individual optimization of each parameter, mimicking human motor learning where grip strength can be refined independently of arm speed. Using EMG as a proxy for stiffness eliminates the need for complex mathematical optimization or trial-and-error.

Experimental Setup

The framework was validated using a Baxter research robot equipped with an ATI Mini-45 six-axis force sensor and a MYO armband. The test task involved pushing two objects: a fixed stapler and a loose bottle. The dataset comprised 15 sets of demonstration data, with EMG sampled at 100 Hz and force data at 200 Hz.

Results and Discussion

The multimodal approach (Condition 1) was compared against two alternatives:

  • Condition 1 (Full Multimodal): The robot successfully and smoothly pushed both objects with low force variance.
  • Condition 2 (No Force Feedforward): The robot failed to push the objects, as learned stiffness alone provided insufficient force.
  • Condition 3 (Constant Stiffness): The robot completed the task but exhibited high force variance and unstable interactions, such as jerky spikes when contacting the bottle.

The results demonstrate that both stiffness and force data are essential for contact-rich tasks. Human-like behavior provides stability; by learning when to be stiff and when to be compliant, the robot becomes robust against external perturbations.

Limitations and Future Work

While promising, the system's performance depends on demonstration consistency; high variance in human input can complicate HSMM training. Current experiments were limited to fixed end-effector orientations. Future work will explore 6-DOF orientations, cross-modal signal fusion for complex strategies, and online refinement to allow the robot to "tweak" its skills during execution.

This summary is based on the paper "Encoding Multiple Sensor Data for Robotic Learning Skills From Multimodal Demonstration."

Summary updated

BibTeX
@article{zeng2019encoding,
  author = {Chao Zeng and Chenguang Yang and Junpei Zhong and Jianwei Zhang},
  title = {{Encoding Multiple Sensor Data for Robotic Learning Skills From Multimodal Demonstration}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2945484},
}
Figure