Introduction
Industrial robots excel at kinematic precision in free space but often struggle with contact-rich tasks—such as pushing objects, scrubbing surfaces, or pressing buttons—where managing interaction forces is critical. Traditional programming for these tasks is labor-intensive and rigid. While Teaching-by-Demonstration (TbD) allows non-experts to guide robots, most systems capture only the trajectory, ignoring the dynamic "human touch"—the adaptive stiffness and force profiles required for stable interaction.
This research introduces a multimodal TbD framework that enables robots to learn tasks requiring both precise motion and force control. By encoding human muscle activity (EMG) and interaction forces into a Hidden semi-Markov Model (HSMM), the researchers enabled a Baxter robot to mimic the nuanced, adaptive stiffness of a human tutor.
Methodology
The proposed architecture is divided into two primary phases: Learning and Reproduction.
Learning Phase
During human demonstration, the system records three distinct data streams:
- Motion: The trajectory and velocity of the robot's endpoint.
- Force: Data from a wrist-mounted force/torque sensor.
- Stiffness: Estimated in real-time from the tutor's muscle activity using a MYO electromyography (EMG) armband.
Reproduction Phase
To perform the task, the robot uses Gaussian Mixture Regression (GMR) to retrieve expected control variables based on its current position. These variables are fed into a joint-space variable impedance controller, allowing the robot to dynamically adjust its "squishiness" and pushing force.
Multimodal Encoding
A key innovation of this work is the use of parallel encoding. The researchers utilize a Hidden semi-Markov Model (HSMM) to capture the temporal duration of task stages more effectively than standard HMMs. Three parallel models are trained to correlate the robot's position ($x$) with its velocity ($\dot{x}$), stiffness ($k$), and force ($F$).
This approach allows for the individual optimization of each parameter, mimicking human motor learning where grip strength can be refined independently of arm speed. Using EMG as a proxy for stiffness eliminates the need for complex mathematical optimization or trial-and-error.
Experimental Setup
The framework was validated using a Baxter research robot equipped with an ATI Mini-45 six-axis force sensor and a MYO armband. The test task involved pushing two objects: a fixed stapler and a loose bottle. The dataset comprised 15 sets of demonstration data, with EMG sampled at 100 Hz and force data at 200 Hz.
Results and Discussion
The multimodal approach (Condition 1) was compared against two alternatives:
- Condition 1 (Full Multimodal): The robot successfully and smoothly pushed both objects with low force variance.
- Condition 2 (No Force Feedforward): The robot failed to push the objects, as learned stiffness alone provided insufficient force.
- Condition 3 (Constant Stiffness): The robot completed the task but exhibited high force variance and unstable interactions, such as jerky spikes when contacting the bottle.
The results demonstrate that both stiffness and force data are essential for contact-rich tasks. Human-like behavior provides stability; by learning when to be stiff and when to be compliant, the robot becomes robust against external perturbations.
Limitations and Future Work
While promising, the system's performance depends on demonstration consistency; high variance in human input can complicate HSMM training. Current experiments were limited to fixed end-effector orientations. Future work will explore 6-DOF orientations, cross-modal signal fusion for complex strategies, and online refinement to allow the robot to "tweak" its skills during execution.
This summary is based on the paper "Encoding Multiple Sensor Data for Robotic Learning Skills From Multimodal Demonstration."
Summary updated