Multisensory Learning Framework for Robot Drumming

Andrey Barsky, Claudio Zito, Hiroki Mori, Tetsuya Ogata, Jeremy L. WyattarXiv 2019 · arXiv

Introduction

Sensorimotor learning—the mapping of sensory inputs to motor actions—is a cornerstone of robotics. However, training deep learning models for complex tasks like drumming requires massive amounts of synchronized data. Collecting thousands of hours of audio, video, and joint-position data on physical hardware is prohibitively slow and risks damaging expensive equipment. Furthermore, real-world sensors are often unreliable; a robot must remain functional even if a camera is obstructed or the environment is noisy.

This research introduces an open-source framework that leverages high-fidelity simulation to generate "pilot data" and a neural architecture capable of cross-modal retrieval, allowing a robot to "imagine" motion from sound alone.

System Overview and Data Generation

The framework is built using the Gazebo simulator and the Robot Operating System (ROS), featuring a simulated Baxter research robot. To overcome the data scarcity problem, the researchers developed a pipeline to generate perfectly synchronized synthetic datasets.

Overview of the drumming simulation environment and the data generation pipeline.
  1. Motion Primitives: The system defines simple three-waypoint movements (start, strike, return).
  2. Motion Planner: These primitives are synchronized to "drum tabs" (rhythm scripts).
  3. Audio Synthesis: When the virtual drumstick collides with a target, the system synthesizes a corresponding 44100 Hz audio signal.
  4. Multimodal Output: The resulting dataset includes 120x120 RGB video, high-fidelity audio, and proprioceptive data (joint angles, velocities, and accelerations).

The Multimodal Integration Network

The core technical contribution is a neural architecture designed to learn a shared representation across different senses.

The Multimodal Integration Network architecture featuring encoders, a shared latent space, and recurrent connections.

Architecture Details

  • Encoders: Separate branches process RGB images via Convolutional Neural Networks (CNNs), audio via spectrograms, and joint angles.
  • The Bottleneck: These features are concatenated into a single multimodal feature vector, creating a shared latent space.
  • Temporal Learning: This vector passes through a Contractive Autoencoder with Long Short-Term Memory (LSTM) connections, enabling the network to understand the temporal evolution of a drum beat.
  • Modality Dropout: During training, specific inputs (like video) are randomly disabled. This forces the network to learn modality-agnostic concepts, ensuring it can reconstruct missing information—such as predicting arm motion from only an audio track.

Key Insights and Results

The framework demonstrates that the latent space becomes a modality-invariant representation of a "drum strike." Key findings include:

  • Cross-Modal Generation: The system successfully generated appropriate drumming motions from previously unseen audio sequences.
  • Denoising Capabilities: The autoencoder effectively filtered white noise from audio signals while retaining the sharp signal of the drum beat.
  • Causal Anticipation: The researchers identified a "causal" challenge: in drumming, motion must begin before the sound is produced. While the network handled this, motion reconstructions derived purely from audio were slightly noisier due to these high-order dependencies.
  • Visual Reconstruction: Even with video input dropped, the network could reconstruct a visual sequence of the arm moving that qualitatively matched the motion required to produce the input sound.

Future Directions

While the simulation provides a powerful sandbox, the authors identify several areas for improvement. Future work will explore Bidirectional RNNs to better capture the anticipatory nature of drumming movements and the integration of tactile feedback to sense physical impact. Ultimately, the goal is "Sim-to-Real" transfer, applying these learned models to physical robotic hardware in complex, real-world environments.

Repository: The framework is open-source and available on GitHub at: https://github.com/andreybarsky/sensorimotor_drumming

Summary updated

BibTeX
@misc{barsky2019multisensory,
  author = {Andrey Barsky and Claudio Zito and Hiroki Mori and Tetsuya Ogata and Jeremy L. Wyatt},
  title = {{Multisensory Learning Framework for Robot Drumming}},
  year = {2019},
  eprint = {1907.09775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.09775},
}
Figure