Learning-In-The-Loop Optimization: End-To-End Control And Co-Design Of Soft Robots Through Learned Deep Latent Representations

Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojciech Matusik, Daniela Rus

Introduction

Soft robots, constructed from compliant materials, possess an infinite number of degrees of freedom. While this flexibility allows for remarkable adaptability, it presents a massive challenge for traditional control theory. Standard rigid-body physics do not apply, and high-fidelity simulations like the Finite Element Method (FEM) generate millions of degrees of freedom—far too many for real-time control.

Existing solutions often rely on hand-tuned features or simplified analytical models that fail to capture complex contacts and non-linear dynamics. To bridge this gap, researchers have introduced a "learning-in-the-loop" framework that simultaneously optimizes a soft robot's physical design and its control strategy by compressing its state into a manageable latent representation.

Methodology

The proposed solution is a three-part pipeline integrated into a differentiable simulation environment.

The learning-in-the-loop framework, illustrating the cycle from unreduced state to observer, controller, actuation, and simulation.
  1. Differentiable Simulation: The framework utilizes ChainQueen, a simulator based on the Moving Least Squares Material Point Method (MLS-MPM). Because it is fully differentiable, the system can use backpropagation to evaluate how minute changes in control signals or material properties (like stiffness) affect the final outcome.
  2. The Observer (VAE): Instead of manually engineering features, the system employs a Convolutional Variational Autoencoder (VAE).
Architecture of the Convolutional Variational Autoencoder used to compress the robot's velocity field into a latent representation.

This VAE takes the high-dimensional velocity field of the robot (represented on a grid) and compresses it into a low-dimensional latent vector. This compact vector serves as the "observation" state for the controller. The variational aspect provides necessary regularization to prevent the state from collapsing into a useless 1D manifold.

  1. Learning-In-The-Loop Co-Optimization: The optimization process relies on alternating minimization. During the Optimization Phase, the VAE is frozen, and gradients are used to update the controller and the robot's material parameters to minimize a task-specific loss (e.g., maximizing walking distance). In the Learning Phase, the VAE is retrained using the latest simulation data to ensure the latent representation accurately reflects the robot's evolving dynamics.

Results and Insights

The framework was evaluated on procedurally generated 2D and 3D model robots, including bipeds, arms, and complex multi-limbed structures, yielding several key insights:

  • Dynamics-Aware Features: Unlike baseline methods like $k$-means clustering that only analyze static geometry, the VAE learns from velocity fields. This enables the controller to "sense" momentum and deformation trends.
  • Emergent Physical Modes: Visualizing the latent space reveals that the VAE naturally discovers distinct "physical modes." For instance, one latent variable might strongly correlate with forward velocity, while another captures high-frequency wobbling.
  • Sample Efficiency and Robustness: The gradient-based approach optimized robots in the equivalent of roughly 400 simulations. In contrast, traditional evolutionary methods often require over 30,000 simulations, marking a nearly 75x improvement in sample efficiency. Furthermore, the VAE-based method converged consistently, whereas clustering baselines frequently got stuck in poor local minima.
  • Successful Co-Design: The system successfully optimized material stiffness alongside control policies, demonstrating that "softer" isn't always better. The algorithm identified specific stiffness gradients that maximized locomotion speed.

Limitations and Future Work

Despite its success, the approach has limitations. While 2D simulations optimize in seconds, 3D optimizations remain computationally heavy, sometimes taking over a day for a single walker.

Additionally, there is a "sim-to-real" gap; the model relies on grid-based velocity data, which the authors suggest could be approximated in the real world using optical flow from cameras. Finally, retraining the VAE can occasionally cause temporary "backward progress" in the controller as it adjusts to a shifted latent space, suggesting a need for refined target-network update schemes borrowed from Reinforcement Learning to mitigate interference.

Summary updated

Abstract

Soft robots have continuum solid bodies that can deform in an infinite number of ways. Controlling soft robots is very challenging as there are no closed form solutions. We present a learning-in-the-loop co-optimization algorithm in which a latent state representation is learned as the robot figures out how to solve the task. Our solution marries hybrid particle-grid-based simulation with deep, variational convolutional autoencoder architectures that can capture salient features of robot dynamics with high efficacy. We demonstrate our dynamics-aware feature learning algorithm on both 2D and 3D soft robots, and show that it is more robust and faster converging than the dynamics-oblivious baseline. We validate the behavior of our algorithm with visualizations of the learned representation.
BibTeX
@inproceedings{spielberg2019learning,
  author = {Andrew Spielberg and Allan Zhao and Yuanming Hu and Tao Du and Wojciech Matusik and Daniela Rus},
  title = {{Learning-In-The-Loop Optimization: End-To-End Control And Co-Design Of Soft Robots Through Learned Deep Latent Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure