Leveraging Multiple Environments for Learning and Decision Making: a Dismantling Use Case

Alejandro Suárez-Hernández, Thierry Gaugry, Javier Segovia Aguas, Antonin Bernardin, Carme Torras, Maud Marchal, Guillem Alenyà

In the field of robotics, learning the outcomes of symbolic actions through real-world experimentation is often prohibitively risky or expensive. A robot attempting to learn how to dismantle a device might inadvertently damage itself or the target object during the exploration phase. This paper, presented at IROS 2020, addresses this challenge by introducing a framework that leverages multiple simulation environments of varying fidelity to accelerate learning while minimizing real-world failures.

The MENID Framework

The core contribution of this work is the MENID (Multiple Environment Noise Indeterministic Deictic) rule. This framework extends standard probabilistic planning rules to track action outcomes across multiple environments simultaneously. By doing so, a robot can utilize "cheap" but potentially inaccurate low-fidelity simulators to build an initial model before graduating to a "dear" high-fidelity target environment.

System architecture of the MENID framework, illustrating how the MDP solver selects between fast simulation and the target environment based on action history and current state.

As shown in the architecture above, the system incorporates an Environment Selector that decides whether to perform an action in a fast simulation or the target environment. This decision is governed by a confidence-based algorithm. The robot uses Dirichlet sampling to estimate the error of its current success probability distribution; if the variance is high, it remains in simulation. Once a confidence threshold ($\delta$) is met, the robot transitions to the target environment.

Bridging the Fidelity Gap

A critical challenge in multi-environment learning is ensuring that simulation errors do not lead to real-world failures. The authors implement a Decreasing-m Estimator, which acts as a mathematical bridge. Initially, the robot's internal model relies heavily on simulation data. However, as real-world data is collected, the influence of the simulator is automatically phased out, ensuring the final planning model is grounded in physical reality.

Experimental Validation: Hard Drive Dismantling

The approach was evaluated using a complex dismantling task involving a hard drive. The robot had to learn the success probabilities for three distinct actions: levering a PCB, shaking the drive bay, and using a suction cup. The study utilized the SOFA Framework for physics simulation, allowing for the modeling of both rigid and deformable objects.

Visualizations of the dismantling tasks (a, b, c) and graphs showing the relationship between model quality, computation time, and symbolic/Euclidean error.

One of the most significant findings is the concept of the "Good Enough" Simulator. The researchers generated 36 different mesh qualities for the hard drive components to study the impact of resolution. They discovered that simulators do not need perfect geometric accuracy to be symbolically useful. As seen in the results above, there is a "sweet spot" where increasing mesh resolution yields diminishing returns for symbolic success, allowing robots to learn effectively even with simplified models.

Key Results and Insights

  • Faster Convergence: The interleaved learning approach reached high reward levels significantly faster than baselines that relied solely on the target environment.
  • Failure Avoidance: By the time the robot moved to the physical system (a WAM robotic arm), it had already developed a reliable "hunch" about which actions were likely to succeed, effectively avoiding the accumulation of failures typical in early-stage learning.
  • Risk-Awareness: The benefits of this multi-tier approach scale with the penalty of failure. In high-stakes environments where breaking a part is costly, the value of safe, virtual pre-testing becomes paramount.

Conclusion and Future Work

The MENID framework demonstrates that low-fidelity simulations, despite their noise, are powerful tools for bootstrapping robotic intelligence. Future work aims to move beyond learning probabilities for pre-defined rules toward learning the structure of the rules themselves, potentially allowing robots to discover entirely new manipulation strategies autonomously.

Summary updated

Abstract

Learning is usually performed by observing real robot executions. Physics-based simulators are a good alternative for providing highly valuable information while avoiding costly and potentially destructive robot executions. We present a novel approach for learning the probabilities of symbolic robot action outcomes. This is done leveraging different environments, such as physics-based simulators, in execution time. To this end, we propose MENID (Multiple Environment Noise Indeterministic Deictic) rules, a novel representation able to cope with the inherent uncertainties present in robotic tasks. MENID rules explicitly represent each possible outcomes of an action, keep memory of the source of the experience, and maintain the probability of success of each outcome. We also introduce an algorithm to distribute actions among environments, based on previous experiences and expected gain. Before using physics-based simulations, we propose a methodology for evaluating different simulation settings and determining the least time-consuming model that could be used while still producing coherent results. We demonstrate the validity of the approach in a dismantling use case, using a simulation with reduced quality as simulated system, and a simulation with full resolution where we add noise to the trajectories and some physical parameters as a representation of the real system.
BibTeX
@inproceedings{suarezhernandez2020leveraging,
  author = {Alejandro Suárez-Hernández and Thierry Gaugry and Javier Segovia Aguas and Antonin Bernardin and Carme Torras and Maud Marchal and Guillem Alenyà},
  title = {{Leveraging Multiple Environments for Learning and Decision Making: a Dismantling Use Case}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341182},
}
Figure