Modular Robot Design Synthesis with Deep Reinforcement Learning

Julian Whitman, Raunaq M. Bhirangi, Matthew J. Travers, Howie ChosetAAAI 2020 · OpenReview

Abstract: Modular robots hold the promise of versatility in that their components can be re-arranged to adapt the robot design to a task at deployment time. Even for the simplest designs, determining the optimal design is exponentially complex due to the number of permutations of ways the modules can be connected. Further, when selecting the design for a given task, there is an additional computational burden in evaluating the capability of each robot, e.g., whether it can reach certain points in the workspace. This work uses deep reinforcement learning to create a search heuristic that allows us to efficiently search the space of modular serial manipulator designs. We show that our algorithm is more computationally efficient in determining robot designs for given tasks in comparison to the current state-of-the-art.

Introduction

Modular robots—composed of discrete joints, links, and brackets—offer "Lego-like" flexibility, allowing users to build a custom robot for a specific job. However, finding the optimal arrangement of these modules is a challenge of combinatorial complexity. For a robot with up to 16 modules and a dozen module types, the search space is astronomical. Beyond the number of permutations, evaluating a single candidate design is computationally expensive, requiring Inverse Kinematics (IK) solutions to verify reachability while avoiding obstacles and respecting joint torque limits.

Researchers from Carnegie Mellon University have developed a deep reinforcement learning (DRL) framework that learns a search heuristic to automatically "assemble" modular robot designs for specific tasks. By treating robot design as a sequential decision process, their algorithm finds higher-performing, lower-cost robot configurations significantly faster than traditional genetic algorithms or exhaustive search methods.

Methodology

The authors frame robot assembly as a Markov Decision Process (MDP). The state consists of the current partial arrangement of modules, the task goal (target position and orientation), and a voxelized grid representing environmental obstacles. Actions involve adding a specific module type—such as a 10cm link, a rotary actuator, or an end-effector—to the current assembly. The reward is sparse, based on whether the final robot can reach the target, minus penalties for the robot's total mass and number of joints to encourage efficiency.

The sequential assembly process of modular components into a complete robot manipulator.

The core of the approach is a Deep Q-Network (DQN) trained as a Universal Value Function Approximator (UVFA), allowing the network to generalize across different workspace targets. To handle the sparse reward problem, the team utilized Hindsight Experience Replay (HER), which re-labels failed attempts as successes for the locations the robot actually managed to reach. Once trained, the DQN's Q-values serve as a learned search heuristic for a best-first graph search, combining neural network "intuition" with the completeness of a tree search.

Experimental Setup

The study used specifications for modular components from Hebi Robotics, including an 11-module set and an expanded 16-module set with varying torque ratings. Evaluation was conducted on 50 sets of 10 random workspace targets, each featuring randomized obstacle grids.

Key Insights and Results

The DQN-based search demonstrated significant improvements over Genetic Algorithms (GA) and state-of-the-art best-first search baselines:

  • Computational Efficiency: The DQN approach found the first feasible solution in 0.26 minutes on average, compared to 3.04 minutes for the baseline best-first search. It evaluated approximately 10 complete robots to find a solution, whereas the GA required roughly 138.
  • Robustness: In tests with strict torque constraints, the GA failed to find a solution in 28 out of 50 trials. The DQN-based search failed only once.
  • Design Quality: The resulting robots generally had lower mass and fewer joints. The DQN naturally learns a "depth-first" tendency, favoring shorter, more efficient robots to maximize the return.
Comparison between a simulated synthesized robot and its real-world implementation reaching into a drawer.

A key technical innovation is the Heuristic Summation Trick. While the DQN is trained on single targets, the authors found that simply summing the Q-values for each target creates a highly effective heuristic for multi-target tasks. This allows the system to implicitly encode robot capabilities like reachability and collision avoidance without expensive optimization at every search step.

Limitations and Future Work

Currently, the system is limited to serial topologies (chain robots) and does not yet support branching structures like multi-legged walkers. Additionally, the DQN must be retrained if a brand-new module type is added to the inventory. Future work may explore "warm-starting" training for inventory changes and incorporating dynamic motion planning—moving beyond the current quasi-static assumption of reachability—to include velocity and acceleration constraints in the reward function.

Summary updated

Abstract

Modular robots hold the promise of versatility in that their components can be re-arranged to adapt the robot design to a task at deployment time. Even for the simplest designs, determining the optimal design is exponentially complex due to the number of permutations of ways the modules can be connected. Further, when selecting the design for a given task, there is an additional computational burden in evaluating the capability of each robot, e.g., whether it can reach certain points in the workspace. This work uses deep reinforcement learning to create a search heuristic that allows us to efficiently search the space of modular serial manipulator designs. We show that our algorithm is more computationally efficient in determining robot designs for given tasks in comparison to the current state-of-the-art.
BibTeX
@inproceedings{whitman2020modular,
  author = {Julian Whitman and Raunaq M. Bhirangi and Matthew J. Travers and Howie Choset},
  title = {{Modular Robot Design Synthesis with Deep Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=xoc4-bk0q-U},
}
Figure