Abstract
The advantages of modular robot systems stem from their ability to change between different configurations, enabling them to adapt to complex and dynamic real-world environments. Then, how to perform the accurate and efficient change of the modular robot system, i.e., the self-reconfiguration problem, is essential. Existing reconfiguration algorithms are based on discrete motion primitives and are suitable for lattice-type modular robots. The modules of freeform modular robots are connected without alignment, and the motion space is continuous. It renders existing reconfiguration methods infeasible. In this paper, we design a parallel distributed self-reconfiguration algorithm for freeform modular robots based on multi-agent reinforcement learning to realize the automatic design of conflict-free reconfiguration controllers in continuous action spaces. To avoid conflicts, we incorporate a collaborative mechanism into reinforcement learning. Furthermore, we design the distributed termination criteria to achieve timely termination in the presence of limited communication and local observability. When compared to the baselines, simulations show that the proposed method improves efficiency and congruence, and module movement demonstrates altruism.
Introduction: The Challenge of Freeform Reconfiguration
Modular robots are systems composed of many smaller robotic units that can rearrange themselves to form different structures, adapting to various tasks or environments. Historically, these have been "lattice-type" robots, which require precise dock-to-dock alignment at discrete connection points. Recently, "freeform" modular robots have emerged, often utilizing magnetic spherical shells that allow continuous, unaligned connections.
While freeform designs are more flexible and robust to hardware variations, they make the self-reconfiguration problem exponentially harder. Because modules can move and connect anywhere in a continuous space, traditional search-based algorithms and discrete control rules suffer from combinatorial explosion. When multiple modules move simultaneously in a decentralized way, they frequently encounter kinematic conflicts—colliding, blocking each other's paths (closure), or breaking the structure apart (disintegration). This paper addresses the gap in achieving conflict-free, distributed self-reconfiguration in continuous spaces for these complex systems.
Methods: Altruism Proximal Policy Optimization (APPO)
The authors frame the reconfiguration problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) and solve it using Multi-Agent Reinforcement Learning (MARL). The core innovation is Altruism Proximal Policy Optimization (APPO). Standard independent PPO makes agents purely selfish, maximizing only their own progress, which often leads to traffic jams and collisions. Conversely, global reward maximization is computationally prohibitive for large swarms.
APPO introduces a sociological concept: an altruism factor ($\alpha_i$) for each module.
- Hybrid Reward: A module's reward is a blend of its own success (moving toward its target) and the "mean-field" reward of its immediate neighbors, weighted by $\alpha_i$.
- Meta-Learning Personalities: Instead of manual tuning, the system uses meta-reinforcement learning to optimize a personalized altruism factor for each module. Some modules learn to be "selfish" (pushing forward quickly), while others learn to be "cooperative" (yielding or waiting).
- Distributed Termination: An asynchronous, local-communication-only termination protocol allows the swarm to know when the shape is complete without global oversight.
Key Results and Insights
The authors evaluated APPO in a custom 2D simulation environment built with Unity ML-Agents, testing tasks like Rectangle-to-Triangle and Rectangle-to-$\Phi$. APPO was compared against Independent PPO (IPPO), Mean Field PPO (MFPPO), and Coordinated Policy Optimization (CoPO).
- Higher Success Rates: APPO significantly outperformed baselines in Mean Congruence. For the complex Rectangle-to-Triangle task, APPO achieved 92.1% congruence, compared to CoPO's 85.3%.
- Fewer Conflicts: APPO drastically reduced collisions and blockages. In the Rectangle-to-$\Phi$ task, APPO had a conflict rate of 19.4%, compared to CoPO's 27.3% and IPPO's 67.1%.
- Emergent Social Behavior: By framing reconfiguration as a non-zero-sum game, modules spontaneously learned complex coordination strategies like queuing and yielding entirely without hard-coded rules.
Conclusion
APPO demonstrates that incorporating altruism into reinforcement learning allows freeform modular robots to navigate continuous motion spaces efficiently and autonomously. While currently restricted to 2D simulations with simplified physics, the results prove that a heterogeneous mix of "personalities" can resolve deadlocks naturally. Future work will extend the algorithm to 3D spaces and deploy the trained policies onto physical hardware currently under development.
Summary updated