dm_control: Software and Tasks for Continuous Control

Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy P. Lillicrap, Nicolas Heess

Abstract

The dm_control software package is a collection of Python libraries and task suites for reinforcement learning agents in an articulated-body simulation. A MuJoCo wrapper provides convenient bindings to functions and data structures. The PyMJCF and Composer libraries enable procedural model manipulation and task authoring. The Control Suite is a fixed set of tasks with standardised structure, intended to serve as performance benchmarks. The Locomotion framework provides high-level abstractions and examples of locomotion tasks. A set of configurable manipulation tasks with a robot arm and snap-together bricks is also included.

Introduction

Developing agents capable of navigating the physical world is a central challenge in Artificial Intelligence. Unlike discrete domains like board games or text, physical tasks are continuous in time, state, and action, governed by complex second-order dynamics. Historically, research in this area was hindered by a fragmented software landscape, requiring researchers to bridge low-level C++ physics engines with high-level Python scripts. This often led to "one-off" environments that were difficult to modify, share, or benchmark.

dm_control addresses these challenges by providing a unified, modular, and Pythonic infrastructure built on top of the MuJoCo physics engine. It streamlines the process of building complex environments—from classic pendulums to multi-agent soccer—while offering a standardized suite of benchmarks for evaluating continuous control algorithms.

The dm_control Architecture

The software stack is designed with modularity at its core, utilizing several key libraries:

  • PyMJCF: This library acts as a "Document Object Model" (DOM) for MuJoCo’s XML format (MJCF). It allows for the procedural assembly of models, enabling researchers to programmatically attach different limb designs to various torso types.
  • Composer: A high-level framework that treats environment components as "Entities." This modularity allows users to easily compose complex scenes by placing robot entities into various arena entities.
  • dm_env: A standardized Python API that defines the interaction protocol (actions, observations, rewards) between the agent and the simulation.
Procedural generation of 'Ant' variants using PyMJCF and Composer.

Task Suites and Benchmarks

dm_control provides several distinct collections of tasks designed for different research needs:

  1. The Control Suite: A set of approximately 20 classic domains, such as Cart-pole, Acrobot, and Humanoid. These serve as fixed benchmarks with standardized physics and rewards to ensure fair algorithmic comparison.
  2. Locomotion Tasks: Frameworks for complex movement, including obstacle-filled corridors, maze navigation, and multi-agent soccer.
  3. Manipulation Tasks: Environments featuring a 6-DOF Kinova Jaco robotic arm, including tasks like stacking bricks that use a "snap-together" model to simulate realistic alignment.
  4. Biological Models: High-fidelity models of a Quadruped, a Dog (with anatomical muscle structures), and a Rodent, facilitating research at the intersection of AI and neuroscience.
The dm_control Control Suite: A diverse set of standardized continuous control benchmarks.

Design Philosophy and Insights

A key innovation in dm_control is the tolerance() function for reward design. It maps physical quantities to a [0, 1] range, providing "soft" rewards that are more informative than sparse success signals while maintaining a consistent scale across different tasks.

The framework also emphasizes the separation of the "Walker" (the agent's body) from the "Arena" (the environment). This allows researchers to test the same agent in vastly different scenarios with minimal code changes. Furthermore, the software includes built-in support for simulating real-world sensor issues, such as signal delays and observation noise. The authors also highlight the importance of simulation stability; since RL agents are adept at exploiting numerical instabilities (Sims' Law), dm_control undergoes extensive verification to ensure tasks remain non-exploitable.

Discussion and Future Work

dm_control has become a foundational tool in RL research, supporting major publications on algorithms like MPO. While it provides a robust suite of benchmarks ranging from simple pendulums to complex humanoid locomotion, challenges remain. Future work includes deepening the anatomical complexity of biological models and further bridging the "Sim-to-Real" gap in high-friction physical interactions.

Summary updated

Abstract

The dm_control software package is a collection of Python libraries and task suites for reinforcement learning agents in an articulated-body simulation. A MuJoCo wrapper provides convenient bindings to functions and data structures. The PyMJCF and Composer libraries enable procedural model manipulation and task authoring. The Control Suite is a fixed set of tasks with standardised structure, intended to serve as performance benchmarks. The Locomotion framework provides high-level abstractions and examples of locomotion tasks. A set of configurable manipulation tasks with a robot arm and snap-together bricks is also included. dm_control is publicly available at https://www.github.com/deepmind/dm_control
BibTeX
@inproceedings{tassa2020dm,
  author = {Yuval Tassa and Saran Tunyasuvunakool and Alistair Muldal and Yotam Doron and Siqi Liu and Steven Bohez and Josh Merel and Tom Erez and Timothy P. Lillicrap and Nicolas Heess},
  title = {{dm\_control: Software and Tasks for Continuous Control}},
  booktitle = {Softw. Impacts},
  year = {2020},
  url = {https://openreview.net/forum?id=o24PnWt_tu},
}
Figure