dm_control: Software and tasks for continuous control

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy P. Lillicrap, Nicolas Heess, Yuval Tassa

TL;DR

DeepMind’s dm_control is a comprehensive Python software stack designed to streamline Reinforcement Learning (RL) research in physics-based simulations. It provides a modular framework for building complex environments—ranging from classic pendulums to multi-agent soccer and robotic manipulation—while offering a standardized suite of benchmarks for evaluating continuous control algorithms.

What Problem Does This Paper Address?

Developing agents that can navigate the physical world is a core challenge in AI. Unlike board games or text, physical tasks are fundamentally continuous in time, state, and action, governed by complex second-order dynamics (positions and velocities).

Historically, researchers faced a fragmented landscape: they had to bridge low-level C++ physics engines (like MuJoCo) with high-level Python RL scripts, often resulting in "one-off" environments that were difficult to modify, share, or benchmark. dm_control addresses this by providing a unified, modular, and Pythonic infrastructure that handles everything from procedural model generation to standardized reward structures.

Methods

The paper introduces a layered software architecture built on top of the MuJoCo physics engine:

  • PyMJCF: A library that acts like a "Document Object Model" (DOM) for MuJoCo’s XML format (MJCF). It allows researchers to procedurally assemble models—for example, programmatically attaching a specific leg design to various torso types.
  • Composer: A high-level framework that treats environment components as "Entities." This modularity allows users to easily compose complex scenes (e.g., placing a "Robot Entity" into a "Maze Arena Entity").
  • dm_env: A standardized Python API for RL environments that clearly defines how actions, observations, and rewards are passed between the agent and the simulation.
  • The tolerance() Function: A novel utility for designing reward functions. It maps physical quantities to a [0, 1] range, allowing for "soft" rewards that are easier for agents to learn from than sparse "success/failure" signals, while maintaining a consistent scale across different tasks.

Datasets and Benchmarks

The paper presents three major collections of tasks:

  1. The Control Suite: A set of ~20 classic and standardized domains (e.g., Cart-pole, Acrobot, Cheetah, Humanoid). These are intended as "fixed" benchmarks where the physics and rewards do not change, ensuring fair comparisons between algorithms.
  2. Locomotion Tasks: High-level frameworks for complex movement, including a "run through corridor" task with obstacles, maze navigation, and a multi-agent soccer environment.
  3. Manipulation Tasks: Environments involving a 6-DOF Kinova Jaco robotic arm. These include tasks like "stacking 3 bricks," where the bricks use a custom "snap-together" model to simulate friction and alignment.
  4. Biological Models: The paper also introduces high-fidelity models of a Quadruped, a Dog (with anatomical muscle structures), and a Rodent, designed for research at the intersection of AI and neuroscience.

Key Insights and Ideas

  • Modularity is King: By separating the "Walker" (the agent's body) from the "Arena" (the environment), researchers can test the same agent in vastly different scenarios (flat ground vs. rocky terrain) with minimal code changes.
  • Standardized Rewards: By forcing rewards into the [0, 1] interval, the suite makes learning curves across different tasks (like "swinging a pendulum" vs. "walking") directly comparable on the same Y-axis.
  • Sims’ Law: The authors acknowledge that RL agents are "bug hunters." If a physics simulation has a tiny numerical leak, the agent will find it and exploit it to gain infinite energy. dm_control includes extensive verification to ensure tasks are stable and non-exploitable.
  • Observation Pipelines: The software includes built-in support for simulating real-world sensor issues, such as delayed signals, corrupted data (noise), and "stacked" observations (giving the agent a short memory of recent frames).

Key Results

While this is a "software and tools" paper rather than an algorithmic breakthrough, its impact is measured by its adoption. dm_control has served as the backbone for numerous high-profile DeepMind publications, including work on MPO (Maximum a Posteriori Policy Optimisation) and Emergence of Locomotion Behaviors.

The paper demonstrates that the provided benchmarks span a wide range of difficulty: from the "Pendulum" (solvable in minutes) to the "Humanoid" and "Stack 3 Bricks" tasks, which remain significant challenges for modern RL agents.

Limitations and Future Work

  • Anatomical Complexity: While the "Dog" model includes muscle and tendon attachment points, the authors note that fully integrating these into the dynamical model requires deeper anatomical knowledge and remains a work in progress.
  • Community Integration: The authors express a desire to integrate more community-contributed models and tasks into the ecosystem.
  • Real-to-Sim Gap: While the manipulation tasks use "snap-together" bricks to simplify the physics of assembly, bridging the gap to truly realistic, high-friction physical interaction remains an ongoing challenge in the field.

Summary updated

Abstract

The dm_control software package is a collection of Python libraries and task suites for reinforcement learning agents in an articulated-body simulation. A MuJoCo wrapper provides convenient bindings to functions and data structures. The PyMJCF and Composer libraries enable procedural model manipulation and task authoring. The Control Suite is a fixed set of tasks with standardised structure, intended to serve as performance benchmarks. The Locomotion framework provides high-level abstractions and examples of locomotion tasks. A set of configurable manipulation tasks with a robot arm and snap-together bricks is also included. dm_control is publicly available at https://www.github.com/deepmind/dm_control
BibTeX
@inproceedings{tunyasuvunakool2020dm,
  author = {Saran Tunyasuvunakool and Alistair Muldal and Yotam Doron and Siqi Liu and Steven Bohez and Josh Merel and Tom Erez and Timothy P. Lillicrap and Nicolas Heess and Yuval Tassa},
  title = {{dm\_control: Software and tasks for continuous control}},
  booktitle = {Softw. Impacts},
  year = {2020},
  url = {https://openreview.net/forum?id=o24PnWt_tu},
}

Loading related content...

Figure