In traditional robotics and automated control, systems are almost always managed through a strict hierarchy. A high-level planner or machine learning policy determines a target state—such as a desired joint angle or spatial coordinate—and passes this setpoint to a low-level classical controller, such as a Proportional-Derivative (PD) loop or a Model Predictive Control (MPC) framework. The classical controller then calculates the raw forces, torques, or voltages required to achieve that state.
While this separation makes systems mathematically tractable and safe, it introduces a severe performance bottleneck. It assumes that the low-level controller can perfectly track the high-level setpoints, an assumption that falls apart in highly dynamic, non-linear, or unpredictable environments.
Direct actuator control cuts out this middleman. By training deep reinforcement learning (DRL) policies to map high-dimensional sensory inputs directly to raw actuator commands—such as motor torques, fluidic pressures, or electrical stimulation intensities—systems can exploit their natural physical dynamics rather than fighting against a rigid tracking loop.
Bypassing the Analytical Model Bottleneck
Traditional control architectures rely heavily on explicit kinematic, hydrodynamic, or thermodynamic models to translate a desired state into an actuator command. However, deriving these equations becomes exceptionally difficult when dealing with complex, high-degree-of-freedom systems.
Direct actuator control circumvents the need for explicit physics equations by allowing an agent to implicitly learn the relationship between force and motion through environmental interaction. For instance, in wearable robotics, Zheng et al. demonstrated in End-to-End High-Level Control of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning that an end-to-end DRL approach can successfully coordinate an exoskeleton without explicit kinematic models of the human-robot interface.
Similarly, in biological and biomechanical systems where manual control law design is notoriously difficult, model-free learning provides a pathway to discover complex coordination strategies. As noted by Uchida et al. in Special issue: Multibody Biomechanics, musculoskeletal reinforcement learning allows deep learning agents to explore and identify control policies for high-dimensional biological systems that traditional kinematic descriptions cannot capture. By framing control as a model-free optimization problem, high-dimensional systems can be mastered rapidly; Peters et al. showed in Reinforcement learning by reward-weighted regression for operational space control that transforming reinforcement learning into a series of weighted regressions allows real-time learning on physical hardware without requiring exhaustive simulation periods.
Exploiting Natural Dynamics and Compliance
When a classical controller forces a robot to track a geometric trajectory, it often fights the environment. If a legged robot hits an unexpected bump, a position-based controller will aggressively spike its torque to force the leg into the pre-planned position, leading to jerky movements or falls.
Direct torque control allows the policy to become a closed-loop feedback controller that reacts to physical interactions natively. In Learning Torque Control for Quadrupedal Locomotion, Zhang et al. demonstrated that training an RL policy to output raw motor torque commands rather than target positions results in a more robust, agile, and natural walker. Because the policy directly dictates torque, it can choose to be compliant when interacting with obstacles and stiff when generating thrust. This capability is echoed by Hwangbo et al. in Learning agile and dynamic motor skills for legged robots, where an end-to-end learned actuator model enabled a quadruped to execute highly dynamic behaviors, such as high-speed locomotion and complex fall recoveries, that surpassed the limits of traditional control. As reviewed by Yu et al. in Learning-based legged locomotion: State of the art and future perspectives, this paradigm shift allows neural networks to handle high-dimensional sensory inputs and map them directly to fluid, reactive physical behaviors.
Hierarchical Control (Traditional):
[Sensory Input] -> [RL Policy] -> [Target Position] -> [PD Controller] -> [Raw Torque]
^
(Fights unexpected physics)
Direct Actuator Control (End-to-End):
[Sensory Input] -> [RL Policy] -----------------------------------------> [Raw Torque]
^
(Natively exploits compliance)
Mastering Non-Linear and Underactuated Systems
Direct actuator control is particularly powerful in domains where the physical dynamics are highly non-linear or underactuated, making classical linear controllers ineffective.
Soft Robotics
Soft robotic hands lack rigid joints, meaning their shape changes continuously based on contact forces. Gupta et al., in Learning dexterous manipulation for a soft robotic hand from human demonstrations., utilized a trajectory-centric RL algorithm to optimize controllers for systems with unknown soft dynamics. The resulting policies functioned as closed-loop feedback controllers, automatically selecting different contact points or appendages based on the system's state rather than simply replaying a fixed trajectory.
Fluid Dynamics
Operating in fluid environments introduces complex hydrodynamic forces that are difficult to model in real time. In Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning, a separate team led by Zhang et al. showed that end-to-end reinforcement learning could achieve high-precision formation control for underactuated robotic fish. The learned policies managed complex environmental disturbances so effectively that tracking errors were significantly smaller than the physical dimensions of the robots themselves.
Magnetic and Plasma Fields
Magnetic actuation is inherently non-linear, as field strength drops off dramatically with distance. The authors of Deep Reinforcement Learning-Based Control for Stomach Coverage Scanning of Wireless Capsule Endoscopy leveraged DRL to manage the unpredictable dynamics of magnetic steering for a wireless capsule endoscope, achieving comprehensive scanning coverage by learning to account for physical interactions that rigid physics equations fail to capture. On a larger scale, Spangher et al. highlighted in Position: Opportunities Exist for Machine Learning in Magnetic Fusion Energy that deep reinforcement learning can manage complex magnetic shape control in dynamic plasma systems, successfully navigating non-linear dynamics and high-dimensional state spaces in real-time where traditional linear controllers fail.
Eliminating Hierarchical Inefficiencies
One might wonder why we cannot simply use an RL agent to tune the parameters (such as the gains) of an underlying classical controller. Sinha investigated this in Co-designing Intelligent Control of Building HVACs and Microgrids, demonstrating that forcing a learning agent to operate through a rigid optimization framework introduces performance bottlenecks and inefficiencies. Direct control of hardware actuators by the RL agent consistently proved more efficient.
Furthermore, direct control eliminates the need for tedious, expert-driven manual calibration. For example, Wannawas et al. showed in Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling that DRL agents can simultaneously discover optimal stimulation timing and intensity for paralyzed muscles without any pre-defined activation patterns, eliminating the need for manual tuning of stimulation sequences for individual users. By mapping inputs directly to raw outputs, model-free deep reinforcement learning can match or exceed the performance of industry-standard controllers that require extensive manual calibration by human experts, significantly reducing the engineering effort required to deploy complex control systems.
Key Foundational Work
Learning dexterous manipulation for a soft robotic hand from human demonstrations. (2016) — contributed 1 concept elements to this direction.
Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration (2016) — contributed 1 concept elements to this direction.
Reinforcement learning by reward-weighted regression for operational space control (2007) — contributed 1 concept elements to this direction.