From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent

Ganghun Lee, Minji Kim, Min Su Lee, Byoung-Tak Zhang

TL;DR

This paper introduces a novel framework for training robotic agents to sketch from scratch, using deep decoupled hierarchical reinforcement learning. The system learns both stroke-based rendering (what to draw) and motor control (how to draw) independently, then combines them for real-world execution, achieving robust and flexible sketching without human-crafted features or demonstrations.

What Problem Does This Paper Address?

Robotic drawing is a challenging task that requires two distinct skills: deciding what to draw (stroke-based rendering, SBR) and executing how to draw it (motor control). While previous SBR methods often relied on virtual rendering or complex optimization, and robotic drawing systems typically used inverse kinematics or manual controls, this paper aims to bridge this gap. The authors address the problem of creating a robotic agent that can learn to sketch any image from scratch, mimicking human drawing processes, without needing pre-existing stroke sequences, trajectories, or human demonstrations. This would enable robots to assist in creative tasks, education, and human-robot interaction.

Methods

The core of the proposed solution is a Deep Decoupled Hierarchical Reinforcement Learning (DHRL) framework. This approach breaks down the complex sketching task into two sub-tasks, each handled by a separate policy:

  • Commander (High-Level Policy): This policy learns stroke-based rendering (SBR). It operates in an imaginary canvas environment and decides on sequential strokes to reconstruct a target image. The Commander's action is a simple straight line defined by a pen-down indicator and relative x/y displacements. It's trained using reinforcement learning, with rewards based on the similarity between its current drawing and the target image, measured by a discriminator.
  • Stroker (Low-Level Policy): This policy learns the motor control for the physical robot arm. It takes the Commander's action (as a goal) and translates it into actual joint angle commands for the robot to execute the stroke on a real canvas. Stroker is trained in a simulated environment using reinforcement learning, with rewards based on how closely the robot's pen tip reaches the goal position and orientation.

The "decoupled" aspect means these two policies are trained independently, which the authors argue is crucial for modularity and manageability, avoiding the complexities of end-to-end training for such disparate tasks. During deployment, the Commander's output becomes the Stroker's goal, and a synchronization mechanism (tracking the pen tip position) is used to keep the imaginary canvas aligned with the real-world canvas, compensating for inevitable errors.

Datasets

  • Quick, Draw! dataset: Used for training the Commander policy. This dataset consists of human-drawn simple doodles, which serve as target images.
  • MuJoCo simulation environment: Used for training the Stroker policy.

The paper does not introduce a new benchmark dataset but rather leverages existing ones for training.

Key Insights and Ideas

  • Decoupling for Complexity: The decision to decouple the high-level SBR policy (Commander) from the low-level motor control policy (Stroker) is a key insight. This separation simplifies training and allows for easier integration of different robot platforms or SBR methods in the future.
  • Imaginary Canvas Synchronization: The use of an imaginary canvas for the Commander and a synchronization mechanism to update it based on the real robot's actions is a clever way to enable camera-free operation and maintain coherence between the planning and execution stages.
  • Simple Stroke Primitives: Limiting the Commander's stroke to a simple straight line, rather than complex Bézier curves, makes the task more manageable for the low-level controller and allows the agent to learn creative drawing by composing these basic strokes.
  • RL for End-to-End Robotic Drawing: Applying RL to both the abstract SBR task and the concrete motor control task from scratch, without requiring demonstrations or hand-crafted features, is a significant step towards more autonomous and adaptable robotic agents.

Key Results

  • Successful Collaboration: The Commander and Stroker policies successfully learned to collaborate, enabling the robot to sketch target images from the Quick, Draw! dataset.
  • Robustness and Flexibility: The agent demonstrated robustness across various drawing tools (board marker, pen, crayon) and surfaces (whiteboard, paper), achieving similar sketch quality. This is attributed to the camera-free methodology.
  • Synchronization Effectiveness: The synchronization technique was shown to be crucial for compensating for Stroker's errors, leading to significantly better final sketches compared to results without synchronization.
  • Performance Metrics: The paper presents performance evaluation plots showing decreases in L2 distance for Commander's drawings and Euclidean/angle errors for Stroker's goal achievement during training, indicating successful learning.

Limitations and Future Work

  • Omission of Details: The agent tended to omit fine details (e.g., strawberry seeds) in sketches, which the authors suggest could be improved by training with more detailed images.
  • Slight Errors in Sketches: Due to the inherent inaccuracies in robot control, final sketches contained slight errors.
  • Future Work - Camera Integration: The authors suggest that incorporating a camera for visual feedback of the canvas could further improve synchronization and reduce discrepancies, especially when using more complex tools like brushes or sprays. This would allow for more direct visual error correction.

Summary updated

Abstract

We present an automated learning framework for a robotic sketching agent that is capable of learning stroke-based rendering and motor control simultaneously. We formulate the robotic sketching problem as a deep decoupled hierarchical reinforcement learning; two policies for stroke-based rendering and motor control are learned independently to achieve sub-tasks for drawing, and form a hierarchy when cooperating for real-world drawing. Without hand-crafted features, drawing sequences or trajectories, and inverse kinematics, the proposed method trains the robotic sketching agent from scratch. We performed experiments with a 6-DoF robot arm with 2F gripper to sketch doodles. Our experimental results show that the two policies successfully learned the sub-tasks and collaborated to sketch the target images. Also, the robustness and flexibility were examined by varying drawing tools and surfaces.
BibTeX
@inproceedings{lee2022from,
  author = {Ganghun Lee and Minji Kim and Min Su Lee and Byoung-Tak Zhang},
  title = {{From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
}
Figure