Can I teach a robot to replicate a line art

Raghav Brahmadesam Venkataramaiyer, Subham Kumar, Vinay P. Namboodiri

Abstract

Translating static line art into robotic motion requires converting pixel data into ordered stroke sequences. This paper presents a multi-stage pipeline that utilizes deep learning-based segmentation and a graph-based feedback loop to achieve high-accuracy "raster-to-stroke" conversion. The system achieves a 98.02% IoU and has been successfully deployed on CNC and robotic arm hardware.

Introduction

While humans intuitively understand the sequence of strokes needed to replicate a drawing, robots struggle with this "raster-to-stroke" conversion. Standard computer vision techniques typically focus on vectorization or semantic segmentation, which often fail to preserve the specific stroke information required for physical drawing. This research addresses the gap between static pixel grids and dynamic robotic motor instructions (GCODE/SVG).

Methodology

The proposed system follows a three-part pipeline to bridge the gap between images and motion.

The multi-stage pipeline from user-created line art to CNC plotter execution.

1. Image-to-Image Translation

The process begins with a modified U-Net architecture that transforms a grayscale input image into a three-channel segmentation mask identifying background, corners (vertices), and lines (edges). To maintain high fidelity, the network employs "valid convolutions" to prevent artifacts and a "max-weighted" loss function. This loss function is critical because structural elements like lines and corners occupy a tiny fraction of the total image area compared to the background.

2. Graph Interpretation and Feedback Loop

The segmentation mask is converted into a mathematical graph where corners are nodes and lines are edges. A unique feature of this system is its iterative feedback loop. The system "renders" its current graph and compares it back to the original image. By identifying "absent blobs" (missing details) or "superfluous blobs" (extra lines), the system dynamically adjusts its detection thresholds to refine the final output.

3. Execution

Finally, a recursive algorithm traverses the refined graph to generate a sequence of strokes. These are converted into GCODE for CNC plotters or SVG paths for robotic arms.

Datasets and Training

The researchers utilized the Quick-draw dataset, rasterizing the original vector strokes into images to simulate robotic vision. Intersections and endpoints were algorithmically labeled to create ground truth data for corners and lines. The final benchmark consisted of approximately 65,000 training samples and 10,000 test samples.

Results

The system demonstrated both quantitative and qualitative success.

Qualitative comparison between original sketches (top) and the reconstructed graph-based strokes (bottom).

Quantitative Performance

The "Net (7 3 4 3) with MWX" model achieved a 98.02% Intersection over Union (IoU), significantly outperforming baseline U-Net architectures (94.23%).

Qualitative and Physical Deployment

The pipeline was successfully deployed on an EMCO Concept Mill 250 CNC machine and a Dobot Magician robotic arm. Notably, the model excelled at handling complex "hatch patterns"—dense, overlapping lines that typically cause traditional vectorization methods to fail or over-simplify.

Conclusion and Future Work

While highly accurate, the system remains sensitive to image noise and relies on heuristic initial thresholds. Future work will focus on improving robustness in real-world environments and enabling real-time collaborative sketching between humans and robots.

Summary updated

Abstract

Line art is arguably one of the fundamental and versatile modes of expression. We propose a pipeline for a robot to look at a grayscale line art and redraw it. The key novel elements of our pipeline are: a) we propose a novel task of mimicking line drawings, b) to solve the pipeline we modify the Quick-draw dataset and obtain supervised training for converting a line drawing into a series of strokes c) we propose a multi-stage segmentation and graph interpretation pipeline for solving the problem. The resultant method has also been deployed on a CNC plotter as well as a robotic arm. We have trained several variations of the proposed methods and evaluate these on a dataset obtained from Quick-draw. Through the best methods we observe an accuracy of around 98% for this task, which is a significant improvement over the baseline architecture we adapted from. This therefore allows for deployment of the method on robots for replicating line art in a reliable manner. We also show that while the rule-based vectorization methods do suffice for simple drawings, it fails for more complicated sketches, unlike our method which generalizes well to more complicated distributions.
BibTeX
@inproceedings{venkataramaiyer2020can,
  author = {Raghav Brahmadesam Venkataramaiyer and Subham Kumar and Vinay P. Namboodiri},
  title = {{Can I teach a robot to replicate a line art}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093434},
}
Figure