Abstract
Translating static line art into robotic motion requires converting pixel data into ordered stroke sequences. This paper presents a multi-stage pipeline that utilizes deep learning-based segmentation and a graph-based feedback loop to achieve high-accuracy "raster-to-stroke" conversion. The system achieves a 98.02% IoU and has been successfully deployed on CNC and robotic arm hardware.
Introduction
While humans intuitively understand the sequence of strokes needed to replicate a drawing, robots struggle with this "raster-to-stroke" conversion. Standard computer vision techniques typically focus on vectorization or semantic segmentation, which often fail to preserve the specific stroke information required for physical drawing. This research addresses the gap between static pixel grids and dynamic robotic motor instructions (GCODE/SVG).
Methodology
The proposed system follows a three-part pipeline to bridge the gap between images and motion.
1. Image-to-Image Translation
The process begins with a modified U-Net architecture that transforms a grayscale input image into a three-channel segmentation mask identifying background, corners (vertices), and lines (edges). To maintain high fidelity, the network employs "valid convolutions" to prevent artifacts and a "max-weighted" loss function. This loss function is critical because structural elements like lines and corners occupy a tiny fraction of the total image area compared to the background.
2. Graph Interpretation and Feedback Loop
The segmentation mask is converted into a mathematical graph where corners are nodes and lines are edges. A unique feature of this system is its iterative feedback loop. The system "renders" its current graph and compares it back to the original image. By identifying "absent blobs" (missing details) or "superfluous blobs" (extra lines), the system dynamically adjusts its detection thresholds to refine the final output.
3. Execution
Finally, a recursive algorithm traverses the refined graph to generate a sequence of strokes. These are converted into GCODE for CNC plotters or SVG paths for robotic arms.
Datasets and Training
The researchers utilized the Quick-draw dataset, rasterizing the original vector strokes into images to simulate robotic vision. Intersections and endpoints were algorithmically labeled to create ground truth data for corners and lines. The final benchmark consisted of approximately 65,000 training samples and 10,000 test samples.
Results
The system demonstrated both quantitative and qualitative success.
Quantitative Performance
The "Net (7 3 4 3) with MWX" model achieved a 98.02% Intersection over Union (IoU), significantly outperforming baseline U-Net architectures (94.23%).
Qualitative and Physical Deployment
The pipeline was successfully deployed on an EMCO Concept Mill 250 CNC machine and a Dobot Magician robotic arm. Notably, the model excelled at handling complex "hatch patterns"—dense, overlapping lines that typically cause traditional vectorization methods to fail or over-simplify.
Conclusion and Future Work
While highly accurate, the system remains sensitive to image noise and relies on heuristic initial thresholds. Future work will focus on improving robustness in real-world environments and enabling real-time collaborative sketching between humans and robots.
Summary updated