Detection and Physical Interaction with Deformable Linear Objects

Azarakhsh Keipour, Mohammadreza Mousaei, Maryam Bandari, Stefan Schaal, Sebastian Scherer

TL;DR

This paper presents an automated pipeline for detecting and modeling Deformable Linear Objects (DLOs)—like cables and ropes—in complex scenarios involving occlusions and self-crossings. By representing DLOs as a chain of segments, the authors provide a robust initialization for robotic tracking and simulation, and further demonstrate the feasibility of using aerial robots for delicate tasks like unplugging a USB cable.

What Problem Does This Paper Address?

While robots have become adept at handling rigid objects, Deformable Linear Objects (DLOs) remain a significant challenge. Cables, wires, and sutures are ubiquitous in industrial and medical settings, yet they are notoriously difficult for robots to perceive because they can take infinite shapes, overlap themselves, and be partially hidden by other objects.

Existing research typically falls into two camps: segmentation (identifying which pixels are "cable") and tracking (following a cable's movement). However, most tracking algorithms require a manual "initial state" to begin. This paper bridges that gap by providing an automated way to extract a physics-ready model of a DLO from a single image, even when the cable is tangled or partially obscured. Furthermore, it explores the untapped potential of aerial manipulation, asking: can a drone perform the precise, high-force tasks required to manipulate cables?

Methods

The authors propose a six-step detection pipeline that transforms raw image data into a mathematical model (a chain of fixed-size segments connected by spherical joints):

  1. Segmentation & Skeletonization: Standard computer vision techniques filter the image and reduce the DLO to a single-pixel-wide "skeleton."
  2. Contour Extraction: The skeleton is converted into a sequence of points.
  3. DLO Fitting & Pruning: The algorithm traverses the points to create a chain of cylindrical segments. Overlapping segments are "pruned" to ensure a clean, non-redundant model.
  4. Merging (The "Aha" Step): To handle gaps caused by occlusions or crossings, the algorithm calculates a "connection cost" between chain ends. This cost is based on Euclidean distance, direction alignment, and "natural curvature"—the idea that a cable's path through a gap likely follows a constant turn rate.

For the manipulation component, the team used the Gazebo and MATLAB ARCAD simulators to test a fully-actuated hexarotor (a drone with six rotors that can tilt) equipped with a gripper.

Datasets

The method was evaluated on:

  • 7 video sequences consisting of 4,230 frames (1280×720 resolution).
  • The sequences included challenging scenarios with significant occlusions and DLO crossings.
  • The authors compared their automated merges and occlusion-filling against ground truth data to calculate accuracy.

Key Insights and Ideas

  • Physics-Ready Output: Unlike simple edge detection, this method outputs a chain-of-segments model. This is crucial because it is the exact format required by modern physics engines (like MuJoCo or Bullet) for dynamic simulation.
  • The "Natural Curvature" Heuristic: The authors realized that you can't just connect two ends of a broken cable with a straight line. By calculating the "natural" path based on the exit angle of the known segments, they can accurately predict where a cable goes when it disappears behind an obstacle.
  • Aerial Wrench Constraints: A key insight in the aerial manipulation section is the analysis of force polytopes. Drones have limited "wrench" (the combination of force and torque they can apply). The paper shows that while a drone can unplug a USB cable (requiring ~15.84N), it operates very close to its physical limits, leaving little room for error or sudden gusts of wind.

Key Results

  • Efficiency: The average detection time was 0.537 seconds per frame on a standard i9 laptop. While not yet "real-time" (30fps), it is fast enough for many autonomous initialization tasks.
  • Accuracy:
    • 83.7% accuracy for overall frame detection.
  • 90.7% accuracy in correctly filling occlusions.
  • 99.6% accuracy in successfully merging chain segments.
  • Feasibility: In simulation, the system successfully guided a UR3 robotic arm to route cables and a hexarotor to grasp cable segments with a mean error of only 7.84 mm.

Limitations and Future Work

  • 2D to 3D: The current detection method primarily operates in 2D image space. The authors identify extending this to full 3D detection as a primary next step.
  • Crossing Logic: While the system handles crossings, it doesn't yet explicitly determine which segment is on top of the other—a vital piece of information for untangling knots.
  • Real-World Aerial Tests: The aerial manipulation results are currently limited to simulation. Moving from Gazebo to real-world flight will introduce challenges like ground effect, battery sag, and sensor noise.
  • Optimization: To reach true real-time performance, the Python-based implementation needs further optimization or porting to a lower-level language like C++.

Summary updated

Abstract

Deformable linear objects (e.g., cables, ropes, and threads) commonly appear in our everyday lives. However, perception of these objects and the study of physical interaction with them is still a growing area. There have already been successful methods to model and track deformable linear objects. However, the number of methods that can automatically extract the initial conditions in non-trivial situations for these methods has been limited, and they have been introduced to the community only recently. On the other hand, while physical interaction with these objects has been done with ground manipulators, there have not been any studies on physical interaction and manipulation of the deformable linear object with aerial robots. This workshop describes our recent work on detecting deformable linear objects, which uses the segmentation output of the existing methods to provide the initialization required by the tracking methods automatically. It works with crossings and can fill the gaps and occlusions in the segmentation and output the model desirable for physical interaction and simulation. Then we present our work on using the method for tasks such as routing and manipulation with the ground and aerial robots. We discuss our feasibility analysis on extending the physical interaction with these objects to aerial manipulation applications.
BibTeX
@misc{keipour2022detection,
  author = {Azarakhsh Keipour and Mohammadreza Mousaei and Maryam Bandari and Stefan Schaal and Sebastian Scherer},
  title = {{Detection and Physical Interaction with Deformable Linear Objects}},
  year = {2022},
  eprint = {2205.08041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.08041},
}
Figure