Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning

Mohamed Ashraf Mostafa, Taher El-Fakharany, Samir KhaledarXiv 2026 · arXiv

TL;DR

StabBot is a robotic system that uses stereo vision and residual reinforcement learning to "catch" and connect swinging drill pipes at matched velocities. By predicting the pipe's pendulum-like motion, it reduces impact forces by nearly 8x and achieves a 100% connection success rate in simulations, potentially removing human workers from one of the most dangerous areas of a drill floor.

What Problem Does This Paper Address?

In oil and gas drilling, "stabbing" is the process of connecting a new joint of pipe (the "pin") to the one already in the well (the "box"). This is a high-stakes game of catch: the new pipe hangs from a 10-meter-long elevator and swings unpredictably.

Current automated systems are "passive"—they park a guide at the center of the well and wait for the swinging pipe to smash into it. This creates massive shock loads that damage equipment and pose a severe safety risk to "floorhands" who often have to manually steady the pipe in the "red zone." The authors aim to transform this violent impact into a controlled, "soft catch."

Methods

The authors developed StabBot, an integrated hardware and software system consisting of:

  • Swing Estimation: A Kalman filter that uses a physical pendulum model (tuned by the pipe's known length and mass) to predict where the pipe will be, even if the camera feed is delayed or interrupted.
  • Stereo Keypoint Vision: A ResNet-18 network that processes 30Hz stereo images to locate the pipe axis, pin nose, and box mouth in 3D space with sub-millimeter accuracy.
  • Velocity-Matched Control: Instead of waiting at the center, the robot arm "shadows" the pipe's motion and closes its clamps only when its speed matches the pipe's speed.
  • Residual Reinforcement Learning (RL): A PPO-trained policy provides fine-tuned corrections to the model-based controller, specifically learning to minimize peak forces during the final milliseconds of contact.
  • Universal Hardware: A novel open V-shaped roller head (Figure 3) designed to handle nine different pipe sizes (from 2-3/8" tubing to 20" casing) without mechanical reconfiguration.
The StabBot system overview, showing the integration of vision, filtering, and RL control.

Datasets

The study utilized a 1:4 Froude-scaled MuJoCo simulation environment.

  • Vision Training: 20,000 rendered stereo pairs with diverse lighting, pipe sizes, and "unseen" conditions (glare, motion blur, occlusions) were used to train the keypoint network.
  • Benchmark: A newly introduced benchmark of 200 paired random jobs. Each job tested a specific pipe size and swing amplitude, allowing for a direct head-to-head comparison between StabBot and traditional fixed-guide methods.

Key Insights and Ideas

  • The Power of Physics-Informed Filtering: By embedding a pendulum model into the Kalman filter, the system became incredibly resilient. It maintained a 100% success rate even with 85% frame loss or 300ms of latency, because the filter "knew" how a pipe is physically allowed to move.
  • Residual RL as a "Polishing" Layer: The authors found that a pure RL approach might "game" the reward system (e.g., closing clamps too early). By using RL only to provide corrections to a stable model-based controller, they achieved the best of both worlds: reliability and extreme gentleness.
  • Touch-Confirmed Approach: The system doesn't just rely on vision. It uses a "guarded approach" where the final millimeters of closure are triggered by force sensors, ensuring a secure grip even if the visual estimate has a slight offset.

Key Results

In the 200-job benchmark, StabBot outperformed the conventional fixed guide across every metric:

  • Success Rate: 100% for StabBot vs. 74% for the fixed guide. The fixed guide frequently missed the pipe entirely or struck the pipe body before the pin could seat.
  • Force Reduction: Median peak impact force dropped from 11.3 kN to 1.4 kN (a 7.8x reduction).
  • Speed Reduction: Median contact speed fell from 18.4 cm/s to 2.3 cm/s.
  • Efficiency: While StabBot takes slightly longer to "watch" the swing (about 1.8s longer on average), it eliminates the need for "re-stabs" caused by missed captures, potentially saving hours of rig time over a thousand connections.
A comparison of a single job: the fixed guide (left) waits for impact, while StabBot (right) tracks the pipe and matches its speed for a soft catch.

Limitations and Future Work

  • Simulation to Reality: While Froude scaling is a standard engineering practice, the results are still based on a MuJoCo simulation. Real-world factors like wind, complex elevator link dynamics, and the "messiness" of a real drill floor (mud, grease on lenses) remain to be tested.
  • Visual Fidelity: The keypoint network struggled slightly with specific pipe colors (yellow) that weren't in the initial training set, though retraining (v3) mitigated this.
  • Next Steps: The authors plan to move to a full-scale physical test stand with real stereo cameras and instrumented pipes to validate the "sim-to-real" transition.

Summary updated

Abstract

A joint of pipe run into the hole swings from the elevator over the well and strikes a fixed stabbing guide at whatever speed the swing has. StabBot, a robotic stabbing system, predicts the swing and meets the pipe at its own speed. A pendulum Kalman filter tuned from the pipe tally tracks the joint in stereo images. A six-axis arm with an open V-shaped roller head closes in at matched velocity, confirms the last millimetres by touch, aims the pin over the box when the elevator hangs off centre, and follows the block down with lateral compliance. A ResNet-18 keypoint network locates the pipe, pin nose and box mouth in 3-D, and a residual reinforcement-learning policy refines the capture. On 200 paired random jobs with nine sizes from 2-3/8-in. tubing to 20-in. casing, StabBot seated every joint with the pipe touching only rollers, against 74% for a conventional fixed guide. Median contact speed fell from 18.4 to 2.3 cm/s and median peak head-pipe force from 11.3 to 1.4 kN, and the force was lower in all 200 jobs (Wilcoxon $p < 10^{-30}$). The network locates the pipe with 0.4-mm median error in 2.3 ms per stereo pair. With the network in the loop, success stayed at 100% under 20 mm of added noise, 300-ms delay, 85% frame loss, 20-mm calibration offset, 20-cm elevator offset and 40-cm swings. All results come from a 1:4 Froude-scaled MuJoCo model and are reported at full scale.
BibTeX
@misc{mostafa2026predict,
  author = {Mohamed Ashraf Mostafa and Taher El-Fakharany and Samir Khaled},
  title = {{Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning}},
  year = {2026},
  eprint = {2610.37535},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37535},
}
Figure