Introduction
Tendon-driven continuum robots (TDCRs) are increasingly utilized in specialized fields such as minimally invasive surgery and search-and-rescue operations due to their ability to navigate complex, constrained environments. Unlike traditional rigid-link robots, TDCRs lack discrete joints and instead bend continuously along their backbone. While this flexibility is a primary advantage, it presents a significant challenge for shape estimation. Traditional physics-based models often fail to account for non-linearities such as material stretching, friction, and mechanical hysteresis—phenomena that are exacerbated when the robot is subjected to external loading or carrying payloads. This paper introduces ST-Net, a spatiotemporal neural network designed to provide robust, high-precision 3D shape estimation by fusing visual data with internal proprioceptive sensors.
The ST-Net Architecture
The proposed ST-Net architecture is a multimodal framework that processes both spatial and temporal data streams to reconstruct the robot's pose. The system is comprised of five distinct modules:
- Spatial Feature Extractor (SFE): Utilizing a U-Net-inspired encoder, this module extracts high-dimensional features from real-time RGB camera images, capturing the visual state of the robot.
- Temporal Feature Extractor (TFE): A Long Short-Term Memory (LSTM) network processes sequences of tendon displacements. This allows the system to "remember" previous states, which is critical for modeling the history-dependent nature of mechanical hysteresis.
- Multimodal Feature Fusion (FF): This module merges the spatial and temporal streams. It employs a spatial attention mechanism to focus the network's processing power on the robot's structure while filtering out irrelevant background noise.
- Pose Predictor (PP): A fully connected layer translates the fused features into a discrete 3D point cloud representing the robot’s backbone.
- Bézier Curve Fitting: To ensure the final output is physically plausible and mathematically smooth, the system fits a 4th-order Bézier curve to the predicted points, resulting in a continuous 3D representation.
Experimental Methodology
To validate the ST-Net, the researchers developed a custom experimental platform featuring a 2-degree-of-freedom TDCR driven by four tendons. An Intel RealSense RGB-D camera provided the visual input, while ground-truth data was established using fiducial markers.
The dataset consisted of 880 samples per trial across five motion cycles. To test the system's robustness under stress, the researchers evaluated the robot in a "Free-Space" (unloaded) configuration and under three distinct external loading conditions ($F_{e1}, F_{e2}, F_{e3}$), simulated by a 10g mass attached via a pulley system. This setup allowed the team to measure how effectively the network could compensate for the deformations caused by external forces.
Key Insights and Results
The experimental results demonstrate that ST-Net significantly outperforms existing state-of-the-art methods. In unloaded conditions, the network achieved a Root Mean Square Error (RMSE) of 0.08 mm. Under external loading, the error remained remarkably low at 0.22 mm.
Several key findings emerged from the study:
- The Necessity of Multimodality: Comparisons showed that vision-only models (like MoSSNet) and proprioception-only models (TFE-Net) both suffered significant accuracy degradation under load. The fusion of "sight" and "touch" is essential for handling external forces.
- Temporal Awareness: The inclusion of the LSTM allowed the network to account for the mechanical "lag" inherent in tendon-driven systems. By considering the sequence of movements rather than just the current frame, the model effectively mitigated the impact of friction and hysteresis.
- Precision and Robustness: ST-Net's 0.08 mm error in free space is nearly ten times more accurate than the 0.78 mm error produced by MoSSNet. Furthermore, the attention mechanism ensured that the model remained accurate even when the environment changed, as the network learned to prioritize the robot's deformable segments over the background.
Conclusion and Future Work
ST-Net represents a significant advancement in the control and monitoring of flexible robotics, offering sub-millimeter precision even under external loads. However, the researchers noted certain limitations, such as the potential for the loading wire to act as a visual cue during training. Future research will focus on scaling this architecture for longer, multi-segment robots and testing its performance in more complex, cluttered environments, such as those encountered during interaction with soft biological tissues in surgical contexts.
Summary updated