Abstract
Sensory feedback is essential for the control of soft robotic systems and to enable deployment in a variety of different tasks. Proprioception refers to sensing the robot's own state and is of crucial importance in order to deploy soft robotic systems outside of laboratory environments, i.e. where no external sensing, such as motion capture systems, is available. A vision-based sensing approach for a soft robotic arm made from fabric is presented, leveraging the high-resolution sensory feedback provided by cameras. No mechanical interaction between the sensor and the soft structure is required and consequently, the compliance of the soft system is preserved. The integration of a camera into an inflatable, fabric-based bellow actuator is discussed. Three actuators, each featuring an integrated camera, are used to control the spherical robotic arm and simultaneously provide sensory feedback of the two rotational degrees of freedom. A convolutional neural network architecture predicts the two angles describing the robot's orientation from the camera images. Ground truth data is provided by a motion capture system during the training phase of the supervised learning approach and its evaluation thereafter. The camera-based sensing approach is able to provide estimates of the orientation in real-time with an accuracy of about one degree. The reliability of the sensing approach is demonstrated by using the sensory feedback to control the orientation of the robotic arm in closed-loop.
Introduction
Soft robots offer high versatility and safety for human interaction, yet their flexible nature introduces complex physical dynamics that are difficult to model. Accurate control requires "proprioception"—the internal sensing of shape and orientation. While laboratory settings often rely on external motion capture systems, real-world deployment necessitates embedded sensing. Traditional methods involve embedding physical strain sensors, which can introduce mechanical stiffness mismatches and stress concentrations, limiting the robot's lifespan. This research addresses the challenge of accurately sensing continuous deformation without mechanical interference or external dependencies.
Methodology
The proposed approach decouples the sensor from the soft material by integrating tiny (7x7 mm) cameras and LED light sources inside three fabric-based pneumatic bellow actuators that drive a spherical robotic arm. The internal fabric of these actuators is marked with a high-contrast white pattern of dots and rings. As the actuators inflate and deflate, the internal cameras capture the resulting deformation of these patterns.
A deep learning framework was employed to interpret these visual changes. A custom, lightweight Convolutional Neural Network (CNN) with approximately 9,300 parameters was designed to process downsampled images from the three cameras. The network directly predicts the orientation of the arm, defined by two angles, $\alpha$ and $\beta$.
Data Collection
A custom dataset was generated using an infrared motion capture system to provide ground truth at 200 Hz. The training set comprised approximately 54,000 images (90 minutes of data), while the validation set included 15,000 images (25 minutes). The arm was moved through a grid of setpoints ranging from $-30^\circ$ to $30^\circ$ for both $\alpha$ and $\beta$.
Results and Discussion
The sensing approach demonstrated high accuracy and efficiency. The CNN achieved a combined Root-Mean-Square Error (RMSE) of $1.15^\circ$ on the test set. Notably, the system operates in real-time at 30 Hz on a standard Intel Core i7 laptop CPU, demonstrating that vision-based proprioception does not require high-end GPUs for deployment.
In closed-loop control experiments, the arm followed sinusoidal trajectories with an RMSE of $1.1^\circ$ for $\alpha$ and $1.5^\circ$ for $\beta$. By looking inward, the sensing system is shielded from external environmental factors like lighting changes or occlusions, ensuring robustness.
Conclusion
This vision-based approach preserves the natural compliance of soft robots by eliminating mechanical wear at the sensor-material interface. Future work will explore incorporating baseline pressure into the model to account for variable stiffness, testing the system under external disturbances, and utilizing Recurrent Neural Networks (RNNs) to capture temporal dynamics such as hysteresis.
Summary updated