A Vision-Based Sensing Approach for a Spherical Soft Robotic Arm

Matthias Hofer, Carmelo Sferrazza, Raffaello D'Andrea

TL;DR

This paper presents a novel proprioceptive sensing approach for a soft robotic arm by placing tiny cameras inside its pneumatic actuators. By training a lightweight Convolutional Neural Network (CNN) to observe the internal deformation of the fabric, the system accurately predicts the arm's orientation in real-time, enabling closed-loop control on a standard laptop CPU without relying on external motion capture systems.

What Problem Does This Paper Address?

Soft robots are highly versatile and safe for human interaction, but their flexible nature makes their physical dynamics incredibly complex and difficult to model. To control them accurately, robots need "proprioception"—the ability to sense their own shape and orientation.

In laboratory settings, researchers often use external motion capture cameras to track soft robots, but this is useless for real-world deployment. Alternatively, embedding physical strain sensors (like piezoresistive or capacitive materials) into the robot's body is common, but this introduces mechanical challenges. The sensor's stiffness must match the robot's soft material, and the physical connection between the two often creates stress concentrations that limit the robot's lifespan. This paper asks: How can we accurately sense the continuous deformation of a soft robot without mechanically interfering with its structure or relying on external sensors?

Methods

The authors tackle this by decoupling the sensor from the soft material entirely. They built a spherical soft robotic arm driven by three fabric-based pneumatic bellow actuators. Inside each actuator, they integrated a tiny (7x7 mm) camera and an LED light source.

To make the deformation easy for the camera to "see," the inner black fabric of the actuators was painted with a high-contrast white pattern of dots and rings. As the actuators inflate and deflate to move the arm, the internal cameras record how this white pattern warps and shifts.

Instead of manually engineering visual features, the team used an end-to-end deep learning approach. They designed a custom, lightweight CNN (inspired by LeNet, with only ~9,300 parameters). The network takes the three downsampled camera images as inputs and directly outputs the two angles ($\alpha$ and $\beta$) that define the robotic arm's orientation. These predicted angles are then fed into a Proportional-Integral (PI) controller to adjust the air pressure in the actuators, achieving closed-loop control.

Datasets

The authors collected a custom dataset to train their CNN. Ground truth data was provided by an external infrared motion capture system tracking the arm at 200 Hz.

  • Training Set: ~54,000 images and corresponding angle labels (about 90 minutes of data).
  • Validation Set: ~15,000 images and labels (about 25 minutes of data). The data was generated by having a controller move the arm through a regular grid of setpoints ranging from $-30^\circ$ to $30^\circ$ for both the $\alpha$ and $\beta$ angles.

Key Insights and Ideas

  • Mechanical decoupling is a game-changer: Because the camera simply looks at the fabric without touching the deforming parts, it preserves the robot's natural compliance. There is no mechanical wear-and-tear at a sensor-material interface.
  • Looking inward solves environmental robustness: By pointing the cameras at the inside of the robot, the sensing system is completely shielded from external lighting changes, visual noise, or occlusions. The robot carries its own perfectly controlled visual environment.
  • Deep learning doesn't always require heavy compute: By keeping the CNN architecture shallow and optimizing the physical visual pattern, the authors proved that vision-based proprioception can run in real-time on standard consumer hardware, bypassing the need for bulky GPUs during deployment.

Key Results

  • High Accuracy: The CNN achieved a combined Root-Mean-Square Error (RMSE) of $1.15^\circ$ on the test set.
  • Real-Time CPU Performance: The network successfully runs at 30 Hz on a standard Intel Core i7 laptop CPU.
  • Successful Closed-Loop Control: During live tracking experiments where the arm was commanded to follow sinusoidal trajectories, the vision-based feedback allowed the arm to track the target with an RMSE of $1.1^\circ$ for the $\alpha$ angle and $1.5^\circ$ for the $\beta$ angle.

Limitations and Future Work

The authors note a few limitations and exciting directions for future research:

  • Variable Stiffness: The current model was trained at a fixed baseline pressure. Because soft robots can change their joint stiffness by altering this baseline pressure (which changes how the fabric deforms), future networks will need to include the baseline pressure as an input to maintain accuracy across different stiffness levels.
  • External Disturbances: The system was evaluated on unhindered trajectory tracking. Future work needs to test how the sensing holds up when the arm interacts with objects or experiences external forces.
  • Hardware Reduction: Because the arm's overall orientation affects the shape of all three actuators, it might be possible to predict the arm's state using only one or two cameras, simplifying the hardware.
  • Temporal Dynamics: The current CNN only looks at static, independent frames. Using Recurrent Neural Networks (RNNs) could help the system understand time-dependent material behaviors, like hysteresis, potentially improving accuracy.

Summary updated

Abstract

Sensory feedback is essential for the control of soft robotic systems and to enable deployment in a variety of different tasks. Proprioception refers to sensing the robot’s own state and is of crucial importance in order to deploy soft robotic systems outside of laboratory environments, i.e. where no external sensing, such as motion capture systems, is available. A vision-based sensing approach for a soft robotic arm made from fabric is presented, leveraging the high-resolution sensory feedback provided by cameras. No mechanical interaction between the sensor and the soft structure is required and consequently the compliance of the soft system is preserved. The integration of a camera into an inflatable, fabric-based bellow actuator is discussed. Three actuators, each featuring an integrated camera, are used to control the spherical robotic arm and simultaneously provide sensory feedback of the two rotational degrees of freedom. A convolutional neural network architecture predicts the two angles describing the robot’s orientation from the camera images. Ground truth data is provided by a motion capture system during the training phase of the supervised learning approach and its evaluation thereafter. The camera-based sensing approach is able to provide estimates of the orientation in real-time with an accuracy of about one degree. The reliability of the sensing approach is demonstrated by using the sensory feedback to control the orientation of the robotic arm in closed-loop.
BibTeX
@inproceedings{hofer2021vision,
  author = {Matthias Hofer and Carmelo Sferrazza and Raffaello D'Andrea},
  title = {{A Vision-Based Sensing Approach for a Spherical Soft Robotic Arm}},
  booktitle = {Frontiers Robotics AI},
  year = {2021},
  url = {https://openreview.net/forum?id=S_B0kPEcbb},
}

Loading related content...

Figure