Rigid robots interact with the world through discrete, predictable joints. If you know the angles of a robot's joints—measured easily by mechanical encoders—you can use standard kinematics to calculate the exact position of its entire body in 3D space.
Soft robots upend this paradigm. Built from highly flexible elastomers, they deform continuously across their entire structure, giving them near-infinite degrees of freedom (Annan Zhang et al.). While this compliance makes them remarkably safe and adaptable, it introduces a fundamental challenge: proprioception. Without rigid joints, how can a soft robot know its own shape, posture, or the forces it is exerting on its environment?
To solve this, researchers are moving away from traditional rigid hardware and developing integrated, data-driven sensing frameworks that capture high-dimensional deformations without destroying the robot's inherent softness.
The Compliance-Sensing Paradox
The most intuitive way to give a soft robot a sense of touch or position is to embed sensors directly into its body. However, traditional electronic force and strain sensors are rigid. As Nan Lin et al. note in their work on pneumatic actuators, integrating these stiff components creates a severe material mismatch, often compromising the robot's flexibility and safety. Even when force sensors are added to soft fingers, they frequently require a rigid backing to ensure accurate readings, which directly undermines the actuator's physical adaptability (Bianca S. Homberg et al.).
To maintain compliance, researchers have turned to flexible alternatives, such as conductive sponges, hydrogels, and soft polymer optical fibers (Jing Shu et al., Lei Zhao et al.). Yet, these materials introduce a new problem: multimodal confusion.
When a soft sensor stretches, twists, or compresses, its electrical or optical properties change. However, a simple single-mode sensor cannot distinguish which type of deformation is occurring. As Cheng Yu et al. point out, a sensor might register the same electrical resistance change whether it is being bent or compressed, severely limiting the robot's state awareness.
Decoupling Structure from Sensing via Internal Vision
To bypass the limitations of skin-embedded electronics, a powerful alternative has emerged: internal vision-based proprioception. Instead of placing sensors on the deforming material, researchers place a miniature camera inside a hollow soft actuator.
[Camera Base] ---> Watches ---> [Internal Cavity Walls] ---> Detects Deformation
In this setup, introduced by Yu She et al., the camera remains isolated from the mechanical stresses of external contact. As the soft finger bends or touches an object, the walls of its internal cavity deform. The camera captures these high-resolution visual changes in real time. Because the camera does not physically restrict the elastomer, the robot retains 100% of its natural compliance.
This optical approach allows a single sensor to achieve dual functionality. For example, the PneuGelSight gripper developed by Rongqiang Zhang et al. uses an internal camera to simultaneously track the overall 3D shape of the finger (proprioception) and the fine textures of grabbed objects (tactile sensing). By looking inward, the robot captures high-dimensional state changes that would require an impossibly complex array of traditional electronic sensors.
Solving the Inverse Problem with Deep Learning
Whether a system uses internal cameras, magnetic fields, or flexible materials, it faces a daunting computational hurdle: the inverse problem. Translating raw, non-linear sensor streams—like a distorted video frame or a fluctuating magnetic field—into precise 3D shapes or contact forces is analytically intractable (Takahiro INOUE et al., Thomas Baaij et al.).
Deep learning bridges this gap by acting as a universal function approximator. Instead of deriving complex physics equations for a squishy material, researchers train neural networks to map sensor signals directly to spatial configurations.
For instance, Xiaohan Zhu et al. fused monocular images with motor actuation signals using a cross-attention mechanism and an LSTM network, allowing a cable-driven soft robot to estimate 3D contact forces with a relative error of less than 1%. Similarly, Fang Wan 0002 et al. applied deep learning to track the internal deformations of a 3D-printed metamaterial finger, successfully extracting high-fidelity 6D force and torque data.
[Raw Sensor Input]
(Internal Video / Resistance)
│
▼
[Neural Network] ───► [Learned Inverse Mapping] ───► [Precise 3D Shape / Force]
By leveraging data-driven models, soft robots can infer external disturbances purely from internal state changes, eliminating the need for external tracking infrastructure (Ryan L. Truby).
Overcoming the Ground-Truth Bottleneck
To train these deep learning models, researchers need a massive amount of labeled data. They must know the exact ground-truth 3D shape or force of the soft robot to teach the network how to interpret the sensor signals. However, obtaining this ground truth in the real world is incredibly difficult; you cannot easily place tracking markers inside a solid, squishy finger without altering its physics (Yashraj Narang et al.).
The solution lies in Sim-to-Real transfer learning. Rather than collecting thousands of data points on physical hardware, researchers build high-fidelity physics simulations (such as Finite Element Method models) of the soft robot.
In a breakthrough approach, Uksang Yoo et al. demonstrated a pipeline where an internal camera's data-driven proprioception model is trained entirely within a virtual environment. Because the simulator knows the exact coordinates of every particle of the virtual robot, it provides perfect ground-truth labels. By applying clever calibration and domain randomization, the learned model achieves "zero-shot" transfer—meaning the neural network can be deployed directly onto a real, physical pneumatic soft robot, instantly reconstructing its 3D shape with high fidelity.
Through this combination of compliant structural design, internal non-invasive sensing, and simulation-accelerated machine learning, modern soft robots are gaining the precise self-awareness required for complex, real-world manipulation tasks.
Key Foundational Work
Sensing Pressure Distribution on a Lower-Limb Exoskeleton Physical Human-Machine Interface (2011) — contributed 1 concept elements to this direction.
Development of a Micro-Force/Moment Sensor Embedded Soft Fingertip and Its Experimental Verification by Compression Test (2007) — contributed 1 concept elements to this direction.