In traditional computer vision, the world is often assumed to be a "rigid body." If you take a picture of a room from two different angles, the furniture doesn't move or change shape between shots. This assumption is the bedrock of Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM).
However, inside the human body, this assumption collapses. As noted by Sven Haase and Anita Rau, biological tissues are in constant motion due to respiration, heartbeats, and peristalsis. When a surgeon interacts with an organ, it deforms. For a robot or a surgeon to navigate safely, they need a 3D map that understands not just where the camera is, but how the "furniture" is breathing.
The Problem of Disentanglement
The fundamental challenge in surgical reconstruction is disentanglement: if a point in an endoscopic video moves, is it because the camera moved, or because the tissue stretched?
Standard SLAM algorithms, like those discussed in the EndoSLAM project by Kutsev Bengisu Ozyoruk, often fail because they cannot distinguish between these two types of motion. Furthermore, as Nicholas J. Durr points out, the light source in an endoscope is attached to the camera. This means the lighting is never static; as the camera moves, the shadows and highlights shift, violating the "brightness constancy" assumption used by most tracking algorithms.
From Static NeRFs to Dynamic 4D Fields
To solve this, researchers have moved toward "4D" representations—models that treat time as a fundamental dimension of the 3D scene.
A landmark approach is BASED (Sainan Liu), which uses Neural Radiance Fields (NeRFs) to jointly optimize for camera pose and tissue deformation. Instead of assuming the scene is a fixed set of coordinates, it learns a "canonical" 3D shape and a set of "deformation fields" that describe how that shape warps at any given moment.
This is further refined by UC-NeRF (Jiaxin Guo), which introduces uncertainty awareness. In surgery, specular reflections (the "wet" look of organs) act like "fake" features that move independently of the tissue. UC-NeRF uses a dual-branch architecture to separate these transient reflections from the stable underlying anatomy, preventing the 3D model from being "fooled" by moving highlights.
Real-Time Mapping with Gaussian Splatting
While NeRFs produce beautiful results, they are often too slow for the operating room. The field is currently pivoting toward 3D Gaussian Splatting (3DGS), a technique that represents the world as a collection of millions of tiny, semi-transparent "blobs" (Gaussians) rather than a continuous neural function.
Recent frameworks like Free-DyGS (Qian Li) and EndoWave (Taoyu Wu) have adapted this for surgery. The intuition behind these models is to attach the Gaussians to a deformable "skeleton" or to use temporal wavelets to describe their motion.
- Free-DyGS uses a "retrospective" learning strategy to reconstruct scenes without pre-calculated camera poses, which is critical when external trackers aren't available.
- EndoWave achieves frame rates of over 70 FPS by using rational wavelets to model the high-frequency deformations of tissue, allowing for a high-fidelity 4D map that updates in real-time.
Physics-Informed Depth Estimation
When multi-view geometry fails—for instance, in a narrow colon where you can't see the same point from two angles—researchers turn to the physics of light.
The PC-Depth framework (Shijie Li) and LightDepth (Javier Rodriguez Puigvert) exploit "illumination decline." In a dark cavity, the brightness of a surface is mathematically tied to its distance from the camera's light source ($1/d^2$). By baking this physical law into the loss function of a deep network, these models can estimate 3D depth from a single 2D image without needing any manual labels.
The Path Forward
The goal of this research direction is to move from "seeing" a video to "understanding" a volume. By combining the speed of Gaussian Splatting with the robustness of physics-informed neural networks, researchers like Qi Dou and Michael C. Yip are building systems that can track a surgeon's tools, stabilize shaky video, and provide a "GPS for the body" that remains accurate even as the patient breathes.