3D Shape Reconstruction of Semi-Transparent Worms

Thomas P. Ilett, Omer Yuval, Thomas Ranner, Netta Cohen, David C. HoggCVPR 2023 · DOI

Abstract

Capturing the 3D movement of microscopic, semi-transparent organisms like C. elegans presents significant challenges for traditional computer vision. We present a robust "project-render-score" pipeline designed to reconstruct the 3D posture of nematodes in complex, noisy environments. By utilizing a differentiable renderer that models the worm as a 3D curve with super-Gaussian transparency, our system achieves state-of-the-art tracking accuracy and duration, significantly outperforming previous methods in the presence of environmental artifacts and focal blur.

Introduction

Traditional 3D reconstruction techniques typically rely on identifying and matching distinct features across multiple camera views. However, C. elegans nematodes are approximately 1mm long, move rapidly, and lack distinct surface textures. Furthermore, they are semi-transparent and often inhabit 3D gels containing bubbles, debris, and "old tracks" that mimic the subject's appearance. These factors frequently cause standard trackers to lose the subject during self-occlusion or when it moves out of the narrow focal plane. This research introduces a tracker capable of handling hours of footage without manual intervention by shifting from geometric feature matching to photometric alignment.

Methodology

The core of the proposed system is an integrated "project-render-score" algorithm that avoids the pitfalls of traditional feature-based tracking.

Geometric Modeling

The worm is represented as a 3D curve. To maintain a stable, intrinsic parametrization that is biologically constrained, the system employs Bishop frames rather than the standard Frenet-Serret frame. This choice prevents mathematical singularities when the curve is straight, ensuring the model remains robust during common worm behaviors.

Differentiable Renderer

The "secret sauce" of the pipeline is a differentiable renderer. The system projects 3D vertices into 2D space and renders them as a stack of super-Gaussian blobs. These blobs are defined by three primary parameters: spread ($\sigma$), intensity ($\iota$), and edge sharpness ($\rho$). This flexibility allows the model to accurately mimic the worm’s transparency and the varying focal blur of the microscopic lenses.

The project-render-score pipeline for 3D posture reconstruction.

Optimization and Refinement

Because the renderer is differentiable, the system can employ gradient descent to jointly optimize the worm's shape, camera positions, and rendering parameters. The objective is to minimize the difference between the rendered model and the raw pixel data from all camera views. To enhance robustness, the system includes:

  • Masking: Dynamically ignores environmental noise by focusing only on pixels near the current estimate.
  • Centre-shifting: Periodically re-aligns the model to prevent the head or tail from becoming stuck on high-contrast artifacts.
  • Self-Calibration: The optimization loop includes relative camera shifts to account for microscopic vibrations and changes in the gel's refractive index.
Super-Gaussian rendering parameters (sigma, iota, rho) used to model worm transparency and focal blur.

Experimental Setup and Results

The system was validated using a dataset of approximately 4 hours and 53 minutes of footage (440,000 frames) captured at 25Hz from three nearly-orthogonal cameras viewing a glass cube filled with a gelatin solution.

Performance Metrics

The proposed method demonstrated significant improvements in both duration and accuracy:

  • Tracking Duration: The method recovered 4 hours and 37 minutes of usable 3D data, a massive increase over previous state-of-the-art methods which managed 1 hour 32 minutes and 45 minutes, respectively.
  • Accuracy: Benchmarked against 487 manual 2D annotations, the system achieved a mean pixel distance of 1.53px, outperforming previous automated attempts.

Qualitative Success

The tracker successfully maintained focus through complex coiling maneuvers and visually obscured segments where the worm appeared only as a faint blur or was partially hidden by bubbles.

Discussion and Conclusion

By treating 3D reconstruction as a photometric alignment problem rather than a geometric one, this approach naturally handles the challenges of semi-transparency. The use of super-Gaussians allows for a versatile representation of subjects moving through different depths of field.

Limitations

Despite its success, the system faces challenges during "extreme coiling," where the worm's topology becomes unresolvable. Additionally, accuracy at the extreme tips of the head and tail is slightly lower (by about one worm-radius) due to increased transparency at these points.

Future Work

Future iterations could incorporate long-term temporal dynamics to better predict movement and resolve ambiguous self-occlusions. Furthermore, the "render-to-compare" framework is generalizable and could be applied to other semi-transparent or poorly-defined 3D subjects in microscopic imaging.

Summary updated

BibTeX
@inproceedings{ilett2023d,
  author = {Thomas P. Ilett and Omer Yuval and Thomas Ranner and Netta Cohen and David C. Hogg},
  title = {{3D Shape Reconstruction of Semi-Transparent Worms}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01209},
}
Figure