TL;DR
This paper introduces a novel deep learning approach to correct distortions and instabilities in endoscopic Optical Coherence Tomography (OCT) videos. By employing a dual-branch CNN architecture, the method effectively compensates for rotational artifacts, leading to significantly improved image quality for real-time diagnosis and minimally invasive procedures.
What Problem Does This Paper Address?
Endoscopic OCT is a powerful tool for visualizing internal tissues with high resolution. However, its application, particularly with proximal scanning systems, is hampered by significant image distortions and instabilities. These artifacts arise from irregularities in motor rotation, friction within the catheter, and synchronization issues. These distortions, often termed Non-Uniform Rotational Distortion (NURD), manifest as "stretch-shrink" effects within individual scans (B-scans) and "shaking and drift" between scans. Such artifacts degrade image quality, making accurate diagnosis and real-time guidance challenging. Existing computational methods struggle with accumulating errors, are not always applicable to all distortion types, or require invasive markers.
Methods
The authors propose a two-branch deep learning architecture for online correction:
- Branch A (A-line Level Shifting Estimation): This branch estimates the non-uniform warping vector between consecutive frames. It utilizes a correlation matrix computed from image patches of the current and previous frames. A Convolutional Neural Network (CNN) then processes this matrix to identify the optimal path, representing the shifting angle of each A-line. To improve robustness, this branch incorporates a continuity loss function during training, encouraging smoother distortion estimates.
- Branch B (Group Rotation Estimation): This branch estimates the overall orientation or group rotation from an image array. It employs a CNN with a contracting and expansion path, designed to capture multi-scale contextual features. This branch is inspired by methods used for stabilizing sports camera videos and incorporates a multi-scale loss function to ensure accuracy at different levels of feature extraction.
These two branches operate in parallel. The outputs are then fused using a PI complementary filter, a technique commonly used to combine easily drifting measurements with stable ones, similar to how gyroscopes and accelerometers are fused in inertial navigation. This fusion helps to mitigate accumulating errors and provides a robust correction.
Datasets
The authors utilized a combination of datasets:
- Training Data:
- Previously published OCT datasets from cardiovascular, respiratory, and digestive systems.
- Self-collected OCT videos from a custom endoscopic OCT system and a rectangular phantom.
- Semi-synthetic OCT videos: These were generated by intentionally adding realistic warping vectors and group rotation values to real OCT images. This approach circumvents the need for manual annotation of distortions, which is extremely difficult for OCT data.
- Evaluation Data:
- Semi-synthetic videos generated with different parameters than the training set.
- In vivo pre-clinical data from swine colon experiments.
- Clinical OCT data from human trials using a tethered capsule endomicroscopy system.
Key Insights and Ideas
- Dual-Branch CNN for Comprehensive Correction: The paper highlights the effectiveness of a dual-branch approach, where one branch handles fine-grained A-line level distortions and the other addresses overall rotational drift. This synergy is crucial for robust correction.
- Semi-Synthetic Data for Training: The innovative use of semi-synthetic data generation, by artificially introducing distortions to real OCT images, is a key enabler. This bypasses the significant challenge of obtaining ground truth annotations for real-world OCT videos.
- PI Complementary Filter for Fusion: The application of a PI complementary filter for fusing the outputs of the two CNN branches is a clever way to combine the strengths of each estimation and mitigate error accumulation.
- Generalization to Unseen Data: The study demonstrates that the trained model generalizes well to unseen in vivo data from different experimental setups and even different modalities of rotational scanning OCT, showcasing its robustness.
Key Results
- Accuracy: The proposed deep learning method significantly outperforms the state-of-the-art GS (Graph Search) based method, achieving estimation errors that are one to two orders of magnitude lower. For example, in digestive tract OCT, the MSE was 9.24±5.60 pixel² compared to 354.8±461.8 pixel² for GS.
- Robustness: The algorithm effectively reduces both short-term and long-term instabilities, as evidenced by significant reductions in Standard Deviation (STD) values.
- Qualitative Improvements: Visual results show a dramatic reduction in artifacts, leading to clearer en-face projections and cross-sectional images. The method preserves tissue characteristics and relative distances while eliminating distortions.
- Processing Speed: The entire algorithm iteration takes approximately 145 ms, making it suitable for near real-time applications.
Limitations and Future Work
- Reference Frame Dependency: The proposed method requires a reference frame for drift and error accumulation correction. While a small period at the beginning of a scan is usually sufficient, this might be a limitation for certain pullback scanning scenarios where the sheath appearance changes significantly.
- Computational Cost: While the processing time is promising, it might not be fast enough for very high frame rate OCT systems (e.g., 60 FPS) due to hardware limitations and large input sizes.
- Pathological Cases: The assumption that appearance changes due to artifacts are faster than tissue changes might be affected in some pathological cases with very rapid tissue appearance changes.
- Future Work: The authors suggest exploring other probabilistic fusion filters or optimizing PI complementary filter parameters for further improvement. They also plan to work on algorithm optimization and testing on more recent hardware to achieve higher update rates (10-20 FPS) for online diagnosis.
Summary updated