Attention-Aware Laparoscopic Image Desmoking Network with Lightness Embedding and Hybrid Guided Embedding

Ziteng Liu, Jiahua Zhu, Bainan Liu, Hao Liu, Wenpeng Gao, Yili Fu

Abstract

This paper presents a novel method of smoke removal from laparoscopic images. Due to the heterogeneous nature of surgical smoke, a two-stage network is proposed to estimate the smoke distribution and reconstruct a clear, smoke-free surgical scene. The utilization of the lightness channel plays a pivotal role in providing vital information pertaining to smoke density. The reconstruction of smoke-free image is guided by a hybrid embedding, which combines the estimated smoke mask with the initial image. Experimental results demonstrate that the proposed method boasts a Peak Signal to Noise Ratio (PSNR) that is 2.79% higher than state-of-the-art methods, while also exhibiting a remarkable 38.2% reduction in run-time. Overall, the proposed method offers superior performance in terms of both smoke removal quality and computational efficiency.

Introduction: The Challenge of Surgical Smoke

In Minimally Invasive Surgery (MIS), surgeons rely on real-time video feeds from a laparoscope. However, the use of electrocautery tools frequently generates surgical smoke that obscures the operative field. This smoke is not merely a visual nuisance; it can lead to surgical complications and significantly degrade the performance of downstream computer vision tasks such as robotic navigation or 3D tissue reconstruction.

Traditional desmoking methods often rely on the Atmospheric Scattering Model (ASM), which is optimized for outdoor haze. Surgical smoke, however, is heterogeneous and its density does not strictly correlate with scene depth. Furthermore, existing high-performance deep learning models often fail to meet the low-latency requirements necessary for live surgical environments.

The L-SAHGNet Architecture

To address these challenges, we propose L-SAHGNet, an end-to-end two-stage network designed for real-time surgical desmoking. The architecture consists of two primary modules: the Smoke Attention Estimator (SAE) and the Hybrid Guided Embedding (HGE).

The architecture of L-SAHGNet, featuring the Smoke Attention Estimator (SAE) and Hybrid Guided Embedding (HGE) modules.

Smoke Attention Estimator (SAE)

The SAE utilizes a U-Net-like structure with Swin Transformer blocks to predict a "smoke attention map." A key innovation is the Lightness Embedding (LE). By recognizing that smoke significantly impacts the "Lightness" (L) channel in the HSL color space, we treat this channel as a learnable positional embedding. This guides the Transformer to focus on regions with high smoke density. Additionally, we introduce Soft Patch Expanding (SPE), a novel up-sampling layer that preserves global attention information by applying normalization across the entire feature map.

Hybrid Guided Embedding (HGE)

The HGE module reconstructs the smoke-free image by combining the original smoky input with the predicted smoke mask. To ensure numerical stability, we employ Restricted Field Transformation (RFT). Unlike traditional exponential models that can suffer from gradient explosion as transmission approaches zero, RFT uses a tanh activation function to map transformations into a stable, manageable range.

Experimental Evaluation

The model was trained and evaluated using both synthetic and real-world datasets:

  • Synthetic Dataset: Using Blender, we rendered synthetic smoke over 18,140 clear frames from the Cholec80 dataset, resulting in over 54,000 samples for robust training.
  • Real Dataset: To validate real-world efficacy, the model was tested on 158 real smoky images from Cholec80 videos not included in the synthetic training set.

Key Results and Performance

L-SAHGNet demonstrates a significant leap in both image quality and processing speed. On synthetic data, it achieved a PSNR of 33.06, a 2.79% improvement over the previous state-of-the-art, DehazeFormer-s. It also recorded the lowest LPIPS score, indicating superior perceptual similarity to ground-truth images.

Quantitative comparison of L-SAHGNet against state-of-the-art methods on synthetic and real datasets.

Crucially for clinical applications, L-SAHGNet is highly efficient. With a run-time of just 5.67ms per frame, it is 38.2% faster than its closest competitor. This translates to approximately 176.3 frames per second, making it ideal for high-definition, real-time surgical displays. Qualitative analysis on real surgical images further confirms that L-SAHGNet produces more natural results with fewer color distortions compared to traditional methods like the Dark Channel Prior (DCP).

Conclusion

L-SAHGNet provides a robust and efficient solution for surgical smoke removal. By integrating lightness as a prior and utilizing a hybrid embedding strategy, the network achieves state-of-the-art performance while maintaining the high frame rates required for the operating room. Future work will focus on expanding the dataset to a broader range of surgical procedures and conducting clinical trials to evaluate its impact on surgical workflows.

Summary updated

Abstract

This paper presents a novel method of smoke removal from the laparoscopic images. Due to the heterogeneous nature of surgical smoke, a two-stage network is proposed to estimate the smoke distribution and reconstruct a clear, smoke-free surgical scene. The utilization of the lightness channel plays a pivotal role in providing vital information pertaining to smoke density. The reconstruction of smoke-free image is guided by a hybrid embedding, which combines the estimated smoke mask with the initial image. Experimental results demonstrate that the proposed method boasts a Peak Signal to Noise Ratio that is $2.79\%$ higher than the state-of-the-art methods, while also exhibits a remarkable $38.2\%$ reduction in run-time. Overall, the proposed method offers comparable or even superior performance in terms of both smoke removal quality and computational efficiency when compared to existing state-of-the-art methods. This work will be publicly available on http://homepage.hit.edu.cn/wpgao
BibTeX
@misc{liu2024attention,
  author = {Ziteng Liu and Jiahua Zhu and Bainan Liu and Hao Liu and Wenpeng Gao and Yili Fu},
  title = {{Attention-Aware Laparoscopic Image Desmoking Network with Lightness Embedding and Hybrid Guided Embedding}},
  year = {2024},
  eprint = {2404.07556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07556},
}
Figure