A Highlight Removal Method for Capsule Endoscopy Images

Shaojie Zhang, Yinghui Wang, Peixuan Liu, Jinlong Yang, Tao Yan, Liangyi Huang, Mingfeng Wang, Huang, Liangyi, Wang, Mingfeng, Atadjanov, Ibragim R.arXiv 2024 · DOI · arXiv

Abstract

The images captured by Wireless Capsule Endoscopy (WCE) always exhibit specular reflections, and removing highlights while preserving the color and texture in the region remains a challenge. To address this issue, this paper proposes a highlight removal method for capsule endoscopy images. Firstly, the confidence and feature terms of the highlight region's edges are computed, where confidence is obtained by the ratio of known pixels in the RGB space's R channel to the B channel within a window centered on the highlight region's edge pixel, and feature terms are acquired by multiplying the gradient vector of the highlight region's edge pixel with the iso-intensity line. Subsequently, the confidence and feature terms are assigned different weights and summed to obtain the priority of all highlight region's edge pixels, and the pixel with the highest priority is identified. Then, the variance of the highlight region's edge pixels is used to adjust the size of the sample block window, and the best-matching block is searched in the known region based on the RGB color similarity and distance between the sample block and the window centered on the pixel with the highest priority. Finally, the pixels in the best-matching block are copied to the highest priority highlight removal region to achieve the goal of removing the highlight region. Experimental results demonstrate that the proposed method effectively removes highlights from WCE images, with a lower coefficient of variation in the highlight removal region compared to the Crinimisi algorithm and DeepGin method. Additionally, the color and texture in the highlight removal region are similar to those in the surrounding areas, and the texture is continuous.

Introduction

Wireless Capsule Endoscopy (WCE) has revolutionized the diagnosis of gastrointestinal diseases by providing a non-invasive means to visualize the digestive tract. However, the internal environment of the human body—characterized by wet, smooth mucosal surfaces and a light source positioned extremely close to the tissue—frequently results in intense specular reflections. These bright white highlights obscure critical medical details, such as lesions, polyps, or bleeding, potentially leading to missed diagnoses.

Existing highlight removal techniques generally fall into three categories: physical models based on Bidirectional Reflectance Distribution Functions (BRDF), standard image inpainting algorithms like the Criminisi method, and deep learning-based approaches. Physical models often leave artifacts or "black holes" where reflections were subtracted. Standard inpainting methods, while better at filling gaps, often struggle with the specific color and texture properties of the gut, leading to jagged edges. Deep learning methods, though powerful, are limited by the scarcity of high-quality, labeled medical datasets showing the same tissue both with and without highlights. This paper proposes a "domain-aware" inpainting method that leverages the biological properties of the gastrointestinal tract to guide the restoration process.

Methodology

The proposed method builds upon the exemplar-based Criminisi inpainting algorithm, introducing several key innovations to adapt it for medical imaging. The overall workflow involves identifying highlight regions, calculating the priority of boundary pixels, selecting the best-matching texture blocks, and performing the final fill.

Flowchart of the proposed highlight removal method, illustrating the interaction between priority calculation, best match block selection, and texture information filling.

1. Domain-Aware Priority Calculation

Standard inpainting determines which pixel to fix first by multiplying a "confidence" term (how much surrounding information is known) and a "data" term (the strength of structural edges). The authors introduce two major improvements here:

  • R/B Ratio Confidence: In the gastrointestinal tract, healthy tissue is rich in hemoglobin, making it appear predominantly red. In normal tissue, the Red (R) channel intensity is significantly higher than the Blue (B) channel. Conversely, in a specular highlight, the R and B values converge toward white. The authors use the R/B ratio as a biological heuristic to identify reliable source pixels.
  • Weighted Priority Sum: In the original Criminisi algorithm, if either the confidence or data term is zero, the total priority becomes zero, causing the algorithm to stall. This method utilizes a weighted sum instead, ensuring that pixels with high confidence are still prioritized even if they do not sit on a sharp structural edge.
Geometric representation of the priority calculation at point P on the boundary, showing the normal vector and the iso-intensity line used to calculate feature terms.

2. Adaptive Matching and Texture Filling

Once the highest-priority point $P$ is identified, the algorithm must find the best patch of tissue to copy into the highlight zone. This is achieved through:

  • Dynamic Windowing: Instead of a fixed block size, the algorithm calculates the local variance. Complex areas with high detail use smaller windows (e.g., 5x5) for precision, while simpler, uniform areas use larger windows (e.g., 9x9) to maintain texture smoothness.
  • Distance-Weighted Search: When searching for a matching block, the algorithm incorporates a distance factor. It assumes that tissue physically closer to the highlight is more likely to share the same color and texture characteristics than distant tissue, preventing the "patchwork" artifacts common in global search methods.

Experimental Results

The method was evaluated using the public CVC-ClinicSpec dataset and the Huawei Tai Lai Dataset. Performance was measured quantitatively using the Coefficient of Variation (COV), which assesses the uniformity and naturalness of the repaired region.

In comparative tests against the standard Criminisi algorithm and the deep learning-based DeepGin method, the proposed approach consistently achieved lower COV scores. For instance, in a representative test case, the proposed method reached a COV of 0.0814, significantly outperforming DeepGin (0.0918) and Criminisi (0.1191). Qualitatively, the results showed superior texture continuity and color accuracy, successfully preserving the linear structures of the intestinal walls without the blurry artifacts often seen in neural network outputs.

Conclusion

By integrating biological heuristics—specifically the R/B channel ratio—and adaptive windowing into the inpainting process, this method provides a robust solution for removing specular highlights in WCE images. While the current implementation is computationally intensive, future work will focus on optimizing the search algorithms to enable real-time clinical application, ensuring that gastroenterologists can view clear, unobstructed images during diagnostic procedures.

Summary updated

Abstract

The images captured by Wireless Capsule Endoscopy (WCE) always exhibit specular reflections, and removing highlights while preserving the color and texture in the region remains a challenge. To address this issue, this paper proposes a highlight removal method for capsule endoscopy images. Firstly, the confidence and feature terms of the highlight region's edges are computed, where confidence is obtained by the ratio of known pixels in the RGB space's R channel to the B channel within a window centered on the highlight region's edge pixel, and feature terms are acquired by multiplying the gradient vector of the highlight region's edge pixel with the iso-intensity line. Subsequently, the confidence and feature terms are assigned different weights and summed to obtain the priority of all highlight region's edge pixels, and the pixel with the highest priority is identified. Then, the variance of the highlight region's edge pixels is used to adjust the size of the sample block window, and the best-matching block is searched in the known region based on the RGB color similarity and distance between the sample block and the window centered on the pixel with the highest priority. Finally, the pixels in the best-matching block are copied to the highest priority highlight removal region to achieve the goal of removing the highlight region. Experimental results demonstrate that the proposed method effectively removes highlights from WCE images, with a lower coefficient of variation in the highlight removal region compared to the Crinimisi algorithm and DeepGin method. Additionally, the color and texture in the highlight removal region are similar to those in the surrounding areas, and the texture is continuous.
BibTeX
@misc{zhang2024highlight,
  author = {Shaojie Zhang and Yinghui Wang and Peixuan Liu and Jinlong Yang and Tao Yan and Liangyi Huang and Mingfeng Wang and Huang, Liangyi and Wang, Mingfeng and Atadjanov, Ibragim R.},
  title = {{A Highlight Removal Method for Capsule Endoscopy Images}},
  year = {2024},
  eprint = {2402.07083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.07083},
}
Figure