A Fast Specular Highlight Removal Method for Smooth Liquor Bottle Surface Combined with U2-Net and LaMa Model

Shaojie Guo, Xiaogang Wang, Jiayi Zhou, Zewei Lian2022 · DOI

TL;DR

This paper introduces a novel two-stage approach for efficiently removing specular highlights from smooth liquor bottle surfaces. By combining the highlight detection capabilities of U²-Net with the powerful inpainting abilities of LaMa, the method achieves superior visual quality and quantitative metrics compared to existing techniques, particularly on challenging, small-sample datasets.

What Problem Does This Paper Address?

Specular highlights, those bright reflections seen on smooth surfaces under light, pose a significant challenge in computer vision and image processing. While they can be aesthetically pleasing in some contexts, they often introduce noise and interfere with automated tasks like object detection, tracking, and recognition. Traditional highlight removal methods, often based on two-color reflection models, struggle with semantic ambiguity, failing to distinguish between true highlights and bright surface colors, leading to inaccurate results and slow processing. More recent deep learning approaches, while improving performance, can be inflexible in architecture, difficult to train, and lack applicability to specialized datasets. This paper specifically targets the problem of removing complex, multi-region specular highlights from smooth liquor bottles, a scenario where existing methods fall short due to the need for precise localization and semantically aware removal.

Methods

The proposed method employs a two-stage pipeline:

  1. Specular Highlight Detection (U²-Net): The first stage utilizes a U²-Net network, known for its effectiveness in salient object detection, to identify and segment specular highlight regions on the liquor bottle's surface. This network is trained to accurately extract highlight area features, generating a precise mask map for the highlight regions. The U²-Net's architecture, featuring Residual U-Blocks (RSU), is designed to capture multi-scale features and contextual information, crucial for distinguishing highlights from surface textures.
  2. Highlight Removal (LaMa): In the second stage, the original liquor bottle image and the highlight mask generated by U²-Net are fed into the LaMa (Large Mask Inpainting) network. LaMa, a state-of-the-art image inpainting model, leverages a Fast Fourier Convolution (FFC) module to achieve a wide receptive field, enabling it to understand global image structures. This allows LaMa to effectively "repair" the highlight areas by synthesizing realistic content, treating the highlight as a missing region to be inpainted. The use of a pre-trained BigLaMa model, which has a deeper generator and was trained on a larger dataset, further enhances its inpainting capabilities.

Datasets

The paper utilizes a self-made liquor bottle surface highlight dataset. This dataset consists of approximately 200 real images captured from various angles in the same scene, specifically designed to feature smooth materials with strong reflections and dense highlight areas. The authors note that this dataset is a "small sample dataset" with "highly targeted" characteristics, which poses a challenge for existing methods.

Key Insights and Ideas

  • Synergistic Combination: The core insight is the powerful synergy achieved by combining a dedicated highlight detection network (U²-Net) with a robust image inpainting network (LaMa). This two-stage approach allows for precise localization of highlights before applying sophisticated inpainting.
  • Semantic Understanding for Inpainting: LaMa's strength lies in its ability to "understand" global image structures, which is crucial for treating highlight removal as an inpainting task. This moves beyond simple pixel manipulation to a more semantically aware repair process.
  • Addressing Small-Sample, Targeted Datasets: The proposed method demonstrates effectiveness even on small, highly specific datasets, a common limitation for many deep learning models that require vast amounts of diverse data.
  • U²-Net's Suitability for Saliency: The paper highlights U²-Net's effectiveness in detecting highlights as salient objects, leveraging its RSU blocks for multi-scale feature extraction.

Key Results

  • Visual Comparison: The proposed method visually outperforms traditional methods (Antonio C. et al., Guo et al., Sun et al.) and a deep learning method (Fu et al.) on the self-made liquor bottle dataset. The results show more accurate highlight detection and removal, with better preservation of original image details and less semantic loss.
  • Quantitative Comparison: The method achieves the highest PSNR (29.873) and SSIM (0.966) scores among the compared methods on the self-made dataset, indicating superior image quality after highlight removal. Sun et al.'s method achieved the second-best SSIM but suffered from content blur and semantic loss.
  • Highlight Detection Performance: The U²-Net model achieved excellent performance on the self-made dataset, as evidenced by its PR curve and F-measure, meeting the requirements for subsequent highlight removal.

Limitations and Future Work

  • Dataset Size: While the method performs well on the self-made small sample dataset, the authors acknowledge that a larger, more diverse dataset would further enhance generalization.
  • Computational Cost: Although described as "fast," the paper doesn't provide specific inference times. The combination of two deep learning models might still have computational demands.
  • Future Work: The paper doesn't explicitly detail future work, but the success on this specific challenging task suggests potential applications to other surfaces or object types with similar highlight characteristics. Further exploration of optimizing the U²-Net and LaMa integration could also be a direction.

Summary updated

BibTeX
@misc{guo2022fast,
  author = {Shaojie Guo and Xiaogang Wang and Jiayi Zhou and Zewei Lian},
  title = {{A Fast Specular Highlight Removal Method for Smooth Liquor Bottle Surface Combined with U2-Net and LaMa Model}},
  year = {2022},
  doi = {10.3390/s22249834},
}
Figure