TL;DR
This paper introduces CTOM-Net, a novel deep learning framework that can generate high-quality mattes for colored transparent objects from a single image. By extending previous methods that only handled colorless objects, CTOM-Net significantly broadens the applicability of transparent object matting for real-world scenarios.
What Problem Does This Paper Address?
Image matting is the process of extracting a foreground object's opacity (matte) from an image, allowing it to be composited onto new backgrounds. While effective for opaque objects, transparent objects pose a significant challenge because their appearance is influenced by both their own properties and the background they refract. Existing methods for transparent object matting often require multiple images or complex setups, making them impractical for many applications. Even recent deep learning approaches like TOM-Net, which can mat transparent objects from a single image, are limited to colorless objects, failing to capture the intrinsic color of many real-world transparent items like colored glass or plastic. This paper aims to bridge this gap by enabling single-image matting for colored transparent objects.
Methods
The authors propose a two-stage deep learning framework called CTOM-Net, building upon the TOM-Net architecture.
- Stage 1: CoarseNet: This encoder-decoder network takes a single input image and simultaneously predicts three outputs:
- An object mask: Delineating the transparent object.
- A color filter: Modeling the intrinsic color of the transparent object.
-
A refractive flow field: Capturing how light is distorted by the object. The CoarseNet uses a shared encoder and three parallel decoders, trained with a combined loss function that includes segmentation loss for the mask, regression losses for the color filter and flow field, and an image reconstruction loss to ensure the composited image resembles the input.
-
Stage 2: RefineNet: This residual network takes the input image and the outputs from CoarseNet to refine the color filter and refractive flow field, producing sharper and more detailed results. The object mask is not refined as CoarseNet's prediction is deemed sufficiently robust.
The key novelty lies in extending the image formation model to account for the object's intrinsic color. Instead of just an attenuation map (as in TOM-Net), CTOM-Net estimates a "color filter" that, combined with the refractive flow, models how the background light is altered by the colored transparent object.
Datasets
The authors introduce a new, large-scale synthetic dataset for training. This dataset was generated using POV-Ray, incorporating shapes from a previous work and backgrounds from the Microsoft COCO dataset. It includes various transparent object types (glass, glass with water, lens) rendered with both colored and colorless properties.
Additionally, they captured a real dataset of colored transparent objects (glass, glass with water, lens) to evaluate the model's performance on real-world imagery. They also leverage a publicly available real dataset for colorless transparent objects.
Key Insights and Ideas
- Modeling intrinsic color is crucial: The paper highlights that simply estimating opacity isn't enough for transparent objects; their inherent color significantly impacts their appearance and how they interact with the background.
- A unified framework for mask, color, and flow: CTOM-Net's strength lies in simultaneously predicting these three components, allowing them to inform each other during the learning process.
- Bridging the synthetic-to-real gap: The authors acknowledge the challenge of domain shift and employ data augmentation techniques, including smoothing object borders and adjusting color filters in synthetic data, to improve generalization to real images.
Key Results
- Improved performance on synthetic data: CTOM-Net outperforms baseline methods (including TOM-Net and background images) across various metrics, including refractive flow field estimation (F-EPE, F-Rol), object mask accuracy (M-IoU), and color filter accuracy (C-MSE).
- Effective generalization to real data: The model demonstrates strong performance on real-world images, achieving higher PSNR and SSIM scores compared to using just the background image.
- Qualitative improvements: Visual examples show CTOM-Net producing more accurate and visually pleasing composites, particularly for colored transparent objects, where TOM-Net struggles. It can better capture the refractive flow within objects like water in a glass.
Limitations and Future Work
- Specular highlights: The current method assumes the background is the sole light source and cannot handle transparent objects with specular highlights.
- Complex objects: The authors omit very complex objects from their datasets due to the difficulty in data generation and augmentation.
Future work will focus on developing better training datasets and network architectures to address these limitations, particularly for objects with specular reflections.
Summary updated