Colored Transparent Object Matting from a Single Image Using Deep Learning

Jamal Ahmed Rahim, Kwan-Yee Kenneth WongarXiv 2019 · OpenReview

TL;DR

This paper introduces CTOM-Net, a novel deep learning framework that can generate high-quality mattes for colored transparent objects from a single image. By extending previous methods that only handled colorless objects, CTOM-Net significantly broadens the applicability of transparent object matting for real-world scenarios.

What Problem Does This Paper Address?

Image matting is the process of extracting a foreground object's opacity (matte) from an image, allowing it to be composited onto new backgrounds. While effective for opaque objects, transparent objects pose a significant challenge because their appearance is influenced by both their own properties and the background they refract. Existing methods for transparent object matting often require multiple images or complex setups, making them impractical for many applications. Even recent deep learning approaches like TOM-Net, which can mat transparent objects from a single image, are limited to colorless objects, failing to capture the intrinsic color of many real-world transparent items like colored glass or plastic. This paper aims to bridge this gap by enabling single-image matting for colored transparent objects.

Methods

The authors propose a two-stage deep learning framework called CTOM-Net, building upon the TOM-Net architecture.

  • Stage 1: CoarseNet: This encoder-decoder network takes a single input image and simultaneously predicts three outputs:
  • An object mask: Delineating the transparent object.
  • A color filter: Modeling the intrinsic color of the transparent object.
  • A refractive flow field: Capturing how light is distorted by the object. The CoarseNet uses a shared encoder and three parallel decoders, trained with a combined loss function that includes segmentation loss for the mask, regression losses for the color filter and flow field, and an image reconstruction loss to ensure the composited image resembles the input.

  • Stage 2: RefineNet: This residual network takes the input image and the outputs from CoarseNet to refine the color filter and refractive flow field, producing sharper and more detailed results. The object mask is not refined as CoarseNet's prediction is deemed sufficiently robust.

The key novelty lies in extending the image formation model to account for the object's intrinsic color. Instead of just an attenuation map (as in TOM-Net), CTOM-Net estimates a "color filter" that, combined with the refractive flow, models how the background light is altered by the colored transparent object.

Datasets

The authors introduce a new, large-scale synthetic dataset for training. This dataset was generated using POV-Ray, incorporating shapes from a previous work and backgrounds from the Microsoft COCO dataset. It includes various transparent object types (glass, glass with water, lens) rendered with both colored and colorless properties.

Additionally, they captured a real dataset of colored transparent objects (glass, glass with water, lens) to evaluate the model's performance on real-world imagery. They also leverage a publicly available real dataset for colorless transparent objects.

Key Insights and Ideas

  • Modeling intrinsic color is crucial: The paper highlights that simply estimating opacity isn't enough for transparent objects; their inherent color significantly impacts their appearance and how they interact with the background.
  • A unified framework for mask, color, and flow: CTOM-Net's strength lies in simultaneously predicting these three components, allowing them to inform each other during the learning process.
  • Bridging the synthetic-to-real gap: The authors acknowledge the challenge of domain shift and employ data augmentation techniques, including smoothing object borders and adjusting color filters in synthetic data, to improve generalization to real images.

Key Results

  • Improved performance on synthetic data: CTOM-Net outperforms baseline methods (including TOM-Net and background images) across various metrics, including refractive flow field estimation (F-EPE, F-Rol), object mask accuracy (M-IoU), and color filter accuracy (C-MSE).
  • Effective generalization to real data: The model demonstrates strong performance on real-world images, achieving higher PSNR and SSIM scores compared to using just the background image.
  • Qualitative improvements: Visual examples show CTOM-Net producing more accurate and visually pleasing composites, particularly for colored transparent objects, where TOM-Net struggles. It can better capture the refractive flow within objects like water in a glass.

Limitations and Future Work

  • Specular highlights: The current method assumes the background is the sole light source and cannot handle transparent objects with specular highlights.
  • Complex objects: The authors omit very complex objects from their datasets due to the difficulty in data generation and augmentation.

Future work will focus on developing better training datasets and network architectures to address these limitations, particularly for objects with specular reflections.

Summary updated

Abstract

This paper proposes a deep learning based method for colored transparent object matting from a single image. Existing approaches for transparent object matting often require multiple images and long processing times, which greatly hinder their applications on real-world transparent objects. The recently proposed TOM-Net can produce a matte for a colorless transparent object from a single image in a single fast feed-forward pass. In this paper, we extend TOM-Net to handle colored transparent object by modeling the intrinsic color of a transparent object with a color filter. We formulate the problem of colored transparent object matting as simultaneously estimating an object mask, a color filter, and a refractive flow field from a single image, and present a deep learning framework for learning this task. We create a large-scale synthetic dataset for training our network. We also capture a real dataset for evaluation. Experiments on both synthetic and real datasets show promising results, which demonstrate the effectiveness of our method.
BibTeX
@misc{rahim2019colored,
  author = {Jamal Ahmed Rahim and Kwan-Yee Kenneth Wong},
  title = {{Colored Transparent Object Matting from a Single Image Using Deep Learning}},
  year = {2019},
  eprint = {1910.02222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.02222},
}
Figure