Soft scissors: an interactive tool for realtime high quality matting

Jue Wang, Maneesh Agrawala, Michael F. CohenACM Trans. Graphics 2007 · DOI

TL;DR

This paper introduces "Soft Scissors," an interactive tool that allows users to quickly and precisely extract foreground objects from images in real-time. By combining an improved offline matting algorithm with an intelligent, dynamically adjusting brush and incremental updates, it achieves high-quality results with unprecedented speed, making complex image matting accessible to a wider audience.

What Problem Does This Paper Address?

Image matting is the process of extracting a foreground object from an image, often dealing with complex boundaries like hair or fur. Traditional matting algorithms require users to provide detailed input, like a "trimap" (marking pixels as foreground, background, or unknown), and then run computationally intensive algorithms. This process is often slow and iterative, requiring users to repeatedly refine the trimap and re-run the algorithm until satisfactory results are achieved. While some methods are fast, they sacrifice quality, and others produce high-quality results but are too slow for interactive use. This leaves a gap for a tool that offers both high quality and real-time interactivity for matting.

Methods

The core of Soft Scissors is an extension of a previously proposed offline matting algorithm. The key innovations for real-time interactivity are:

  • Incremental Matte Estimation: Instead of recomputing the entire matte, the system identifies a small "update region" around the user's brush stroke. This region is determined by a graph-labeling problem that propagates the influence of the new input. Only pixels within this region have their alpha values recomputed, significantly speeding up the process.
  • Incremental Foreground Color Estimation: To ensure the composite updates instantly and accurately, the system also incrementally estimates the foreground colors for pixels within the update region. This uses a separate graph-labeling problem that leverages the newly computed alpha values and ensures color smoothness.
  • Intelligent User Interface (Soft Scissors): The "soft scissor" brush dynamically adjusts its width and boundary conditions. The width adapts to edge complexity, while boundary conditions (whether the brush edge is considered foreground or background) are automatically determined based on local image statistics and Gaussian Mixture Models (GMMs) of foreground and background colors. This allows the brush to intelligently follow complex or reversing boundaries.

Datasets

The paper uses a custom test dataset created by compositing foreground objects (shot against solid backgrounds) onto new, complex backgrounds. Ground-truth mattes were generated using Bayesian matting. This dataset is used to evaluate the performance of Soft Scissors against other matting algorithms.

Key Insights and Ideas

  • The power of incremental updates: By focusing computation on small, dynamically determined "update regions," the system achieves real-time performance without sacrificing quality. This is a crucial insight for interactive graphics applications.
  • Dynamic brush adaptation is key: The "softness" of the scissors, both in width and boundary definition, is not just a user preference but a critical component for robustly handling diverse and complex image boundaries.
  • Combining graph-based methods for speed and quality: The paper effectively leverages graph-labeling (Random Walks) for both matte and color estimation, but crucially applies them incrementally to achieve real-time performance.

Key Results

  • Soft Scissors achieves real-time performance, allowing users to see instant feedback as they paint.
  • The system generates high-quality mattes, comparable to or better than existing offline methods, even for challenging objects like furry animals.
  • In a user study, Soft Scissors was rated significantly faster and more intuitive than other interactive matting tools, with users preferring its real-time feedback.
  • Quantitative results show lower Mean Squared Error (MSE) compared to several other matting algorithms on their test dataset.

Limitations and Future Work

  • User guidance is still required: While interactive, the system still relies on the user to trace the foreground edge.
  • Potential for more automation: The authors suggest that for video matting, a hybrid approach combining automated segmentation with user guidance might be necessary.
  • Formal user study: The paper mentions a preliminary user study and suggests that a more extensive and formal study would be beneficial to draw solid quantitative conclusions about user experience.

Summary updated

BibTeX
@article{wang2007soft,
  author = {Jue Wang and Maneesh Agrawala and Michael F. Cohen},
  title = {{Soft scissors: an interactive tool for realtime high quality matting}},
  journal = {ACM Trans. Graphics},
  year = {2007},
  doi = {10.1145/1276377.1276389},
}
Figure