Segmenting a Low-Depth-of-Field Image Using Morphological Filters and Region Merging

Changick KimIEEE TIP 2005 · OpenReview

TL;DR

This paper introduces a novel, parameterless algorithm for automatically segmenting the object of interest (OOI) in low depth-of-field (DOF) images. By analyzing pixel sharpness and color, the method effectively separates the focused OOI from the blurred background, outperforming existing techniques in robustness and accuracy, especially on complex images.

What Problem Does This Paper Address?

In photography, low depth-of-field (DOF) is a common technique to draw attention to a specific subject by rendering it sharp while blurring the background. This creates an implicit segmentation that humans easily perceive. However, automatically segmenting these "objects of interest" (OOIs) from low DOF images is challenging for algorithms. Unlike images with clear edges, the transition between sharp and blurred regions can be gradual, making it difficult to define precise boundaries. This automatic segmentation is crucial for applications like image retrieval, adaptive compression, and content-based image analysis, where understanding the OOI is key.

Methods

The proposed algorithm employs a five-stage process:

  1. Deviation Scoring: This stage identifies sharp pixels by comparing the original image with a blurred version. The difference in edge information between the two provides a score, with higher scores indicating sharper regions. This method leverages the L*a*b* color space for more accurate color difference calculations compared to RGB.
  2. Score Clustering: Pixels with high sharpness scores are clustered using a density-based clustering algorithm (DBSCAN). This helps to distinguish genuine sharp regions of the OOI from isolated noise pixels. DBSCAN is chosen for its ability to handle non-convex shapes.
  3. Mask Approximation: A preliminary binary mask of the OOI is generated by computing the convex hull of the clustered sharp pixels and then filling these hulls. Morphological operations (closing and dilation by reconstruction) are applied to smooth this mask and close small holes.
  4. Color Segmentation: The approximate mask is further refined by grouping pixels within the mask that have similar colors in the original image, using the L*a*b* color space and a color distance threshold.
  5. Region Scoring: Finally, each color region is assigned a relevance score based on its surrounding sharpness scores. Regions with low relevance are removed, iteratively refining the mask to accurately delineate the OOI.

Datasets

The experiments were conducted on a diverse dataset of 65 images downloaded from Flickr and created by the authors. This dataset includes images from various categories with varying degrees of DOF, fuzziness, and background complexity, aiming to avoid overfitting to specific image types.

Key Insights and Ideas

  • Leveraging Perceived Sharpness: The core idea is to mimic human perception by focusing on regions of sharpness, but doing so in a robust, automated way that goes beyond simple edge detection.
  • L*a*b* Color Space for Robustness: Using the L*a*b* color space for color difference calculations provides a more perceptually uniform measure, leading to better results than traditional RGB.
  • Density-Based Clustering for Complex Shapes: DBSCAN's ability to identify clusters of arbitrary shapes is crucial for segmenting OOIs that are not necessarily convex.
  • Parameterless Design: A significant contribution is the algorithm's parameterless nature, with parameters being either image-independent or automatically determined, making it highly adaptable.
  • Impact of DOF on Image Similarity: The paper demonstrates that segmenting OOIs in low DOF images significantly improves image similarity measures, particularly for content-based image retrieval.

Key Results

  • The proposed algorithm achieves superior robustness and accuracy compared to three reference methods ([7], [11], [18]) across a diverse test set.
  • The average spatial distortion error was 0.21, significantly lower than the best competing algorithm's average of 0.51.
  • The algorithm performs well even on images with complex backgrounds and higher resolutions, outperforming competitors that struggle with these conditions.
  • The paper shows that increasing image size generally leads to improved segmentation quality, with the algorithm maintaining accuracy.
  • The algorithm's runtime increases with image size, but it remains competitive, especially for larger images where other methods lose accuracy.

Limitations and Future Work

  • Runtime for Large Images: While accurate, the algorithm's runtime can be substantial for very large images, though it scales better than some competitors.
  • Future Work: The authors plan to further improve processing speed and accuracy, apply the algorithm to video, and develop an automatic detection mechanism for low DOF images. They also intend to release their dataset and code as ImageJ plugins.

Summary updated

Abstract

We propose a novel algorithm to partition an image with low depth-of-field (DOF) into focused object-of-interest (OOI) and defocused background. The proposed algorithm unfolds into three steps. In the first step, we transform the low-DOF image into an appropriate feature space, in which the spatial distribution of the high-frequency components is represented. This is conducted by computing higher order statistics (HOS) for all pixels in the low-DOF image. Next, the obtained feature space, which is called HOS map in this paper, is simplified by removing small dark holes and bright patches using a morphological filter by reconstruction. Finally, the OOI is extracted by applying region merging to the simplified image and by thresholding. Unlike the previous methods that rely on sharp details of OOI only, the proposed algorithm complements the limitation of them by using morphological filters, which also allows perfect preservation of the contour information. Compared with the previous methods, the proposed method yields more accurate segmentation results, supporting faster processing.
BibTeX
@article{kim2005segmenting,
  author = {Changick Kim},
  title = {{Segmenting a Low-Depth-of-Field Image Using Morphological Filters and Region Merging}},
  journal = {IEEE Trans. Image Process.},
  year = {2005},
  url = {https://openreview.net/forum?id=BZ_x2SvPD_r},
}
Figure