Semantic Pixel Distances for Image Editing

Josh Myers-Dean, Scott WehrweinWorkshops @ CVPR 2020 · DOI

Abstract

Traditional image editing algorithms often rely on low-level color gradients, which frequently fail to capture the semantic importance of objects within a scene. This paper introduces a method to incorporate high-level semantic features from pre-trained Convolutional Neural Networks (CNNs) into standard editing workflows. By defining a "semantic gradient" based on feature embeddings, the authors demonstrate improved performance in content-aware resizing and range masking, better preserving the structural integrity of complex subjects.

Introduction

Most classic image editing tools are "content-blind." Algorithms like seam carving, used for resizing images by removing low-energy pixel paths, typically calculate energy based on simple color or luminance gradients. However, these low-level metrics do not always align with human perception. For instance, a statue might share similar colors with its background, leading standard algorithms to cut through critical structural elements, such as the statue's leg.

Comparison of seam carving methods: (a) Original, (b) Standard RGB-based carving showing distortion in the statue's leg, (c) Semantic carving preserving the subject.

Methodology

The core of the proposed approach is the "Combined Energy Map." Instead of relying solely on RGB values, the authors extract feature embeddings for every pixel using the penultimate layer of a ResNet-101 model pre-trained on the ADE20k dataset.

Semantic Gradients

The authors define a new energy function as a weighted sum of the standard RGB gradient and a "semantic gradient"—the rate of change within the high-dimensional CNN feature space. This penalizes seams that cross boundaries where the semantic meaning of pixels changes, even if the color remains consistent. This ensures that boundaries between distinct objects (e.g., a dog and grass) are treated as high-energy regions.

Visualizing the energy map: (a) Original image, (b) Combined energy map, (c) Resizing via RGB energy, (d) Resizing via combined semantic energy.

Dimensionality Reduction

To maintain interactive performance, the 512-dimensional CNN features are compressed to 128 dimensions using random projection. This reduction significantly boosts computational speed with negligible impact on output quality, making the tool viable for real-time applications.

Key Applications and Results

The utility of semantic distances was evaluated across two primary tasks:

  • Semantic Seam Carving: Qualitative results show superior preservation of people, animals, and statues during aggressive resizing. In a user study involving 20 images and 10 participants, the semantic method was preferred over traditional RGB carving by a wide margin: 145 votes to 69.
  • Semantic Range Masking: By combining luminance and semantic distances, selection tools become more context-aware. For example, a user can select a sky pixel, and the mask will intelligently include other sky regions while excluding foreground objects (like a white shirt) that share similar luminance values.

Conclusion and Future Work

While off-the-shelf CNN features provide a robust baseline for semantic editing, they are not specifically optimized for pixel-wise distance metrics. The authors suggest that the next logical step is training purpose-built embeddings using metric-learning techniques (such as triplet loss) to further align computational "distance" with human-perceived object boundaries. Moving toward lower-dimensional, more efficient embeddings would also make these techniques more viable for mobile editing platforms.

Summary updated

BibTeX
@inproceedings{myersdean2020semantic,
  author = {Josh Myers-Dean and Scott Wehrwein},
  title = {{Semantic Pixel Distances for Image Editing}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00275},
}
Figure