Learning Ordinal Relationships for Mid-Level Vision

Daniel Zoran, Phillip Isola, Dilip Krishnan, William T. FreemanICCV 2015 · OpenReview

Abstract

We propose a framework that infers mid-level visual properties of an image by learning about ordinal relationships. Instead of estimating metric quantities directly, the system proposes pairwise relationship estimates for points in the input image. These sparse probabilistic ordinal measurements are globalized to create a dense output map of continuous metric measurements. Estimating order relationships between pairs of points has several advantages over metric estimation: it solves a simpler problem than metric regression, humans are better at relative judgements, so data collection is easier, ordinal relationships are invariant to monotonic transformations of the data, thereby increasing the robustness of the system and providing qualitatively different information. We demonstrate that this framework works well on two important mid-level vision tasks: intrinsic image decomposition and depth from an RGB image.

Introduction

In computer vision, tasks like estimating depth or separating an image into its reflectance (true color) and shading (lighting) components are typically treated as metric regression problems. This approach attempts to predict absolute values for every pixel—for example, predicting that a specific pixel is exactly 3.42 meters away. However, metric regression is challenging because absolute values are sensitive to lighting and camera settings, and human annotators are far better at making relative judgments (e.g., "point A is closer than point B") than absolute ones.

This paper introduces a paradigm shift: learning ordinal relationships. By focusing on simple "greater than," "less than," or "equal to" comparisons, the system gains robustness and simplifies the learning objective.

The Ordinal Framework

The proposed system operates in three distinct stages to transform an input image into a dense mid-level representation.

The framework extracts sparse ordinal comparisons (closer/farther, darker/lighter) and globalizes them into dense metric maps for depth and reflectance.
  1. Point Pair Selection: The system identifies pairs of points, prioritizing those in homogeneous regions separated by gradients. This strategy targets areas where depth or reflectance changes are most likely to occur.
  2. The Ordinal Network: A deep Convolutional Neural Network (CNN) serves as a comparative judge. It processes local patches around the selected points alongside a global context of the entire image to classify the relationship between the points (e.g., is A darker than B?).
  3. Globalization: To convert sparse pairwise judgments into a continuous map, the authors employ a constrained quadratic optimization. This step "stitches" the relative comparisons together, using superpixel constancy to ensure spatial coherence and natural image statistics.

Experimental Results

The framework was evaluated on two primary tasks: intrinsic image decomposition and depth estimation.

Intrinsic Image Decomposition

Using the Intrinsic Images in the Wild (IIW) dataset, the model achieved a Weighted Human Disagreement Rate (WHDR) of 17.86%. This outperformed the previous state-of-the-art (20.30%), marking a significant improvement in the field's ability to separate reflectance from shading in real-world scenes.

Qualitative comparison of intrinsic image decomposition against baseline methods, showing superior separation of reflectance and shading.

Depth Estimation

When applied to the NYU Depth v2 dataset, the model produced depth maps with sharp edges and accurate global structures. While specialized metric models may achieve lower absolute error, the ordinal approach proved highly competitive on relative distance measures (WKDR) and demonstrated superior generalization to outdoor and novel scenes.

Perceptual Insights

Analysis of the trained models revealed that the networks independently learned common perceptual heuristics. For depth, the network learned that points lower in the vertical field of view are generally closer—a common prior in human perception. For reflectance, it adopted a version of the "Retinex" rule, interpreting small intensity changes as shading and large changes as shifts in the actual color (reflectance) of the object. These findings suggest that ordinal learning effectively captures the "rules of thumb" used in human vision.

Conclusion

By reframing mid-level vision as a series of ordinal comparisons, this work provides a robust and human-centric alternative to metric regression. While the globalization process remains a potential bottleneck, the success in intrinsic image decomposition and depth estimation highlights the power of relative judgments in understanding the 3D world. Future research may focus on multi-task learning, where depth and reflectance estimates inform and refine one another.

Summary updated

Abstract

We propose a framework that infers mid-level visual properties of an image by learning about ordinal relationships. Instead of estimating metric quantities directly, the system proposes pairwise relationship estimates for points in the input image. These sparse probabilistic ordinal measurements are globalized to create a dense output map of continuous metric measurements. Estimating order relationships between pairs of points has several advantages over metric estimation: it solves a simpler problem than metric regression, humans are better at relative judgements, so data collection is easier, ordinal relationships are invariant to monotonic transformations of the data, thereby increasing the robustness of the system and providing qualitatively different information. We demonstrate that this frame-work works well on two important mid-level vision tasks: intrinsic image decomposition and depth from an RGB image. We train two systems with the same architecture on data from these two modalities. We provide an analysis of the resulting models, showing that they learn a number of simple rules to make ordinal decisions. We apply our algorithm to depth estimation, with good results, and intrinsic image decomposition, with state-of-the-art results.
BibTeX
@inproceedings{zoran2015learning,
  author = {Daniel Zoran and Phillip Isola and Dilip Krishnan and William T. Freeman},
  title = {{Learning Ordinal Relationships for Mid-Level Vision}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=B1ND7bMO-B},
}
Figure