Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation

Vibhav Vineet, Carsten Rother, Philip H. S. TorrNeurIPS 2013 · OpenReview

Introduction: Bridging Physics and Semantics

In computer vision, researchers have traditionally treated "low-level" physics and "high-level" semantics as entirely separate problems.

  1. Intrinsic Image Decomposition: This task involves determining what part of a pixel's color originates from the object's actual surface color (reflectance) versus the environmental lighting (shading) and the 3D structure (shape).
  2. Semantic Segmentation: This involves identifying high-level concepts, such as recognizing that a specific group of pixels represents a "wooden table."

This paper argues that these two domains are mutually informative. For instance, if a set of pixels is known to belong to a "table," those pixels likely share the same material and reflectance properties. Conversely, a region with uniform reflectance is highly likely to be a single object. Solving these tasks in isolation leads to cascading errors, whereas solving them jointly allows a model to resolve ambiguities that a single-task approach would miss.

To address this, the authors introduce a unified framework that jointly estimates an image's intrinsic properties alongside high-level semantic information. By utilizing "higher-order priors" to enforce consistency, the model achieves superior results across both computer vision tasks simultaneously.

Joint estimation of intrinsic properties and semantics

A Joint Energy Minimization Framework

The authors propose a joint energy minimization framework that combines two established approaches: the SIRFS (Shape, Illumination, and Reflectance from Shading) model for intrinsic properties, and a CRF (Conditional Random Field) for object and attribute labeling.

The core innovation lies in the introduction of Higher Order Priors:

  • Reflectance-Object Term ($E^{RO}$): This term enforces a logical physical constraint: if a group of pixels (a segment or "clique") is labeled as the same object, the variance in their reflectance values should be low.
  • Reflectance-Attribute Term ($E^{RA}$): Similarly, this term links specific material attributes (such as "shiny" or "metallic") to corresponding reflectance profiles.

Instead of evaluating pixels in isolation, the model uses unsupervised segmentation to group similar pixels into cliques. The higher-order priors operate on these groups, which is computationally efficient and physically intuitive. By knowing what an object is, the mathematical modeling of how light interacts with it becomes significantly more constrained.

Inference via Dual Decomposition

The resulting optimization problem is highly complex because it mixes continuous variables (reflectance, depth, illumination) with discrete variables (object and attribute labels). To solve this, the authors employ Dual Decomposition. They break the massive joint problem into four simpler "slave" sub-problems that are optimized separately. A "master" problem then coordinates these sub-problems to ensure they converge on a consistent global solution.

Experimental Results

The authors evaluated their model on two primary datasets, which they specifically extended with per-pixel attribute labels:

  • aNYU 2: An extended version of the NYU Depth V2 indoor dataset, featuring 15 object classes and 8 attribute classes (e.g., wooden, painted, cloth).
  • aPascal: An extended version of the Pascal VOC dataset, annotated with 7 material attributes (e.g., skin, metal, glass).

The joint approach consistently outperformed state-of-the-art baselines (such as AHCRF and DenseCRF) in both quantitative metrics and qualitative visual outputs.

Qualitative comparisons for object and attribute segmentation
  • Semantic Accuracy: On the aNYU 2 dataset, the joint model achieved a 30.14% Average Intersection over Union (IoU) for object accuracy, a notable improvement over the 28.88% achieved by the standalone AHCRF baseline. Attribute labeling saw a similar jump, reaching 24.175% Average IoU compared to the baseline's 21.9%.
  • Intrinsic Quality: Visually, the generated reflectance maps were cleaner, and the shading maps better captured the interplay of light and geometry compared to traditional models.
  • Denoising via Semantics: A surprising secondary benefit was that the joint model produced much smoother and cleaner depth maps than the raw data from a Kinect sensor. Because the model "knows" the semantics of an object (e.g., a wall should be flat), it naturally smooths out sensor noise.

Limitations and Future Work

While highly effective, the approach has limitations. The authors note that because the sub-problems are solved approximately, they cannot strictly guarantee the mathematical convergence of the dual decomposition approach. Furthermore, the model currently assumes a single illumination model for the entire scene, which may be too simplistic for complex real-world environments with multiple distinct light sources.

For future work, the authors suggest integrating even more specific structural priors—such as "slanted" or "cylindrical" geometric attributes—which could provide even stronger constraints for depth and surface normal estimation.

Summary updated

Abstract

Many methods have been proposed to recover the intrinsic scene properties such as shape, reflectance and illumination from a single image. However, most of these models have been applied on laboratory datasets. In this work we explore the synergy effects between intrinsic scene properties recovered from an image, and the objects and attributes present in the scene. We cast the problem in a joint energy minimization framework; thus our model is able to encode the strong correlations between intrinsic properties (reflectance, shape, illumination), objects (table, tv-monitor), and materials (wooden, plastic) in a given scene. We tested our approach on the NYU and Pascal datasets, and observe both qualitative and quantitative improvements in the overall accuracy.
BibTeX
@inproceedings{vineet2013higher,
  author = {Vibhav Vineet and Carsten Rother and Philip H. S. Torr},
  title = {{Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
  url = {https://openreview.net/forum?id=SkW_DdZdZH},
}
Figure