Introduction
A fundamental challenge in computer vision is image segmentation: how a visual system divides a scene into distinct objects based on color. While achieving full "color constancy" (recognizing the exact same color under varying lighting conditions) is the ultimate goal, accurate segmentation is a necessary first step.
The primary difficulty is that the light hitting a sensor—image irradiance—is a complex mixture of surface reflectance and illumination. A naive color-change detector might easily mistake a shadow or a specular highlight for a completely new object. To address this, researchers Anya Hurlbert and Tomaso Poggio proposed a parallel "averaging network" that segments images by finding material boundaries and "spreading" uniform colors within them, effectively ignoring illumination changes and mimicking human visual perception.
The Averaging Network
The paper introduces a simple, iterative network designed for parallel processors. Its core mechanisms include:
- Color Labels (Chromaticity): Instead of relying on raw RGB values, the system uses chromaticity coordinates (hues $u$ and $v$), defined as $u = R/(R+G+B)$ and $v = G/(R+G+B)$. Under a single light source, these values remain relatively constant across a surface regardless of shading, changing primarily at actual material boundaries.
- Edge Integration: The network uses "edge maps"—derived from luminance or color changes—as barriers to prevent color from bleeding across object boundaries.
- Iterative Scheme: In each step, a pixel’s value is replaced by the average of its neighbors. A neighbor only contributes if it is immediately adjacent, not separated by an edge, and within a certain value range (to filter out extreme noise).
- Mathematical Foundation: This iterative process acts as a form of non-linear diffusion. It minimizes an energy function via gradient descent, smoothing out noise within regions while preserving sharp discontinuities at the boundaries.
Datasets and Evaluation
Published in 1989, the study evaluated the algorithm on synthetic images and psychophysical stimuli rather than modern large-scale datasets.
A primary test case was the Mondrian-textured sphere—a synthetic sphere covered in random color patches, rendered with realistic shading and specular highlights. The network was also tested on the Koffka Ring, a classic visual illusion consisting of a uniform grey ring on a split black-and-white background.
Key Insights and Results
The network demonstrated several critical insights into both artificial and biological vision:
- Segmentation vs. Constancy: The system proves that you do not need to know the exact color of an object to know where it ends. Segmentation only requires finding discontinuities in reflectance.
- Robustness to Shading and Specularities: The algorithm successfully segmented the shaded spheres. It treated the entire surface of a single-colored patch as one region despite significant brightness changes across the curve. While specular highlights (shiny spots) can distort color, their effects are minimized in $u, v$ space and effectively "averaged out" by the network.
- Biological Mimicry: The network's behavior mirrors "filling-in" phenomena in human vision. By requiring local smoothness within boundaries, the network naturally produces piecewise constant regions.
- Illusion Replication: Remarkably, the network successfully replicated the Koffka Ring illusion. When the background boundary was continuous across the ring, the network "split" the uniform grey ring into two different shades (light and dark grey), perfectly matching human perception. This suggests the model captures a fundamental mechanism of biological image processing.
Limitations and Future Work
Despite its successes, the model has limitations. While it handles minor highlights, very strong specularities can still create "false" edges that disrupt the averaging process, potentially requiring higher-level knowledge to resolve. Additionally, the current model assumes a single light source; multiple light sources with different colors would complicate the chromaticity constancy. Finally, the segmentation quality is heavily dependent on the initial edge detection—if a boundary edge is missed, colors will inevitably "leak" into adjacent regions.
Summary updated