Scale Invariant Auto-context for Object Segmentation and Labeling

Hongwei Ji, Jiangping He, Xin YangKSII 2014 · DOI

Abstract

In complex visual environments, contextual information is vital for accurate image segmentation and labeling. While the auto-context (AC) algorithm has proven effective by iteratively refining probability maps, its reliance on fixed spatial sampling radii makes it sensitive to significant scale variations in objects. This paper presents the Scale Invariant Auto-context (SIAC) algorithm, which achieves scale invariance by iteratively estimating the optimal scale of an image and adapting the context sampling radius accordingly. Experimental results on horse segmentation and human body labeling demonstrate that SIAC consistently outperforms standard auto-context when objects vary in size.

Introduction: The Context Challenge

Contextual cues—such as the spatial relationship between a horse's head and its torso—are essential for robust computer vision models. The standard Auto-context (AC) algorithm captures these cues using a sequence of classifiers where each learns from both the original image features and the probability maps generated by the preceding step.

However, AC suffers from a "ruler problem." It samples context at fixed distances (radii) from a target pixel. If an object appears at different scales across images, these fixed radii will land on different anatomical parts, leading to feature inconsistency. For instance, a fixed radius might capture the saddle on a small horse but only the neck on a larger one, confusing the classifier.

Methodology: Scale Invariant Auto-context (SIAC)

To overcome scale sensitivity, the SIAC algorithm introduces an iterative feedback loop that makes the model "scale-aware."

1. Initial Classification and Scale Estimation

The process begins with a standard appearance-based classifier to generate a rough probability map. From this map, the algorithm estimates the image scale by counting foreground pixels ($N_F$). By comparing this count to a learned average foreground size ($\theta$), the image is categorized into discrete scale buckets: "small," "medium," or "large."

2. Adaptive Sampling

Once the scale is estimated, the algorithm selects a corresponding radius sequence for context sampling.

  • Small Scale: Sampling distances are halved to maintain relative spatial context.
  • Large Scale: Sampling distances are doubled to capture the full extent of the larger object.
Adaptive sampling points applied to objects at different scales. The algorithm adjusts the sampling grid (magenta stars) to ensure features remain consistent regardless of the object's size in the frame.

3. Iterative Refinement

A new classifier is then trained or applied using these scale-corrected features. This process repeats, with the classification map and the estimated scale refining each other until convergence. This approach effectively allows the model to "zoom" its contextual field of view to match the object's physical size in the image.

Experimental Results

The researchers evaluated SIAC on two challenging datasets:

  • Weizmann Horse Dataset: Using a version of the dataset manually scaled to create high size variety, SIAC achieved a higher F-measure than standard AC. Qualitatively, SIAC produced cleaner and more complete object masks, avoiding the fragmentation common in fixed-scale models.
  • Baseball Player Dataset: In a complex 14-part labeling task (head, torso, limbs, etc.), SIAC reached a pixel-wise accuracy of 78.9%, significantly higher than the 75.2% achieved by standard AC.

The algorithm typically converges within 3–5 iterations, proving that the self-correcting scale estimation is both stable and efficient.

Conclusion and Future Work

SIAC addresses a fundamental weakness in context-based segmentation by making the spatial sampling adaptive. By reducing intra-class variation caused by scale, the learning task becomes simpler and more accurate. Future research aims to improve the coarse scale estimation (currently based on simple pixel counts) to handle occlusions and multiple objects, and to incorporate orientation invariance to handle rotated objects.

Summary updated

Abstract

In complicated environment, context information plays an important role in image segmentation/labeling. The recently proposed auto-context algorithm is one of the effective context-based methods. However, the standard auto-context approach samples the context locations utilizing a fixed radius sequence, which is sensitive to large scale-change of objects. In this paper, we present a scale invariant auto-context (SIAC) algorithm which is an improved version of the auto-context algorithm. In order to achieve scale-invariance, we try to approximate the optimal scale for the image in an iterative way and adopt the corresponding optimal radius sequence for context location sampling, both in training and testing. In each iteration of the proposed SIAC algorithm, we use the current classification map to estimate the image scale, and the corresponding radius sequence is then used for choosing context locations. The algorithm iteratively updates the classification maps, as well as the image scales, until convergence. We demonstrate the SIAC algorithm on several image segmentation/labeling tasks. The results demonstrate improvement over the standard auto-context algorithm when large scale-change of objects exists.
BibTeX
@inproceedings{ji2014scale,
  author = {Hongwei Ji and Jiangping He and Xin Yang},
  title = {{Scale Invariant Auto-context for Object Segmentation and Labeling}},
  booktitle = {KSII},
  year = {2014},
  doi = {10.3837/tiis.2014.08.018},
}
Figure