Consistent Scale Normalization for Object Recognition

Zewen He, He Huang, Yudong Wu, Guan Huang, Wensheng ZhangarXiv 2019 · arXiv

Overview

Instance Scale Normalization (ISN) is a training and testing paradigm designed to mitigate the impact of scale variation in computer vision tasks. By restricting the neural network's focus to objects within a consistent, moderate scale range across all image resolutions, ISN enhances performance in object detection, instance segmentation, and pose estimation without necessitating complex architectural modifications.

The Challenge of Scale Variation

Deep learning models often struggle to recognize objects at extreme sizes—ranging from tiny distant birds to large close-up faces. While Feature Pyramid Networks (FPN) and multi-scale training (image pyramids) are common solutions, they often introduce inconsistencies.

Previous methods determined training eligibility based on an object's size in the original image. This creates a paradox: two objects that appear identical in size to the network (due to different image resolutions) might be treated differently—one being trained upon while the other is ignored. This inconsistency complicates the optimization process.

Instance Scale Normalization (ISN)

The proposed ISN method compresses the scale space of objects into a consistent "ISN range" $[s_l, s_u]$ during both training and inference.

Figure 4: Visualization of scale normalization logic. (a, c) show original images with all objects. (b, d) show the selection process where only objects within the valid scale range are trained (green) or ignored (red). (e, f) show the distribution of trained vs. ignored object scales.

Core Components

  1. Consistent Selection: ISN evaluates an object's scale at the current resolution. If an image is upsampled, a previously "tiny" object may enter the moderate ISN range and become a valid training sample.
  2. Training Protocol: Gradients are only backpropagated for objects falling within the ISN range at a specific resolution, ensuring the network learns from scales it is optimized to handle.
  3. Testing Strategy: Predictions are generated at multiple resolutions, but only those within the ISN range are retained and subsequently fused using Non-Maximum Suppression (NMS).
  4. FPN Integration: ISN complements FPNs by optimizing the overall range of objects the pyramid processes, ensuring consistency across layers.

Key Insights

The effectiveness of ISN stems from forcing the network to operate within its "natural" scale range—the "sweet spot" where receptive fields and semantic features are most effective. By normalizing scale at the input level, ISN eliminates the training contradictions found in earlier approaches.

Figure 2: Example of scale-based object selection in a soccer scene. Green boxes indicate objects within the valid scale range for training at this resolution, while red boxes are ignored to maintain consistency.

Furthermore, ISN simplifies hyperparameter tuning. Instead of defining valid ranges for every level of an image pyramid, users only need to tune the lower ($s_l$) and upper ($s_u$) bounds of the ISN range.

Results and Performance

Evaluated on the COCO dataset, ISN demonstrated significant gains:

  • State-of-the-Art: A ResNet-101 model achieved 46.5 mAP on COCO test-dev.
  • Detection & Pose: Faster R-CNN (ResNet-50) improved from 38.0 to 42.9 mAP. Keypoint detection saw a +3.6% AP boost.
  • Efficiency for Small Models: Lightweight backbones like ResNet-18 saw a 5-point mAP increase, making them more viable for resource-constrained applications.

Limitations and Future Work

Despite its performance, ISN's reliance on image pyramids during testing increases computational cost and reduces inference speed compared to single-shot detectors. Additionally, the optimal ISN range was determined via a greedy search; future work may explore automated or dynamic range selection and more efficient multi-scale inference techniques.

Summary updated

Abstract

Scale variation remains a challenging problem for object detection. Common paradigms usually adopt multiscale training & testing (image pyramid) or FPN (feature pyramid network) to process objects in a wide scale range. However, multi-scale methods aggravate more variations of scale that even deep convolution neural networks with FPN cannot handle well. In this work, we propose an innovative paradigm called Instance Scale Normalization (ISN) to resolve the above problem. ISN compresses the scale space of objects into a consistent range (ISN range), in both training and testing phases. This reassures the problem of scale variation fundamentally and reduces the difficulty of network optimization. Experiments show that ISN surpasses multi-scale counterpart significantly for object detection, instance segmentation, and multi-task human pose estimation, on several architectures. On COCO test-dev, our single model based on ISN achieves 46.5 mAP with a ResNet-101 backbone, which is among the state-of-the-art (SOTA) candidates for object detection.
BibTeX
@misc{he2019consistent,
  author = {Zewen He and He Huang and Yudong Wu and Guan Huang and Wensheng Zhang},
  title = {{Consistent Scale Normalization for Object Recognition}},
  year = {2019},
  eprint = {1908.07323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.07323},
}
Figure