Overview
Instance Scale Normalization (ISN) is a training and testing paradigm designed to mitigate the impact of scale variation in computer vision tasks. By restricting the neural network's focus to objects within a consistent, moderate scale range across all image resolutions, ISN enhances performance in object detection, instance segmentation, and pose estimation without necessitating complex architectural modifications.
The Challenge of Scale Variation
Deep learning models often struggle to recognize objects at extreme sizes—ranging from tiny distant birds to large close-up faces. While Feature Pyramid Networks (FPN) and multi-scale training (image pyramids) are common solutions, they often introduce inconsistencies.
Previous methods determined training eligibility based on an object's size in the original image. This creates a paradox: two objects that appear identical in size to the network (due to different image resolutions) might be treated differently—one being trained upon while the other is ignored. This inconsistency complicates the optimization process.
Instance Scale Normalization (ISN)
The proposed ISN method compresses the scale space of objects into a consistent "ISN range" $[s_l, s_u]$ during both training and inference.
Core Components
- Consistent Selection: ISN evaluates an object's scale at the current resolution. If an image is upsampled, a previously "tiny" object may enter the moderate ISN range and become a valid training sample.
- Training Protocol: Gradients are only backpropagated for objects falling within the ISN range at a specific resolution, ensuring the network learns from scales it is optimized to handle.
- Testing Strategy: Predictions are generated at multiple resolutions, but only those within the ISN range are retained and subsequently fused using Non-Maximum Suppression (NMS).
- FPN Integration: ISN complements FPNs by optimizing the overall range of objects the pyramid processes, ensuring consistency across layers.
Key Insights
The effectiveness of ISN stems from forcing the network to operate within its "natural" scale range—the "sweet spot" where receptive fields and semantic features are most effective. By normalizing scale at the input level, ISN eliminates the training contradictions found in earlier approaches.
Furthermore, ISN simplifies hyperparameter tuning. Instead of defining valid ranges for every level of an image pyramid, users only need to tune the lower ($s_l$) and upper ($s_u$) bounds of the ISN range.
Results and Performance
Evaluated on the COCO dataset, ISN demonstrated significant gains:
- State-of-the-Art: A ResNet-101 model achieved 46.5 mAP on COCO
test-dev. - Detection & Pose: Faster R-CNN (ResNet-50) improved from 38.0 to 42.9 mAP. Keypoint detection saw a +3.6% AP boost.
- Efficiency for Small Models: Lightweight backbones like ResNet-18 saw a 5-point mAP increase, making them more viable for resource-constrained applications.
Limitations and Future Work
Despite its performance, ISN's reliance on image pyramids during testing increases computational cost and reduces inference speed compared to single-shot detectors. Additionally, the optimal ISN range was determined via a greedy search; future work may explore automated or dynamic range selection and more efficient multi-scale inference techniques.
Summary updated