Fine-Grained Dynamic Head for Object Detection

Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Hongbin Sun, Jian Sun, Nanning Zheng

Standard object detectors treat every part of an object the same way, often ignoring that a person’s hand might need a different resolution than their torso. To address this, researchers have introduced a Fine-Grained Dynamic Head that uses pixel-level dynamic routing to select the best feature scales for different sub-regions of an object. This approach boosts accuracy while slashing computational costs through spatially sparse convolutions.

The Problem: Coarse-Grained Scaling and Computational Waste

Most modern object detectors rely on Feature Pyramid Networks (FPNs) to handle objects of different sizes. Typically, an FPN assigns an entire object (an "instance") to a single level of the pyramid based on its overall bounding box size.

The authors identify two major flaws in this conventional approach:

  1. Scale Inflexibility: Different parts of an object have different characteristic scales. For example, in a photo of a skier, the skier's body might be best represented at a low resolution, but their skis or face might require high-resolution features for accurate detection.
  2. Computational Waste: Standard detection "heads" (the sub-networks that predict boxes and classes) perform dense convolutions over the entire image. This means the model spends just as much energy processing "easy" background pixels as it does "hard" foreground objects.
Comparison of conventional instance-level routing versus fine-grained pixel-level routing.

The Solution: Fine-Grained Dynamic Routing

To overcome these limitations, the authors propose replacing the static, one-size-fits-all detection head with a Fine-Grained Dynamic Head. Its architecture is built on three core pillars:

1. Pixel-Level Dynamic Routing

Instead of assigning an entire object to one scale, the model uses "routers" at every pixel. For each pixel, the router chooses between three paths: staying at the current scale, scaling up, or scaling down. This allows the network to "stitch together" a custom feature representation for every object, capturing small, representative details within large objects.

Architecture of the Dynamic Head showing routing paths across different scales and depths.

2. Spatial Gates and Sparse Convolutions

To save power, the model uses lightweight spatial gates to decide which pixels actually need processing. If a gate is "off," the model uses Spatially Sparse Convolutions, skipping the calculation entirely for those regions. By using a resource budget loss, the model learns to be "lazy" where it can afford to be—allocating more layers and higher resolutions to actual objects and ignoring the background.

3. A Differentiable Activation Function for Hard Gating

Gating often involves "hard" decisions (0 or 1) that are difficult to train because they aren't differentiable. Many dynamic networks behave differently during training (using soft probabilities) than during inference (using hard gates), which hurts performance. The authors designed a new activation function—a generic variant of the restricted tanh—that allows for a hard zero to disable paths while remaining smooth enough for stable training.

Key Results and Performance

The researchers evaluated their method primarily on the COCO dataset, integrating the proposed head into several state-of-the-art frameworks (FCOS, RetinaNet, ATSS, and EfficientDet). The results show a rare "win-win" in deep learning: higher accuracy and lower latency.

  • Accuracy Boost: When added to the FCOS detector with a ResNet-50 backbone, the Dynamic Head achieved a 2.3% mAP (mean Average Precision) gain over the static baseline.
  • Massive Efficiency Gains: The "FCOS-D8@L" configuration achieved 41.0 mAP using only 203.6G FLOPs, compared to the static baseline's 39.5 mAP at a much heavier 512.5G FLOPs.
  • Real-world Speed: On a Tesla V100 GPU, the "Small" version of their model was significantly faster (35.1ms) than the standard FCOS-D6 baseline (46.8ms) while still being more accurate.
  • Versatility: The method consistently improved every detector it was tested on, including a 2.4% mAP gain for RetinaNet.

Limitations and Future Work

While the paper demonstrates impressive gains, there are a few areas for further exploration. Spatially sparse convolutions require specialized software kernels to achieve actual wall-clock speedups on hardware; they aren't as "plug-and-play" as standard convolutions in some frameworks. Additionally, the resource budget relies on a hyperparameter to balance speed and accuracy, which might require careful tuning for different hardware constraints. Finally, the authors suggest this fine-grained approach could be extended to other dense prediction tasks like semantic segmentation or video analysis, paving the way for more efficient vision models across the board.

Summary updated

Abstract

The Feature Pyramid Network (FPN) presents a remarkable approach to alleviate the scale variance in object representation by performing instance-level assignments. Nevertheless, this strategy ignores the distinct characteristics of different sub-regions in an instance. To this end, we propose a fine-grained dynamic head to conditionally select a pixel-level combination of FPN features from different scales for each instance, which further releases the ability of multi-scale feature representation. Moreover, we design a spatial gate with the new activation function to reduce computational complexity dramatically through spatially sparse convolutions. Extensive experiments demonstrate the effectiveness and efficiency of the proposed method on several state-of-the-art detection benchmarks. Code is available at https://github.com/StevenGrove/DynamicHead.
BibTeX
@inproceedings{song2020fine,
  author = {Lin Song and Yanwei Li and Zhengkai Jiang and Zeming Li and Hongbin Sun and Jian Sun and Nanning Zheng},
  title = {{Fine-Grained Dynamic Head for Object Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=9v_1piEk6z7},
}
Figure