iffDetector: Inference-aware Feature Filtering for Object Detection

Mingyuan Mao, Yuxin Tian, Baochang Zhang, Qixiang Ye, Wanquan Liu, Guodong Guo, David S. DoermannIEEE TNNLS 2020 · DOI · OpenReview

Abstract

Modern object detectors typically follow an open-loop architecture, where features are extracted and processed in a single forward pass. This design lacks a mechanism to refine features based on high-level semantic predictions during inference, making it sensitive to background noise. This paper summarizes the iffDetector, which introduces an Inference-aware Feature Filtering (IFF) module. By treating object detection as a closed-loop control system, the IFF module utilizes feedback from initial predictions to iteratively refine feature maps, suppressing noise and enhancing object-related signals with minimal computational overhead.

Introduction

The advancement of deep convolutional neural networks (CNNs) has propelled object detection to new heights. However, most contemporary detectors are "one-way" systems: they extract features from an image and pass them forward to generate a prediction. If the initial features are noisy or ambiguous—often the case in complex scenes—the final prediction suffers. The iffDetector addresses this by transforming the detection process into a closed-loop system using the Inference-aware Feature Filtering (IFF) module.

Problem Statement: The Open-Loop Bottleneck

Most research in object detection focuses on optimizing network architectures during the training phase. However, once deployed for inference, the process remains static. In challenging environments with complex backgrounds or poor lighting, backbone networks may produce "polluted" feature maps where background regions are falsely activated. Standard models lack a mechanism to perform inference-time feature optimization, which the authors identify as a major bottleneck for accuracy in complex real-world scenarios.

Methodology: Inference-aware Feature Filtering (IFF)

The core innovation of the iffDetector is the IFF module, a "plug-and-play" component that uses a model's own high-level predictions to refine its internal feature maps during inference.

The IFF module transforms a standard detector into a closed-loop system by feeding high-level semantic information back into the feature maps.

Closed-Loop Feedback and Stability

The IFF module takes high-level semantic information (initial object scores) and feeds it back into earlier feature maps. This process uses a lightweight convolutional layer to transform the detection output, which is then added back to the original feature map. Drawing from control theory, the authors implement this as a "negative feedback" mechanism. In engineering, negative feedback stabilizes systems; here, it stabilizes feature learning by pulling "polluted" features back toward an ideal, noise-free state.

The researchers provide a theoretical guarantee for this denoising process using Fourier transform analysis and Lyapunov stability theory. They demonstrate that the IFF module ensures feature map noise is reduced over successive iterations without causing the system to become erratic.

Detailed architectural integration of the IFF module within a detector (e.g., YOLOv2), highlighting the iterative feedback path and parameter sharing.

Efficiency and Versatility

The IFF module is designed to be generic and can be integrated into both anchor-based (e.g., SSD, YOLOv2) and anchor-free (e.g., CenterNet) detectors. Crucially, it maintains high efficiency. For example, an SSD model with IFF adds less than 2% to the parameter count and maintains a high frame rate (44.62 FPS vs. 45.45 FPS for the baseline), making the accuracy boost nearly "free" in terms of computational cost.

Experiments and Results

The iffDetector was evaluated on the PASCAL VOC and MS COCO 2017 datasets, consistently outperforming baseline models.

  • MS COCO: Adding IFF to FreeAnchor (ResNet-101) improved Average Precision (AP) from 47.3% to 47.9%. For CenterNet (Hourglass-104), AP increased from 45.1% to 45.8%.
  • PASCAL VOC: YOLOv2 with IFF achieved a mAP of 60.4%, compared to the 59.3% baseline.
  • Visual Evidence: Heatmap visualizations demonstrate that the IFF module significantly clears up background "false alarms," resulting in more focused and intense activations on actual objects.

Conclusion

The iffDetector introduces a novel perspective to object detection by incorporating feedback mechanisms from control theory. By enabling inference-time feature optimization, the IFF module effectively suppresses background noise and enhances detection accuracy across various architectures. This work suggests that "inference-aware filtering" is a promising direction for improving robustness in a wide range of computer vision tasks.

Summary updated

Abstract

Modern CNN-based object detectors focus on feature configuration during training but often ignore feature optimization during inference. In this paper, we propose a new feature optimization approach to enhance features and suppress background noise in both the training and inference stages. We introduce a generic Inference-aware Feature Filtering (IFF) module that can easily be combined with modern detectors, resulting in our iffDetector. Unlike conventional open-loop feature calculation approaches without feedback, the IFF module performs closed-loop optimization by leveraging high-level semantics to enhance the convolutional features. By applying Fourier transform analysis, we demonstrate that the IFF module acts as a negative feedback that theoretically guarantees the stability of feature learning. IFF can be fused with CNN-based object detectors in a plug-and-play manner with negligible computational cost overhead. Experiments on the PASCAL VOC and MS COCO datasets demonstrate that our iffDetector consistently outperforms state-of-the-art methods by significant margins\footnote{The test code and model are anonymously available in https://github.com/anonymous2020new/iffDetector }.
BibTeX
@article{mao2020iffdetector,
  author = {Mingyuan Mao and Yuxin Tian and Baochang Zhang and Qixiang Ye and Wanquan Liu and Guodong Guo and David S. Doermann},
  title = {{iffDetector: Inference-aware Feature Filtering for Object Detection}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2020},
  doi = {10.1109/tnnls.2021.3081864},
}
Figure