Smartadapt: Multi-branch Object Detection Framework for Videos on Mobiles

Ran Xu, Fangzhou Mu, Jayoung Lee, Preeti Mukherjee, Somali Chaterji, Saurabh Bagchi, Yin Li

Abstract

Mobile video object detection faces a significant trade-off between accuracy and latency. While high-performance detectors are often too heavy for mobile hardware, lightweight alternatives frequently sacrifice precision. We present SmartAdapt, a framework that transforms standard object detectors into dynamic "multi-branch" systems. By utilizing a content-aware scheduler to adjust internal parameters in real-time based on video content, SmartAdapt achieves 70% mAP at 20 FPS on NVIDIA Jetson TX2 hardware, significantly outperforming static baselines.

Introduction

The "accuracy-vs-latency" wall remains a primary obstacle in mobile computer vision. Traditional approaches typically deploy static models tuned for average-case scenarios. However, video content is inherently dynamic; a scene with large, slow-moving objects requires fewer resources than one with small, fast-moving targets. SmartAdapt addresses this inefficiency by exposing internal "tuning knobs" and automating their adjustment at runtime.

Methods

Multi-branch Object Detection Framework (MBODF)

SmartAdapt moves away from the "black box" model of object detection. It exposes five critical parameters, or "knobs," that can be adjusted to balance performance:

  1. Detector Interval ($di$): Frequency of full detection vs. tracking.
  2. Detector Resolution ($rd$): Input image dimensions.
  3. Number of Proposals ($nprop$): Quantity of regions inspected by the model.
  4. Tracker Resolution ($rt$): Input size for the tracking module.
  5. Confidence Threshold ($ct$): Minimum score for object tracking.

Content-aware Scheduler (CAS)

To manage these knobs, we developed the Content-aware Scheduler (CAS). This lightweight neural network analyzes content features—such as color histograms and intermediate backbone features—to predict the optimal branch for the current scene within a specified latency budget.

The SmartAdapt system architecture, featuring the Content-aware Scheduler and the Multi-branch Object Detection Framework.

Optimal Branch Election (OBE)

Given the thousands of possible parameter combinations, we employ Optimal Branch Election (OBE). This offline process identifies a Pareto-optimal subset of "candidate branches," making real-time selection computationally feasible.

Experimental Setup

Evaluation was conducted using the ILSVRC 2015 VID dataset. We created a specialized snippet-granularity version by clipping videos into 100-frame segments, allowing the scheduler to learn optimal configurations for diverse visual contexts, such as varying motion speeds and background complexity.

Key Results and Insights

SmartAdapt demonstrates that standard detectors like Faster R-CNN are inherently adaptive. By exposing internal parameters, we can achieve performance gains previously thought to require new architectures.

A critical finding is the "Oracle gap." When using an Oracle scheduler with perfect foresight, accuracy improved by up to 8.3% mAP over static baselines. This confirms that content awareness is essential for maximizing performance on constrained hardware.

Accuracy vs. Latency comparison between the Oracle scheduler and static baselines across different branch counts.

On the NVIDIA Jetson TX2, SmartAdapt achieved 70% mAP at 20 FPS. Compared to FastAdapt, a leading adaptive method, SmartAdapt provided a 20.9% to 23.6% higher mAP under identical latency constraints. The framework proved versatile, enhancing multiple architectures including EfficientDet, SSD, and YOLOv3.

Limitations and Future Work

While the scheduler is lightweight, feature extraction (e.g., via MobileNetV2) can still consume significant time (~65ms), potentially bottlenecking very tight latency requirements. Additionally, the current scheduler relies on historical frame data; sudden scene cuts can cause brief sub-optimal performance. Future research will explore more efficient feature extractors and tighter integration between tracking and detection at the feature level.

Summary updated

Abstract

Several recent works seek to create lightweight deep net-works for video object detection on mobiles. We observe that many existing detectors, previously deemed computationally costly for mobiles, intrinsically support adaptive inference, and offer a multi-branch object detection frame-work (MBODF). Here, an MBODF is referred to as a so-lution that has many execution branches and one can dy-namically choose from among them at inference time to sat-isfy varying latency requirements (e.g. by varying resolution of an input frame). In this paper, we ask, and answer, the wide-ranging question across all MBODFs: How to expose the right set of execution branches and then how to sched-ule the optimal one at inference time? In addition, we un-cover the importance of making a content-aware decision on which branch to run, as the optimal one is conditioned on the video content. Finally, we explore a content-aware scheduler, an Oracle one, and then a practical one, leveraging various lightweight feature extractors. Our evaluation shows that layered on Faster R-CNN-based MBODF, compared to 7 baselines, our Smartadapt achieves a higher Pareto optimal curve in the accuracy-vs-latency space for the ILSVRC VID dataset.
BibTeX
@inproceedings{xu2022smartadapt,
  author = {Ran Xu and Fangzhou Mu and Jayoung Lee and Preeti Mukherjee and Somali Chaterji and Saurabh Bagchi and Yin Li},
  title = {{Smartadapt: Multi-branch Object Detection Framework for Videos on Mobiles}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00256},
}
Figure