Abstract
Mobile video object detection faces a significant trade-off between accuracy and latency. While high-performance detectors are often too heavy for mobile hardware, lightweight alternatives frequently sacrifice precision. We present SmartAdapt, a framework that transforms standard object detectors into dynamic "multi-branch" systems. By utilizing a content-aware scheduler to adjust internal parameters in real-time based on video content, SmartAdapt achieves 70% mAP at 20 FPS on NVIDIA Jetson TX2 hardware, significantly outperforming static baselines.
Introduction
The "accuracy-vs-latency" wall remains a primary obstacle in mobile computer vision. Traditional approaches typically deploy static models tuned for average-case scenarios. However, video content is inherently dynamic; a scene with large, slow-moving objects requires fewer resources than one with small, fast-moving targets. SmartAdapt addresses this inefficiency by exposing internal "tuning knobs" and automating their adjustment at runtime.
Methods
Multi-branch Object Detection Framework (MBODF)
SmartAdapt moves away from the "black box" model of object detection. It exposes five critical parameters, or "knobs," that can be adjusted to balance performance:
- Detector Interval ($di$): Frequency of full detection vs. tracking.
- Detector Resolution ($rd$): Input image dimensions.
- Number of Proposals ($nprop$): Quantity of regions inspected by the model.
- Tracker Resolution ($rt$): Input size for the tracking module.
- Confidence Threshold ($ct$): Minimum score for object tracking.
Content-aware Scheduler (CAS)
To manage these knobs, we developed the Content-aware Scheduler (CAS). This lightweight neural network analyzes content features—such as color histograms and intermediate backbone features—to predict the optimal branch for the current scene within a specified latency budget.
Optimal Branch Election (OBE)
Given the thousands of possible parameter combinations, we employ Optimal Branch Election (OBE). This offline process identifies a Pareto-optimal subset of "candidate branches," making real-time selection computationally feasible.
Experimental Setup
Evaluation was conducted using the ILSVRC 2015 VID dataset. We created a specialized snippet-granularity version by clipping videos into 100-frame segments, allowing the scheduler to learn optimal configurations for diverse visual contexts, such as varying motion speeds and background complexity.
Key Results and Insights
SmartAdapt demonstrates that standard detectors like Faster R-CNN are inherently adaptive. By exposing internal parameters, we can achieve performance gains previously thought to require new architectures.
A critical finding is the "Oracle gap." When using an Oracle scheduler with perfect foresight, accuracy improved by up to 8.3% mAP over static baselines. This confirms that content awareness is essential for maximizing performance on constrained hardware.
On the NVIDIA Jetson TX2, SmartAdapt achieved 70% mAP at 20 FPS. Compared to FastAdapt, a leading adaptive method, SmartAdapt provided a 20.9% to 23.6% higher mAP under identical latency constraints. The framework proved versatile, enhancing multiple architectures including EfficientDet, SSD, and YOLOv3.
Limitations and Future Work
While the scheduler is lightweight, feature extraction (e.g., via MobileNetV2) can still consume significant time (~65ms), potentially bottlenecking very tight latency requirements. Additionally, the current scheduler relies on historical frame data; sudden scene cuts can cause brief sub-optimal performance. Future research will explore more efficient feature extractors and tighter integration between tracking and detection at the feature level.
Summary updated