DeepScale: Online Frame Size Adaptation for Multi-object Tracking on Smart Cameras and Edge Servers

Keivan Nalaie, Renjie Xu, Rong Zheng

This academic paper presents DeepScale, a framework designed to make real-time multi-object tracking (MOT) feasible on resource-constrained hardware like smart cameras and edge servers. By dynamically adjusting the resolution of video frames based on the complexity of the scene, DeepScale balances the trade-off between tracking accuracy and computational speed.

TL;DR

DeepScale accelerates multi-object tracking by automatically downscaling video frames when a scene is simple and upscaling them only when fine detail is required. It achieves up to a 1.57x speedup with minimal accuracy loss (~2.3%) by using a self-supervised "detectability" branch to pick the optimal resolution on the fly.

What Problem Does This Paper Address?

Multi-object tracking (MOT)—the task of identifying and following multiple moving objects in a video—is a cornerstone of modern surveillance and autonomous driving. However, state-of-the-art MOT systems rely on heavy Convolutional Neural Networks (CNNs) that are computationally expensive.

The central dilemma is resolution:

  1. High resolution is necessary to detect small or distant objects but is slow and power-hungry.
  2. Low resolution is fast but often misses critical details, leading to tracking failures.

Existing solutions like model compression or cloud offloading often ignore the fact that video content changes. An empty hallway doesn't require the same pixel density as a crowded city square. DeepScale addresses this by treating frame resolution as a dynamic "tuning knob" that adapts to the visual content in real-time.

Methods

The authors propose a model-agnostic framework that can be layered on top of existing "one-shot" trackers (trackers that perform detection and re-identification in a single pass).

  • The Detectability Branch: DeepScale adds a small, parallel branch to the tracker's architecture. This branch predicts "detectability scores" for several candidate resolutions. It essentially asks: "If I shrink this frame to 50%, will I still be able to see the objects?"
  • Self-Supervised Learning: A key innovation is how the model is trained. It doesn't require manual labels for "best resolution." Instead, it uses the output of the tracker on a full-resolution frame as the "ground truth" to train the branch to predict how much information is lost at lower resolutions.
  • Online Adaptation: During inference, the system processes a full-resolution frame every $K$ frames to re-evaluate the scene. It then selects the smallest resolution that maintains a user-defined detectability threshold for the subsequent $K-1$ frames.
  • DeepScale++: An enhanced version where the model is trained using multi-resolution data augmentation, making the underlying tracker more robust to the blurriness of low-resolution inputs.
  • Computation Partitioning: The paper explores four ways to split the work between a camera and an edge server, including SOAT (Edge server-only with adaptive transmission), which reduces network congestion by only sending low-res frames when the scene allows.

Datasets

The researchers evaluated DeepScale using several standard MOT benchmarks:

  • MOT15, MOT16, MOT17, and MOT20: These datasets contain diverse indoor and outdoor pedestrian tracking scenarios with varying crowd densities.
  • CrowdHuman: Used for pre-training the models to ensure robust object detection.

Key Insights and Ideas

  • FCNs are Resolution-Flexible: Because Fully Convolutional Networks (FCNs) don't rely on fixed-size input layers, they can process different image sizes without modification. DeepScale exploits this inherent property of modern AI.
  • Temporal Redundancy: Visual complexity doesn't change instantly. By updating the resolution every $K$ frames (e.g., every 5 or 10 frames), the system saves significant overhead without losing track of objects.
  • Shared Features: The detectability branch shares the "backbone" (feature extractor) with the main tracker. This means the logic for picking a resolution adds almost zero extra computational cost.

Key Results

The authors tested DeepScale on an NVIDIA Jetson TX2 (a common edge device) and a GPU server.

  • Speed vs. Accuracy: On the MOT15 dataset, DeepScale++ achieved a 1.57x speedup compared to the state-of-the-art FairMOT tracker, with only a 2.3% drop in accuracy (MOTA score).
  • Superior to Fixed Scaling: DeepScale++ outperformed trackers running at a fixed medium resolution. For example, it achieved higher accuracy and higher frame rates than a tracker locked at a static 704x384 resolution.
  • Efficiency: In MOT17 tests, DeepScale++ reached 20.5 FPS (frames per second), while the baseline FairMOT struggled at 15.4 FPS.

Limitations and Future Work

  • Re-ID Sensitivity: The authors note that changing resolutions can slightly alter the "appearance features" used to identify objects across frames. While they use Intersection over Union (IoU) matching to help, very aggressive downscaling can still lead to "identity switches."
  • Backbone Dependency: The performance gains are somewhat tied to the specific CNN backbone used (like DLA-34). Different architectures might yield different scaling efficiencies.
  • Future Directions: The authors suggest exploring more advanced backbones and extending the framework to multi-camera systems where several cameras share a single edge server's resources.

Summary updated

Abstract

In surveillance and search and rescue applications, it is important to perform multi-target tracking (MOT) in real-time on low-end devices. Today's MOT solutions employ deep neural networks, which tend to have high computation complexity. Recognizing the effects of frame sizes on tracking performance, we propose DeepScale, a model agnostic frame size selection approach that operates on top of existing fully convolutional network-based trackers to accelerate tracking throughput. In the training stage, we incorporate detectability scores into a one-shot tracker architecture so that DeepScale can learn representation estimations for different frame sizes in a self-supervised manner. During inference, it can adapt frame sizes according to the complexity of visual contents based on user-controlled parameters. To leverage computation resources on edge servers, we propose two computation partition schemes tailored for MOT, namely, edge server only with adaptive frame-size transmission and edge server-assisted tracking. Extensive experiments and benchmark tests on MOT datasets demonstrate the effectiveness and flexibility of DeepScale. Compared to a state-of-the-art tracker, DeepScale++, a variant of DeepScale achieves 1.57X accelerated with only moderate degradation ~2.3\ in tracking accuracy on the MOT15 dataset in one configuration. We have implemented and evaluated DeepScale++ and the proposed computation partition schemes on a small-scale testbed consisting of an NVIDIA Jetson TX2 board and a GPU server. The experiments reveal non-trivial trade-offs between tracking performance and latency compared to server-only or smart camera-only solutions.
BibTeX
@misc{nalaie2021deepscale,
  author = {Keivan Nalaie and Renjie Xu and Rong Zheng},
  title = {{DeepScale: Online Frame Size Adaptation for Multi-object Tracking on Smart Cameras and Edge Servers}},
  year = {2021},
  eprint = {2107.10404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.10404},
}
Figure