RA-MOSAIC: Resource Adaptive Edge AI Optimization over Spatially Multiplexed Video Streams

Ila Gokarn, Yigong Hu, Tarek F. Abdelzaher, Archan MisraACM TOMM 2025 · DOI

TL;DR

This paper introduces MOSAIC, a novel system that significantly boosts the throughput of edge AI devices processing multiple video streams by intelligently packing critical regions from different frames into a single "canvas" frame. This approach achieves substantial performance gains with minimal accuracy loss, addressing the growing challenge of high-fidelity perception on resource-constrained edge devices.

What Problem Does This Paper Address?

Modern edge AI applications, such as autonomous driving and urban surveillance, increasingly rely on processing multiple high-resolution video streams concurrently. However, edge devices like NVIDIA Jetson TX2 have limited computational power, making it difficult to maintain both high fidelity (accuracy) and high throughput (frames per second) for complex deep neural network (DNN) models. Traditional solutions often involve using less accurate models, lower resolutions, or more expensive hardware, leading to an unfavorable cost/complexity vs. accuracy tradeoff. This paper tackles the challenge of optimizing edge AI performance for multiple concurrent video streams by proposing a criticality-aware processing approach that selectively focuses computation on important parts of the video frames.

Methods

MOSAIC introduces a "criticality-aware canvas-based processing" system that spatially multiplexes regions of interest (RoIs) from multiple video streams into a single canvas frame for DNN inference. The core innovation lies in the Mosaic Across Scales (MoS) pipeline, which comprises three key stages:

  1. Multi-scale Tiling: Input frames are tiled at various scales to capture RoIs of different sizes. A Min-Cost Min-Set Cover algorithm is used to select an optimal subset of these tiles that accurately represent all critical objects while minimizing wasted space.
  2. Differential Resizing: Selected tiles are then resized using a Min-Max optimization technique. This stage aims to give larger spatial areas to critical regions, especially smaller objects, to preserve their detectability without disproportionately penalizing larger objects.
  3. 2D Inverse Bin Packing: A Differential Evolution Algorithm is employed to pack these differentially resized tiles onto a single canvas frame. This process ensures that all selected tiles fit within the canvas while respecting their spatial constraints and elasticity.

MOSAIC also incorporates a Periodic Stabilization (PS) operation. This phase runs full-frame DNN inference periodically to refresh object detection models, identify stationary objects, and update object trackers. This helps MOSAIC adapt to dynamic scene characteristics and ensures robustness against missed detections during the MoS operation.

Datasets

The paper evaluates MOSAIC using two benchmark datasets:

  • Okutama-Action: Used for drone-based pedestrian detection. This dataset features 4K resolution video sequences.
  • UFPR-ALPR: Used for license plate recognition (LPR). This dataset consists of 1920x1080 resolution video sequences.

The authors consider each video sequence as a distinct camera stream for evaluation.

Key Insights and Ideas

  • Spatial Multiplexing as a Throughput Booster: The core idea of packing critical regions from multiple frames into a single canvas frame is a powerful way to increase edge device throughput without needing more powerful hardware.
  • Criticality-Awareness is Key: Simply downsampling all frames uniformly leads to significant accuracy loss, especially for smaller objects. MOSAIC's approach of identifying and prioritizing critical regions is crucial for maintaining fidelity.
  • Multi-Scale Tiling for Diverse Objects: Objects in video streams vary greatly in size. The MoS pipeline's ability to tile at multiple scales ensures that objects of different sizes are adequately represented.
  • Differential Resizing Balances Accuracy and Size: The intelligent resizing of tiles is critical. It allows smaller objects to occupy a larger relative space on the canvas, improving their detectability, while larger objects are handled without excessive pixelation.
  • Periodic Stabilization for Robustness: The PS operation is essential for adapting to changing environments and ensuring that the system doesn't miss objects that might be stationary or occluded during the faster MoS cycles.

Key Results

MOSAIC demonstrates significant improvements in the throughput-vs.-fidelity tradeoff:

  • Pedestrian Detection (Okutama-Action):
    • With 6 cameras and batch size 4, MOSAIC achieved 4.75x higher throughput (23 FPS per camera, cumulatively 138 FPS) with ≤ 1% accuracy loss compared to a First Come First Serve (FCFS) paradigm.
  • Compared to a uniform packing approach (Uniform-6), MOSAIC-6 achieved significantly higher accuracy (+7.8%) with only a minor reduction in throughput.
  • License Plate Recognition (UFPR-ALPR):
    • MOSAIC achieved a Character Error Rate (CER) of <33% for a 3-camera setup, significantly outperforming uniform resizing which resulted in 100% CER.
  • For a 3-camera system, MOSAIC achieved a throughput of 23 FPS per camera (cumulative 69 FPS).

Limitations and Future Work

  • PS Periodicity Tradeoff: The paper acknowledges that the Periodic Stabilization (PS) operation, while crucial for accuracy, reduces overall achievable throughput. The choice of PS periodicity involves a tradeoff between accuracy and throughput.
  • Canvas Size Optimization: While MOSAIC performs well across different canvas sizes, the paper suggests that a 640x640 canvas size offers the best balance of throughput and accuracy. Larger canvas sizes increase computational load and latency.
  • Future Work: The authors propose extending MOSAIC with temporal packing, where high-priority regions from consecutive frames are extracted and packed onto a single canvas. They also suggest exploring further optimizations for spatiotemporally correlated camera streams and integrating criticality-aware transmission from cameras.

Summary updated

BibTeX
@article{gokarn2025ra,
  author = {Ila Gokarn and Yigong Hu and Tarek F. Abdelzaher and Archan Misra},
  title = {{RA-MOSAIC: Resource Adaptive Edge AI Optimization over Spatially Multiplexed Video Streams}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2025},
  doi = {10.1145/3715133},
}

Loading related content...

Figure