Introduction
Deep Learning (DL) deployment at the "Consumer Edge"—encompassing smart homes and offices—frequently utilizes model cascades to balance performance and resource constraints. In a typical cascade, a lightweight model resides on a low-power IoT device to process "easy" samples locally. If the local model's confidence is low, the "hard" sample is forwarded to a more powerful model on a nearby edge server. While effective for isolated devices, this paradigm faces significant challenges in multi-tenant environments where dozens of devices share a single edge hub.
The Multi-Tenancy Challenge
Existing cascade methods often rely on fixed confidence thresholds for offloading. In a multi-tenant scenario, this leads to a "binary failure" mode: either the edge server becomes overwhelmed by simultaneous requests, leading to massive latency spikes, or devices are forced to process all samples locally to avoid the queue, resulting in poor accuracy. MultiTASC addresses this by establishing a dynamic feedback loop between the server’s real-time processing capacity and the devices' offloading behavior.
The MultiTASC Architecture
MultiTASC is a novel scheduler that resides on the edge server and orchestrates connected devices through four core mechanisms:
1. Reconfigurable Forwarding
Instead of static thresholds, MultiTASC employs dynamic thresholds based on the Best-versus-Second-Best (BvSB) metric. The scheduler dynamically instructs devices to adjust their offloading criteria based on current server load, ensuring that only the most uncertain samples are forwarded during periods of high congestion.
2. The Capacity Metric
The "brain" of the scheduler is the Capacity Metric, which calculates the maximum number of samples a server can process within a specific Latency Service-Level Objective (SLO). MultiTASC treats this as an unbounded Knapsack problem, utilizing a greedy algorithm to determine optimal batch sizes that maximize throughput without violating the latency constraints.
3. Fractional Updates
To prevent system oscillation—where all devices simultaneously switch behavior and cause volatile load swings—MultiTASC implements fractional updates. Only a small percentage ($P$) of the device population is updated during any single scheduling window, ensuring a smooth transition to the target load state.
4. Heterogeneity-Aware Prioritization
Recognizing that a high-end smartphone (e.g., Samsung S20) has a more capable local model than a low-end IoT sensor, MultiTASC prioritizes offloads from the weakest devices. This strategy ensures that devices with the lowest local accuracy benefit most from the server-side model when resources are scarce.
Experimental Evaluation
The system was evaluated using the ImageNet 1k-class classification task. The device-side utilized MobileNetV2, EfficientNetLite0, and EfficientNetB0, while the server-side employed InceptionV3 and EfficientNetB3. Hardware emulation included a Tesla T4 GPU as the edge hub and a heterogeneous mix of smartphones, including the Sony Xperia C5, Samsung A71, and Samsung S20 FE.
Key Results
- Improved SLO Satisfaction: MultiTASC improved the rate of inferences meeting latency targets by 20-25 percentage points compared to state-of-the-art static cascade methods.
- Robust Scalability: The system successfully managed over 40 simultaneous devices on a single edge server. Unlike static baselines that plateau as queues grow, MultiTASC maintained linear throughput scaling by gracefully throttling offload rates.
- Accuracy Preservation: Even under heavy load, the system maintained accuracy within 2 percentage points of the baseline, demonstrating an effective management of the accuracy-for-latency trade-off.
Conclusion and Future Work
MultiTASC demonstrates that multi-tenancy-aware scheduling is essential for maintaining the viability of model cascades at the edge. By treating server capacity as a dynamic resource and accounting for device heterogeneity, it prevents the catastrophic latency failures common in static systems.
Future research will explore extending MultiTASC to handle mixed-task environments (e.g., simultaneous video and speech analytics), mitigating extreme network jitter, and incorporating energy-aware scheduling to preserve the battery life of mobile IoT devices.
Summary updated