Abstract
With the prevalence of big-data-driven applications, such as face recognition on smartphones and tailored recommendations from Google Ads, we are on the road to a lifestyle with significantly more intelligence than ever before. Various neural network powered models are running at the back end of their intelligence to enable quick responses to users. Supporting those models requires lots of cloud-based computational resources, e.g., CPUs and GPUs. The cloud providers charge their clients by the amount of resources that they occupy. Clients have to balance the budget and quality of experiences (e.g., response time). The budget leans on individual business owners, and the required Quality of Experience (QoE) depends on usage scenarios of different applications. For instance, an autonomous vehicle requires an real-time response, but unlocking your smartphone can tolerate delays. However, cloud providers fail to offer a QoE-based option to their clients. In this paper, we propose DQoES, differentiated quality of experience scheduler for deep learning inferences. DQoES accepts clients’ specifications on targeted QoEs, and dynamically adjusts resources to approach their targets. Through the extensive cloud-based experiments, DQoES demonstrates that it can schedule multiple concurrent jobs with respect to various QoEs and achieve up to 8x times more satisfied models when compared to the existing system.
Introduction: Bridging the Resource-QoE Gap
In the current cloud landscape, providers like AWS or Azure sell resources in "t-shirt sizes" (e.g., 4 vCPUs, 8GB RAM). However, developers don't actually want "CPUs"—they want their models to respond within a specific timeframe. This creates a significant challenge because different applications have vastly different needs. An autonomous vehicle's object detection model must react in milliseconds to ensure safety, while a smartphone's face-unlock feature can tolerate a one-second delay.
Standard cloud orchestrators, such as Docker Swarm or Kubernetes, typically distribute resources evenly or based on static limits. This leads to a "resource-QoE gap": some tasks receive more power than necessary to meet their goals, while others starve and fail their performance targets. DQoES addresses this by shifting the scheduling abstraction from raw hardware units to user satisfaction units.
Methodology: The DQoES Framework
DQoES is a manager-worker framework designed to run atop Docker Swarm. The core innovation lies in its Dynamic Resource Reallocation algorithm and the use of an Adaptive Listener for efficient monitoring.
The system continuously monitors the response times of inference jobs and classifies them into three buckets:
- Class G (Great): Jobs performing significantly better than their target.
- Class S (Satisfied): Jobs hitting their target within a 10% tolerance.
- Class B (Bad): Jobs failing to meet their target.
DQoES "harvests" CPU capacity from Class G containers—slowing them down just enough that they still meet their targets—and reallocates that "slack" to Class B containers. To maintain efficiency, the Adaptive Listener uses an exponential back-off scheme. If the system is stable, it checks in less frequently; if performance fluctuates or new jobs arrive, it ramps up monitoring to react quickly.
Experimental Evaluation
The system was evaluated using a suite of popular pre-trained deep learning models (VGG-16, ResNet-50, Inception V3, Xception, and NASNetMobile) across both PyTorch and TensorFlow frameworks. Experiments were conducted on the NSF Cloudlab using M510 physical nodes.
Key Results
- 8x Satisfaction Boost: In a cluster environment with 40 concurrent models, DQoES successfully met the objectives for 26 models, whereas the default Docker Swarm scheduler only satisfied 3.
- Efficient Vertical Scaling: The system proved it could dynamically adjust "soft limits" on CPU usage at runtime without requiring container restarts or manual intervention.
- Convergence in Chaos: The authors demonstrated that even with "Burst" or "Random" schedules, the DQoES algorithms eventually converge to a stable state where the maximum number of users are satisfied.
Discussion and Future Directions
The fundamental shift introduced by DQoES is moving scheduling toward "user satisfaction units." By identifying models that are "too fast" for their specific business requirements, the system creates a pool of resources to rescue failing tasks.
However, several areas remain for future work:
- Horizontal Scaling: Current efforts focus on vertical scaling (adjusting CPU limits on a single node). Future versions should incorporate horizontal scaling, moving containers between nodes to balance load.
- Fairness vs. Utility: Maximizing the number of satisfied models might "starve" resource-heavy models to satisfy multiple smaller ones. Business logic may require different weighting.
- Multi-Resource Management: Future research will apply these QoE-based logic gates to memory and network bandwidth, which are also critical for deep learning performance.
Summary updated