TL;DR
Text-to-image (T2I) models are computationally expensive, making high-throughput serving challenging. ARGUS introduces a novel system that intelligently balances image quality and inference speed by dynamically adapting approximation levels for each prompt, achieving significant improvements in throughput and reduced SLO violations compared to existing methods.
What Problem Does This Paper Address?
The explosion in popularity of text-to-image (T2I) models like DALL-E and Stable Diffusion has created a pressing need for scalable and efficient inference serving systems. However, these diffusion models are notoriously slow due to their iterative denoising process, leading to high inference latency. This makes it difficult to serve a large volume of user requests while maintaining acceptable response times and image quality. Traditional approaches like batching, which work well for smaller ML models, are ineffective for these compute-bound T2I models. Furthermore, simply using faster, approximated models can lead to significant quality degradation if not carefully managed. The core challenge is to design a system that can dynamically adapt to fluctuating workloads and prompt characteristics to deliver high throughput without sacrificing image quality.
Methods
ARGUS tackles this challenge with a multi-pronged approach:
- Prompt-Aware Approximation: The system uses a lightweight classifier to predict the optimal level of approximation for each incoming prompt. This classifier is trained to understand how much approximation a prompt can tolerate without a noticeable drop in image quality.
- Two Approximation Strategies: ARGUS supports two primary strategies for reducing latency:
- Smaller Model Variants (SM): Using distilled or smaller versions of the T2I model.
- Approximate Caching (AC): Reusing intermediate denoising states from previous generations, reducing the number of steps needed for a new prompt.
- Quality-Aware Scheduling: An optimization solver (using Integer Linear Programming) determines the ideal distribution of prompts across different approximation levels to meet throughput targets while maximizing overall image quality.
- Probabilistic Approximation Shift Map (PASM): When the ideal prompt-to-approximation mapping isn't feasible due to system load, PASM provides a mechanism to probabilistically redistribute prompts to alternate approximation levels with minimal quality degradation.
- Adaptive Strategy Switching: ARGUS dynamically switches between AC and SM strategies based on real-time network conditions and workload characteristics. For instance, if cache retrieval latency spikes due to network issues, it can seamlessly switch to SM to maintain performance.
Datasets
- DiffusionDB [76]: Used for training the prompt classifier and for evaluating image quality and system performance. This dataset contains a large collection of text-to-image prompts.
- Twitter Traces: A real-world workload trace used for evaluation, known for its diurnal patterns and unexpected spikes, making it representative of production traffic.
- SysX Workload: A proprietary T2I service trace, normalized to match the SysX trace's min-max range, used to capture realistic production load patterns.
- Synthetic Workloads: Bursty and increasing QPS patterns were generated to stress-test the system under various load conditions.
Key Insights and Ideas
- Prompt Tolerance is Key: A significant portion of T2I prompts are surprisingly tolerant to approximation, meaning faster, less precise models can still produce images of comparable quality. This is a critical observation that ARGUS leverages.
- Indiscriminate Approximation Hurts Quality: While approximation is necessary for speed, applying it without considering prompt-specific tolerance leads to unacceptable quality degradation. ARGUS's prompt-aware approach is crucial here.
- No One-Size-Fits-All Approximation: Neither Approximate Caching nor Smaller Model Variants are universally superior. The optimal choice depends on the current workload and network conditions, necessitating an adaptive switching mechanism.
- Batching is Ineffective for T2I: Due to the compute-bound nature of diffusion models, batching does not provide the throughput gains seen in other ML tasks. ARGUS avoids batching entirely.
- Classifier Accuracy Directly Impacts Image Quality: The performance of the prompt-aware classifier is directly correlated with the quality of the generated images. Even small drops in classifier accuracy can lead to noticeable quality degradation.
Key Results
- 10x Fewer SLO Violations: ARGUS significantly reduces Service Level Objective (SLO) violations compared to baselines.
- 10% Higher Average Quality: Achieves better image quality on average.
- 40% Higher Throughput: Demonstrates a substantial increase in the number of images served.
- Improved Cluster Utilization: Achieves 1.5-2x higher GPU utilization compared to static over-provisioning.
- Human Perception Study: ARGUS-generated images were consistently preferred by human evaluators for prompt relevance and overall quality.
- Robustness to Faults: The system gracefully handles GPU failures and cache retrieval issues by adapting its strategy.
Limitations and Future Work
- No Autoscaling: ARGUS currently operates on a fixed-sized cluster and does not implement automatic scaling (adding/removing GPUs). The authors suggest that saturation of approximation levels could serve as a signal for horizontal scaling.
- Classifier Retraining: While the classifier is retrained periodically to adapt to data drift, a more robust solution could involve online or active learning for real-time adaptation.
- Solver Scalability: While the ILP solver is efficient for current cluster sizes, its latency could become a bottleneck for extremely large clusters or very frequent reconfigurations. Future work might explore lightweight heuristics or hierarchical optimization.
- Mixed Approximation Strategies: ARGUS currently uses either AC or SM, not a combination, due to the complexity of training a classifier for mixed strategies. Future work could explore this.
Summary updated