Introduction
Generative image editing, encompassing tasks like virtual try-on and face swapping, has become a cornerstone of AI cloud workloads. Despite its popularity, the underlying diffusion models are computationally intensive. Current serving systems often treat editing an image as a full generation task, ignoring the fact that large portions of the image (the "unmasked" regions) remain unchanged. This leads to a "high-cost, high-latency" bottleneck, as standard attention mechanisms require processing every pixel to maintain global context.
Technical Innovations
InstGenIE addresses these inefficiencies through three core technical pillars designed to exploit spatial sparsity without sacrificing image quality.
1. Mask-aware Caching and Scheduling
The system identifies that activations for unmasked regions are highly redundant across different editing requests for the same template. InstGenIE caches these intermediate activations. During inference, the GPU computes only the masked tokens and "replenishes" the remaining context from the cache. To manage varying computational loads, a regression-based load balancer routes requests based on mask size to minimize overall latency.
2. Bubble-free Pipeline
Because cached activations are too large for GPU memory (HBM), they are stored in host RAM (DRAM). InstGenIE utilizes a Dynamic Programming (DP) algorithm to schedule a pipeline that overlaps the loading of these activations with the computation of masked tokens. This effectively hides transfer latency, preventing "bubbles" in the processing pipeline.
3. Disaggregated Continuous Batching
Borrowing from Large Language Model (LLM) serving, InstGenIE implements continuous batching, allowing new requests to join a running batch after any single denoising step. To prevent CPU-bound tasks like image encoding from stalling the GPU, these processes are disaggregated into separate execution environments.
Evaluation and Results
The authors evaluated InstGenIE using production-scale traces (including a 14-day workload from a face-swap service) and benchmarks across models like SD2.1, SDXL, and Flux.
- Throughput and Latency: InstGenIE achieves up to 3× higher throughput and reduces average request latency by up to 14.7× compared to baselines like Diffusers and TeaCache.
- Inference Speedup: On the SDXL model using an H800 GPU, the system delivered a 1.7× model inference speedup.
- Image Quality: The system maintains near-perfect fidelity, achieving SSIM scores of 0.99. It also demonstrated superior text-image alignment (CLIP scores) compared to existing sparse-inference methods.
- Tail Latency: The combination of continuous batching and disaggregation reduced P95 tail latency by 29%.
Discussion and Future Work
While InstGenIE provides significant gains for masked edits, its benefits are limited for global transformations like style transfer where every pixel changes. Additionally, extremely high-traffic services must manage host memory effectively to avoid cache misses. Future research may extend these mask-aware techniques to video editing, where temporal redundancy between frames offers further opportunities for computational savings.
Summary updated