MLModelCI: An Automatic Cloud Platform for Efficient MLaaS

Huaizheng Zhang, Yuanming Li, Yizheng Huang, Yonggang Wen, Jianxiong Yin, Kyle Guan

Abstract

MLModelCI provides multimedia researchers and developers with a one-stop platform for efficient machine learning (ML) services. The system leverages DevOps techniques to optimize, test, and manage models. It also containerizes and deploys these optimized and validated models as cloud services (MLaaS). In its essence, MLModelCI serves as a housekeeper to help users publish models. The models are first automatically converted to optimized formats for production purpose and then profiled under different settings (e.g., batch size and hardware). The profiling information can be used as guidelines for balancing the trade-off between performance and cost of MLaaS. Finally, the system dockerizes the models for ease of deployment to cloud environments. A key feature of MLModelCI is the implementation of a controller, which allows elastic evaluation which only utilizes idle workers while maintaining online service quality. Our system bridges the gap between current ML training and serving systems and thus free developers from manual and tedious work often associated with service deployment. We release the platform as an open-source project on GitHub under Apache 2.0 license, with the aim that it will facilitate and streamline more large-scale ML applications and research projects.

Introduction

The "deployment gap" in machine learning remains a significant bottleneck in the industry. While training a model in a Python script is relatively straightforward, turning that model into a scalable, efficient, and robust web service is an arduous journey. Industry data suggests that 40% of companies take over a month to deploy a single model due to environment complexity, the need for performance optimization, and the difficulty of navigating performance-cost trade-offs.

MLModelCI addresses these challenges by introducing a "one-stop" platform inspired by standard software DevOps (Continuous Integration/Continuous Deployment). It automates the transition from a trained model to a production-ready cloud service, reducing the complexity of deploying Machine Learning as a Service (MLaaS) from hundreds of lines of boilerplate code to just a few.

System Architecture

MLModelCI is designed with a modular, framework-agnostic architecture that spans from user interfaces to backend serving engines.

System architecture of MLModelCI, illustrating the layered approach from interface to backend.

The platform's core consists of several key modules:

  • ModelHub: A centralized repository using MongoDB and GridFS to store model weights and metadata.
  • Auto-Converter: Automatically translates models from research frameworks (like PyTorch or TensorFlow) into deployment-ready formats (TorchScript, ONNX, TensorRT).
  • Dynamic Profiler: Measures real-world latency (P50, P95, P99) and throughput on actual hardware with various batch sizes, moving beyond deceptive static metrics like FLOPs.
  • Elastic Controller: A management layer that schedules resource-intensive profiling tasks only when hardware is idle (e.g., utilization < 40%), ensuring background testing does not interrupt live services.
  • Dispatcher: Handles containerizing models and deploying them via systems like TensorFlow-Serving or Triton Inference Server.

Evaluation and Key Results

To evaluate the system, the authors utilized widely recognized pre-trained models such as ResNet50 and Mask R-CNN. The results demonstrate significant improvements in both developer productivity and operational insight.

Code and Time Efficiency

In a deployment test for Mask R-CNN, using standard TensorFlow-Serving required over 500 lines of code. With MLModelCI, the same task was accomplished in roughly 20 lines. This reduction allows the development cycle for MLaaS to be compressed from weeks to minutes.

Granular Profiling

The system generates detailed performance reports that allow developers to make informed decisions about hardware and software stacks. By comparing throughput and GPU utilization across different serving platforms and hardware (e.g., Tesla T4 vs. P4), users can identify the most cost-effective configuration for their specific needs.

Profiling results comparing throughput across serving platforms and GPU utilization by batch size.

Conclusion

MLModelCI streamlines the "last mile" of machine learning deployment by providing a robust, automated pipeline for model conversion, profiling, and containerization. By harvesting idle resources for elastic evaluation and lowering the barrier to entry for researchers, the platform bridges the gap between training and serving. Future work will focus on expanding framework support and further optimizing the delivery pipeline to move closer to real-time "push-to-deploy" capabilities for large-scale applications.

Summary updated

Abstract

MLModelCI provides multimedia researchers and developers with a one-stop platform for efficient machine learning (ML) services. The system leverages DevOps techniques to optimize, test, and manage models. It also containerizes and deploys these optimized and validated models as cloud services (MLaaS). In its essence, MLModelCI serves as a housekeeper to help users publish models. The models are first automatically converted to optimized formats for production purpose and then profiled under different settings (e.g., batch size and hardware). The profiling information can be used as guidelines for balancing the trade-off between performance and cost of MLaaS. Finally, the system dockerizes the models for ease of deployment to cloud environments. A key feature of MLModelCI is the implementation of a controller, which allows elastic evaluation which only utilizes idle workers while maintaining online service quality. Our system bridges the gap between current ML training and serving systems and thus free developers from manual and tedious work often associated with service deployment. We release the platform as an open-source project on GitHub under Apache 2.0 license, with the aim that it will facilitate and streamline more large-scale ML applications and research projects.
BibTeX
@inproceedings{zhang2020mlmodelci,
  author = {Huaizheng Zhang and Yuanming Li and Yizheng Huang and Yonggang Wen and Jianxiong Yin and Kyle Guan},
  title = {{MLModelCI: An Automatic Cloud Platform for Efficient MLaaS}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=g30VDKcuXj},
}
Figure