Abstract
MLModelCI provides multimedia researchers and developers with a one-stop platform for efficient machine learning (ML) services. The system leverages DevOps techniques to optimize, test, and manage models. It also containerizes and deploys these optimized and validated models as cloud services (MLaaS). In its essence, MLModelCI serves as a housekeeper to help users publish models. The models are first automatically converted to optimized formats for production purpose and then profiled under different settings (e.g., batch size and hardware). The profiling information can be used as guidelines for balancing the trade-off between performance and cost of MLaaS. Finally, the system dockerizes the models for ease of deployment to cloud environments. A key feature of MLModelCI is the implementation of a controller, which allows elastic evaluation which only utilizes idle workers while maintaining online service quality. Our system bridges the gap between current ML training and serving systems and thus free developers from manual and tedious work often associated with service deployment. We release the platform as an open-source project on GitHub under Apache 2.0 license, with the aim that it will facilitate and streamline more large-scale ML applications and research projects.
Introduction
The "deployment gap" in machine learning remains a significant bottleneck in the industry. While training a model in a Python script is relatively straightforward, turning that model into a scalable, efficient, and robust web service is an arduous journey. Industry data suggests that 40% of companies take over a month to deploy a single model due to environment complexity, the need for performance optimization, and the difficulty of navigating performance-cost trade-offs.
MLModelCI addresses these challenges by introducing a "one-stop" platform inspired by standard software DevOps (Continuous Integration/Continuous Deployment). It automates the transition from a trained model to a production-ready cloud service, reducing the complexity of deploying Machine Learning as a Service (MLaaS) from hundreds of lines of boilerplate code to just a few.
System Architecture
MLModelCI is designed with a modular, framework-agnostic architecture that spans from user interfaces to backend serving engines.
The platform's core consists of several key modules:
- ModelHub: A centralized repository using MongoDB and GridFS to store model weights and metadata.
- Auto-Converter: Automatically translates models from research frameworks (like PyTorch or TensorFlow) into deployment-ready formats (TorchScript, ONNX, TensorRT).
- Dynamic Profiler: Measures real-world latency (P50, P95, P99) and throughput on actual hardware with various batch sizes, moving beyond deceptive static metrics like FLOPs.
- Elastic Controller: A management layer that schedules resource-intensive profiling tasks only when hardware is idle (e.g., utilization < 40%), ensuring background testing does not interrupt live services.
- Dispatcher: Handles containerizing models and deploying them via systems like TensorFlow-Serving or Triton Inference Server.
Evaluation and Key Results
To evaluate the system, the authors utilized widely recognized pre-trained models such as ResNet50 and Mask R-CNN. The results demonstrate significant improvements in both developer productivity and operational insight.
Code and Time Efficiency
In a deployment test for Mask R-CNN, using standard TensorFlow-Serving required over 500 lines of code. With MLModelCI, the same task was accomplished in roughly 20 lines. This reduction allows the development cycle for MLaaS to be compressed from weeks to minutes.
Granular Profiling
The system generates detailed performance reports that allow developers to make informed decisions about hardware and software stacks. By comparing throughput and GPU utilization across different serving platforms and hardware (e.g., Tesla T4 vs. P4), users can identify the most cost-effective configuration for their specific needs.
Conclusion
MLModelCI streamlines the "last mile" of machine learning deployment by providing a robust, automated pipeline for model conversion, profiling, and containerization. By harvesting idle resources for elastic evaluation and lowering the barrier to entry for researchers, the platform bridges the gap between training and serving. Future work will focus on expanding framework support and further optimizing the delivery pipeline to move closer to real-time "push-to-deploy" capabilities for large-scale applications.
Summary updated