Cost-effective Deployment of BERT Models in Serverless Environment

Katarína Benesová, Andrej Svec, Marek Suppa

Abstract

In this study we demonstrate the viability of deploying BERT-style models to serverless environments in a production setting. Since the freely available pre-trained models are too large to be deployed in this way, we utilize knowledge distillation and fine-tune the models on proprietary datasets for two real-world tasks: sentiment analysis and semantic textual similarity. As a result, we obtain models that are tuned for a specific domain and deployable in serverless environments. The subsequent performance analysis shows that this solution results in latency levels acceptable for production use and that it is also a cost-effective approach for small-to-medium size deployments of BERT models, all without any infrastructure overhead.

Introduction

Large language models (LLMs) like BERT have revolutionized natural language processing but present significant challenges for production deployment. Traditionally, these models require expensive, always-on infrastructure, such as GPU-enabled virtual machines, to manage their substantial size and computational requirements.

Serverless computing, or Function-as-a-Service (FaaS), offers a compelling alternative with its automatic scaling and pay-per-use pricing model. However, serverless platforms impose strict constraints:

  1. Size Limits: Deployment packages are often capped (e.g., 250MB–500MB), whereas a standard BERT model with its dependencies can exceed 1GB.
  2. Hardware Limitations: Most serverless functions lack GPU acceleration, leading to high latency.
  3. Memory Constraints: Limited RAM can lead to execution failures for large models.

This study addresses these hurdles, demonstrating how to fit BERT models into serverless environments without significant accuracy loss.

Methodology

The researchers utilized a three-pronged approach to optimize models for serverless deployment:

  1. Knowledge Distillation: Large BERT-base "teacher" models were used to train smaller "student" models, specifically TinyBERT and MobileBERT. These students mimic the teacher's behavior on specific tasks while being significantly smaller.
  2. Task-Specific Fine-Tuning: The study focused on two use cases: Sentiment Analysis (using a classification head on the [CLS] token) and Semantic Textual Similarity (STS) (using a Sentence-BERT architecture for dense vector embeddings).
  3. ONNX Runtime: To bypass the large footprint of standard libraries like PyTorch (~400MB), models were converted to the Open Neural Network Exchange (ONNX) format. The ONNX Runtime library is only 14MB, significantly reducing the deployment package size.
Sentiment analysis model distillation and deployment pipeline.

Data Strategy

The training involved a combination of public benchmarks (SNLI, Multi-Genre NLI, STSb) and proprietary "gold" data from Slido (68,000 sentiment examples and 2,856 similarity pairs). To enhance distillation, "silver" datasets were generated—900,000 unlabeled sentiment examples and 32,000 similarity pairs—using techniques like back-translation and cross-encoder labeling.

Key Results and Performance

The optimization strategy yielded impressive results in both model size and performance:

  • Compression: Models were reduced from 420MB (BERT-base) to 56MB (TinyBERT) and 98MB (MobileBERT).
  • Accuracy: TinyBERT achieved an F1 score of 0.82, nearly matching BERT-base’s 0.84, proving that the performance trade-off is minimal for production tasks.
  • Latency: In AWS Lambda with 1GB of RAM, the 99th percentile (q99) execution time remained under 100ms for both tasks.
Execution time vs. Lambda allocated RAM for (a) Sentiment analysis and (b) SBERT encoding.

The cost analysis revealed that 1 million predictions on serverless cost approximately $2. Compared to the ~$8/month minimum for a dedicated EC2 instance, serverless is highly cost-effective for low-to-medium traffic scenarios.

Discussion and Conclusion

While serverless deployment is highly efficient for moderate traffic, dedicated instances may become more economical for very high, consistent loads due to per-request pricing. Additionally, "cold starts"—the latency experienced when a function initializes after inactivity—remain a factor in real-world environments.

In conclusion, the combination of knowledge distillation, task-specific fine-tuning, and the ONNX Runtime enables the deployment of powerful BERT models in serverless environments. This approach provides a scalable, cost-effective solution for production NLP tasks without the overhead of managing traditional infrastructure.

Summary updated

Abstract

In this study we demonstrate the viability of deploying BERT-style models to serverless environments in a production setting. Since the freely available pre-trained models are too large to be deployed in this way, we utilize knowledge distillation and fine-tune the models on proprietary datasets for two real-world tasks: sentiment analysis and semantic textual similarity. As a result, we obtain models that are tuned for a specific domain and deployable in serverless environments. The subsequent performance analysis shows that this solution results in latency levels acceptable for production use and that it is also a cost-effective approach for small-to-medium size deployments of BERT models, all without any infrastructure overhead.
BibTeX
@inproceedings{benesova2021cost,
  author = {Katarína Benesová and Andrej Svec and Marek Suppa},
  title = {{Cost-effective Deployment of BERT Models in Serverless Environment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Se1HePXz-7g},
}
Figure