TL;DR
Automating the update of machine learning models in production is crucial but challenging due to model variance. This paper introduces Automated Data-Shape Stratified Model Updates (ADSMU), a novel framework that iteratively improves models by stratifying data and testing performance within these strata. ADSMU achieved a 26% relative accuracy improvement on new use cases for a large-scale NLU system, significantly reducing manual effort.
What Problem Does This Paper Address?
Machine learning systems, especially those using complex deep learning architectures, are prone to high model variance. This means that simply retraining a model with new data doesn't guarantee accurate performance across all types of inputs or use cases. Two critical challenges arise:
- Uneven Coverage: Retraining might improve performance on some new data but fail to adequately cover specific subsets of new use cases.
- Regression: Updates can inadvertently degrade performance on previously working parts of the system.
Traditionally, addressing these issues in production involves extensive manual retraining and failure analysis, which is time-consuming and costly, especially given the complexity of production systems and the need for rigorous testing of both inference and engineering performance. Existing AutoML research primarily focuses on building the initial model, not on the ongoing maintenance and updating of deployed models.
Methods
The paper proposes Automated Data-Shape Stratified Model Updates (ADSMU), a framework designed to automate the process of updating deployed ML models. ADSMU operates in three main phases:
- Automated Data Stratification: The system uses a user-defined "rule" to split the training data into meaningful groups (strata) based on data characteristics. This can be achieved through:
- Exact Match: Stratifying based on identical patterns in data features (e.g., utterance shapes in NLU).
- Fuzzy Match: Allowing for more variation in patterns, potentially matching based on slot trails rather than exact labels.
- Model-Based: Using unsupervised methods like clustering on sentence embeddings to group data based on semantic and syntactic similarity.
-
Stratified Model Measurement: Instead of relying on a single overall accuracy metric, ADSMU measures performance by averaging the performance across each stratification group. This ensures that underperformance in less common or rarer data subsets is not overlooked. The proposed metric is: $ \bar{x} = \frac{1}{L} \sum_{h=1}^{L} T_h $ where $L$ is the number of strata and $T_h$ is the average success metric for stratum $h$.
-
Iterative Model Improvement: When a model shows failures within a specific stratification group, ADSMU synthetically generates training data tailored to improve performance in that failing group. This process is iterative: data is added, the model is retrained, performance is re-evaluated across strata, and if necessary, more data is added to failing groups. This cycle repeats until performance reaches an acceptable level or a maximum number of iterations is met. The framework is designed to be architecture-agnostic and can be applied on top of existing models.
Datasets
The paper describes experiments conducted on a two-stage model for domain classification (DC) and a joint intent classification (IC) and named entity recognition (NER) task. The training data used was a large, complex production system's data, including user data (preprocessed for de-identification) and synthetic data, representing utterances across multiple domains and intents. The authors do not introduce new benchmarks but use a realistic proxy for production NLU systems.
Key Insights and Ideas
- The "Maintenance Gap" in AutoML: While AutoML excels at building initial models, the critical task of updating and maintaining deployed models remains largely manual and challenging.
- Stratification as a Key to Robustness: Simply retraining a model isn't enough. Stratifying data based on meaningful characteristics allows for targeted measurement and improvement, ensuring performance across diverse data subsets.
- Beyond Average Accuracy: Focusing solely on overall accuracy can mask significant performance drops in specific, potentially critical, data segments. Stratified measurement provides a more nuanced view.
- Synthetic Data for Targeted Improvement: Generating synthetic data specifically for underperforming strata is an effective way to boost performance without requiring extensive manual data collection.
- Iterative Refinement is Powerful: The iterative nature of ADSMU allows for continuous improvement, adapting to new data and use cases more effectively than a single update.
Key Results
- ADSMU achieved a 26% relative improvement in accuracy for new use cases compared to a naive retrained baseline and current cutting-edge methods.
- The adoption of ADSMU corresponded to a 94% timeline reduction in model updates for supporting new NLU use cases, shifting from months to days.
- In an exemplar case comparing manual model updates to ADSMU for a spoken language understanding task, ADSMU converged faster and performed better, showing a relatively 24% lower error rate for the new use case.
- Combining ADSMU with submodular optimization yielded even more significant gains, highlighting the potential for hybrid approaches.
Limitations and Future Work
The authors acknowledge three key limitations:
- Domain Knowledge Dependency: The reliance on user-defined stratification rules requires a certain degree of domain expertise to define effective stratification heuristics.
- Suitability for High-Variance Architectures: The framework is most beneficial for high-variance model architectures, where performance variability across data types and builds is more pronounced.
- Computational Cost: The iterative nature of ADSMU imposes a higher computational cost compared to a single model rebuild.
Future work could explore:
- Removing the need for manual iteration by optimizing stratified loss functions or adding training data incrementally within epochs.
- Developing methods to automatically discover optimal stratification heuristics.
- Investigating ways to reduce the computational overhead of the iterative process.
Summary updated