MLink: Linking Black-Box Models for Collaborative Multi-Model Inference

Mu Yuan, Lan Zhang, Xiang-Yang Li

Introduction

The proliferation of intelligent services, ranging from smart city monitoring to autonomous driving, necessitates the simultaneous execution of multiple machine learning (ML) models. These models often perform diverse tasks such as object detection, scene classification, and image captioning. However, deploying these models on resource-constrained edge devices presents a significant challenge: the cumulative demand for GPU memory and compute often exceeds the available hardware budget.

Traditional approaches like Multi-Task Learning (MTL) require joint training and architectural redesign, which is impractical for proprietary black-box models. Model compression and frame filtering, while reducing load, often degrade accuracy or ignore the semantic relationships between concurrent tasks. MLink addresses these issues by introducing a "model linking" framework that exploits the inherent correlations between different models' output spaces.

Methodology: Model Linking

MLink bridges the knowledge of disparate black-box models by learning lightweight mappings, or "model links," between their outputs. This allows the system to execute only a subset of models and predict the results of the remaining ones.

To accommodate various output formats, MLink utilizes four specialized architectures:

  • Vec-to-Vec: Employs a Multi-Layer Perceptron (MLP) for fixed-length vector mappings (e.g., classification).
  • Seq-to-Vec: Uses an LSTM to map variable-length sequences to vectors.
  • Vec-to-Seq: Utilizes an Encoder-Decoder framework for vector-to-sequence mappings (e.g., image-to-caption).
  • Seq-to-Seq: Implements a standard sequence-to-sequence framework for mapping between variable-length outputs.

The scheduler is responsible for selecting the optimal subset of models to execute given a resource budget $B$. It maximizes the average output accuracy by balancing exact inference for "activated" models and predicted outputs for the rest.

Optimization objective for the MLink scheduler to maximize accuracy under a resource budget.

The scheduler employs a greedy selection algorithm based on a heuristic "activation probability," which considers both the execution cost and the predictive utility of each model. To handle dynamic environments, the scheduler periodically re-profiles input data to adapt its selection strategy.

Key Insights: Semantic Overlap

A core observation of this work is the "overlearning" phenomenon: ML models often capture more information than their specific task requires. For instance, an object detector and a scene classifier frequently focus on the same semantic regions of an image. MLink leverages this shared knowledge.

Correlation between attention overlap ratio and average prediction accuracy.

Empirical analysis shows a strong correlation between the attention overlap of two models and the accuracy of the links between them. As shown in the figure above, higher attention overlap ratios lead to significantly better average prediction accuracy, validating the use of output-space mappings to capture inter-model dependencies.

Furthermore, the authors found that linking models via final outputs is more effective than using intermediate features (knowledge distillation), as outputs provide more explicit semantic signals for scheduling. These links are remarkably sample-efficient, requiring as little as 1% of the original training data to achieve high performance.

Evaluation and Results

MLink was evaluated on the Hollywood2 video dataset and real-world video analytics systems (Smart Building and City Traffic Monitoring). The results demonstrate that MLink can save up to 66.7% of inference computations while maintaining 94.1% output accuracy.

In a resource-constrained Smart Building testbed, MLink achieved 94.1% accuracy, a substantial improvement over the 33.3% accuracy achieved by a standalone execution strategy under the same budget. The overhead of the model links themselves is negligible, ensuring compatibility with mobile and edge hardware.

Conclusion

MLink provides a practical and efficient framework for collaborative multi-model inference. By treating models as black boxes and linking them through their output spaces, it enables high-accuracy execution on resource-limited devices. Future work will explore scaling to larger model sets and enhancing online adaptation to rapid environmental changes.

Summary updated

Abstract

The cost efficiency of model inference is critical to real-world machine learning (ML) applications, especially for delay-sensitive tasks and resource-limited devices. A typical dilemma is: in order to provide complex intelligent services (e.g. smart city), we need inference results of multiple ML models, but the cost budget (e.g. GPU memory) is not enough to run all of them. In this work, we study underlying relationships among black-box ML models and propose a novel learning task: model linking. Model linking aims to bridge the knowledge of different black-box models by learning mappings (dubbed model links) between their output spaces. Based on model links, we developed a scheduling algorithm, named MLink. Through collaborative multi-model inference enabled by model links, MLink can improve the accuracy of obtained inference results under the cost budget. We evaluated MLink on a multi-modal dataset with seven different ML models and two real-world video analytics systems with six ML models and 3,264 hours of video. Experimental results show that our proposed model links can be effectively built among various black-box models. Under the budget of GPU memory, MLink can save 66.7% inference computations while preserving 94% inference accuracy, which outperforms multi-task learning, deep reinforcement learning-based scheduler and frame filtering baselines.
BibTeX
@inproceedings{yuan2022mlink,
  author = {Mu Yuan and Lan Zhang and Xiang-Yang Li},
  title = {{MLink: Linking Black-Box Models for Collaborative Multi-Model Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i9.21180},
}

Loading related content...

Figure