MLink: Linking Black-Box Models From Multiple Domains for Collaborative Inference

Mu Yuan, Lan Zhang, Zimu Zheng, Yi-Nan Zhang, Xiang-Yang LiIEEE TPAMI 2022 · DOI · OpenReview

Introduction

In the era of smart cities and pervasive computing, edge devices are increasingly tasked with running multiple machine learning models simultaneously—for instance, tracking traffic flow, ensuring pedestrian safety, and detecting accidents. However, executing a suite of high-performance models often exceeds the GPU memory and latency constraints of edge hardware. MLink addresses this "inference dilemma" by introducing a novel framework for model linking. Instead of executing every model in a system, MLink learns to predict the outputs of unexecuted "black-box" models using the results of a subset of executed ones, exploiting inherent semantic correlations between different tasks.

The core innovation of MLink is the transition from standalone inference to a collaborative "Network of Models" (NeMo). By treating models as black boxes, MLink avoids the need for retraining shared backbones (as in multi-task learning) or accessing proprietary internal weights.

Overview of the MLink framework showing the transition from standalone inference to collaborative inference via a Network of Models (NeMo).

Methodology

  1. Model Link Architectures: MLink employs four lightweight architectures (Vec-to-Vec, Seq-to-Vec, Vec-to-Seq, and Seq-to-Seq) to bridge diverse output formats, such as classification vectors, bounding boxes, or text sequences.
  2. Collaborative Inference Scheduler: A greedy algorithm determines which models to activate based on an "Activation Probability," which balances execution cost against the model's predictive power for the rest of the set.
  3. Online Adaptive Training: To mitigate domain shift (e.g., lighting changes in surveillance), MLink utilizes uncertainty-based active learning to selectively retrain links on the fly.
  4. Cross-Domain Aggregation: A federated approach allows for the creation of robust global links by aggregating knowledge from multiple edge devices without compromising raw data privacy.

Why Model Linking Works: Exploiting "Overlearning"

The effectiveness of MLink stems from the observation that deep learning models often "overlearn," capturing semantic information beyond their primary task. For example, a model trained for scene recognition (e.g., "living room") naturally attends to specific objects (e.g., "bed") even if not explicitly trained to detect them.

Visualization of semantic overlap between object and scene models and the resulting correlation between attention overlap and prediction accuracy.

As shown in the figure above, there is a strong correlation between the spatial attention overlap of different models and the accuracy of predicting one from the other. MLink harvests these "unintended semantics" at the output level, where correlations are often more explicit than in intermediate feature maps. Because these links are lightweight (often just a few MLP layers), they are extremely sample-efficient, requiring as little as 1% of the original training data.

Experimental Results

The researchers evaluated MLink across diverse datasets, including the Hollywood2 multi-modal dataset, a large-scale smart building system, and city traffic monitoring.

  • Efficiency: MLink achieved up to a 66.7% reduction in inference computation in real-world video analytics scenarios.
  • Accuracy: Despite the reduced execution, the system maintained 94% of the original standalone accuracy. In some cases, aggregating predictions from multiple source models even improved accuracy by up to 7.9% over standalone baselines.
  • Low Overhead: The scheduling and link execution overhead on mobile platforms (Qualcomm Kryo 485) was negligible, adding less than 0.02s of latency.

Conclusion and Future Work

MLink demonstrates that linking black-box models is a viable and efficient strategy for multi-model inference on resource-constrained devices. While highly effective when semantic correlations exist, the framework's performance is naturally bounded by the relatedness of the tasks. Future research will focus on "smart selection" algorithms to automatically identify the most valuable links within massive, growing model zoos, ensuring scalability as the number of potential pairwise links increases.

Summary updated

Abstract

The cost efficiency of model inference is critical to real-world machine learning (ML) applications, especially for delay-sensitive tasks and resource-limited devices. A typical dilemma is: in order to provide complex intelligent services (e.g., smart city), we need inference results of multiple ML models, but the cost budget (e.g., GPU memory) is not enough to run all of them. In this work, we study underlying relationships among black-box ML models and propose a novel learning task: model linking, which aims to bridge the knowledge of different black-box models by learning mappings (dubbed model links) between their output spaces. We propose the design of model links which supports linking heterogeneous black-box ML models. Also, in order to address the distribution discrepancy challenge, we present adaptation and aggregation methods of model links. Based on our proposed model links, we developed a scheduling algorithm, named MLink . Through collaborative multi-model inference enabled by model links, MLink can improve the accuracy of obtained inference results under the cost budget. We evaluated MLink on a multi-modal dataset with seven different ML models and two real-world video analytics systems with six ML models and 3,264 hours of video. Experimental results show that our proposed model links can be effectively built among various black-box models. Under the budget of GPU memory, MLink can save 66.7% inference computations while preserving 94% inference accuracy, which outperforms multi-task learning, deep reinforcement learning-based scheduler and frame filtering baselines.
BibTeX
@misc{yuan2022mlink,
  author = {Mu Yuan and Lan Zhang and Zimu Zheng and Yi-Nan Zhang and Xiang-Yang Li},
  title = {{MLink: Linking Black-Box Models From Multiple Domains for Collaborative Inference}},
  year = {2022},
  eprint = {2209.13883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.13883},
}
Figure