Introduction
The proliferation of intelligent services, ranging from smart city monitoring to autonomous driving, necessitates the simultaneous execution of multiple machine learning (ML) models. These models often perform diverse tasks such as object detection, scene classification, and image captioning. However, deploying these models on resource-constrained edge devices presents a significant challenge: the cumulative demand for GPU memory and compute often exceeds the available hardware budget.
Traditional approaches like Multi-Task Learning (MTL) require joint training and architectural redesign, which is impractical for proprietary black-box models. Model compression and frame filtering, while reducing load, often degrade accuracy or ignore the semantic relationships between concurrent tasks. MLink addresses these issues by introducing a "model linking" framework that exploits the inherent correlations between different models' output spaces.
Methodology: Model Linking
MLink bridges the knowledge of disparate black-box models by learning lightweight mappings, or "model links," between their outputs. This allows the system to execute only a subset of models and predict the results of the remaining ones.
Model Link Architectures
To accommodate various output formats, MLink utilizes four specialized architectures:
- Vec-to-Vec: Employs a Multi-Layer Perceptron (MLP) for fixed-length vector mappings (e.g., classification).
- Seq-to-Vec: Uses an LSTM to map variable-length sequences to vectors.
- Vec-to-Seq: Utilizes an Encoder-Decoder framework for vector-to-sequence mappings (e.g., image-to-caption).
- Seq-to-Seq: Implements a standard sequence-to-sequence framework for mapping between variable-length outputs.
The MLink Scheduler
The scheduler is responsible for selecting the optimal subset of models to execute given a resource budget $B$. It maximizes the average output accuracy by balancing exact inference for "activated" models and predicted outputs for the rest.
The scheduler employs a greedy selection algorithm based on a heuristic "activation probability," which considers both the execution cost and the predictive utility of each model. To handle dynamic environments, the scheduler periodically re-profiles input data to adapt its selection strategy.
Key Insights: Semantic Overlap
A core observation of this work is the "overlearning" phenomenon: ML models often capture more information than their specific task requires. For instance, an object detector and a scene classifier frequently focus on the same semantic regions of an image. MLink leverages this shared knowledge.
Empirical analysis shows a strong correlation between the attention overlap of two models and the accuracy of the links between them. As shown in the figure above, higher attention overlap ratios lead to significantly better average prediction accuracy, validating the use of output-space mappings to capture inter-model dependencies.
Furthermore, the authors found that linking models via final outputs is more effective than using intermediate features (knowledge distillation), as outputs provide more explicit semantic signals for scheduling. These links are remarkably sample-efficient, requiring as little as 1% of the original training data to achieve high performance.
Evaluation and Results
MLink was evaluated on the Hollywood2 video dataset and real-world video analytics systems (Smart Building and City Traffic Monitoring). The results demonstrate that MLink can save up to 66.7% of inference computations while maintaining 94.1% output accuracy.
In a resource-constrained Smart Building testbed, MLink achieved 94.1% accuracy, a substantial improvement over the 33.3% accuracy achieved by a standalone execution strategy under the same budget. The overhead of the model links themselves is negligible, ensuring compatibility with mobile and edge hardware.
Conclusion
MLink provides a practical and efficient framework for collaborative multi-model inference. By treating models as black boxes and linking them through their output spaces, it enables high-accuracy execution on resource-limited devices. Future work will explore scaling to larger model sets and enhancing online adaptation to rapid environmental changes.
Summary updated