Introduction
In the era of smart cities and pervasive computing, edge devices are increasingly tasked with running multiple machine learning models simultaneously—for instance, tracking traffic flow, ensuring pedestrian safety, and detecting accidents. However, executing a suite of high-performance models often exceeds the GPU memory and latency constraints of edge hardware. MLink addresses this "inference dilemma" by introducing a novel framework for model linking. Instead of executing every model in a system, MLink learns to predict the outputs of unexecuted "black-box" models using the results of a subset of executed ones, exploiting inherent semantic correlations between different tasks.
The MLink Framework
The core innovation of MLink is the transition from standalone inference to a collaborative "Network of Models" (NeMo). By treating models as black boxes, MLink avoids the need for retraining shared backbones (as in multi-task learning) or accessing proprietary internal weights.
Methodology
- Model Link Architectures: MLink employs four lightweight architectures (Vec-to-Vec, Seq-to-Vec, Vec-to-Seq, and Seq-to-Seq) to bridge diverse output formats, such as classification vectors, bounding boxes, or text sequences.
- Collaborative Inference Scheduler: A greedy algorithm determines which models to activate based on an "Activation Probability," which balances execution cost against the model's predictive power for the rest of the set.
- Online Adaptive Training: To mitigate domain shift (e.g., lighting changes in surveillance), MLink utilizes uncertainty-based active learning to selectively retrain links on the fly.
- Cross-Domain Aggregation: A federated approach allows for the creation of robust global links by aggregating knowledge from multiple edge devices without compromising raw data privacy.
Why Model Linking Works: Exploiting "Overlearning"
The effectiveness of MLink stems from the observation that deep learning models often "overlearn," capturing semantic information beyond their primary task. For example, a model trained for scene recognition (e.g., "living room") naturally attends to specific objects (e.g., "bed") even if not explicitly trained to detect them.
As shown in the figure above, there is a strong correlation between the spatial attention overlap of different models and the accuracy of predicting one from the other. MLink harvests these "unintended semantics" at the output level, where correlations are often more explicit than in intermediate feature maps. Because these links are lightweight (often just a few MLP layers), they are extremely sample-efficient, requiring as little as 1% of the original training data.
Experimental Results
The researchers evaluated MLink across diverse datasets, including the Hollywood2 multi-modal dataset, a large-scale smart building system, and city traffic monitoring.
- Efficiency: MLink achieved up to a 66.7% reduction in inference computation in real-world video analytics scenarios.
- Accuracy: Despite the reduced execution, the system maintained 94% of the original standalone accuracy. In some cases, aggregating predictions from multiple source models even improved accuracy by up to 7.9% over standalone baselines.
- Low Overhead: The scheduling and link execution overhead on mobile platforms (Qualcomm Kryo 485) was negligible, adding less than 0.02s of latency.
Conclusion and Future Work
MLink demonstrates that linking black-box models is a viable and efficient strategy for multi-model inference on resource-constrained devices. While highly effective when semantic correlations exist, the framework's performance is naturally bounded by the relatedness of the tasks. Future research will focus on "smart selection" algorithms to automatically identify the most valuable links within massive, growing model zoos, ensuring scalability as the number of potential pairwise links increases.
Summary updated