Abstract
Modern artificial intelligence relies on increasingly large models that exceed the computational capacity of edge devices. While offloading to a server is a common solution, it is often constrained by cost, latency, and connectivity. This paper presents the "Learning to Help" (L2H) framework, a multi-class classification approach designed to augment "legacy" devices with fixed local models. By training a server-side classifier and a local "rejector" using a novel stage-switching surrogate loss, the system learns to selectively offload difficult queries. Experimental results on standard benchmarks demonstrate significant accuracy improvements and the ability of the system to automatically identify and patch the local model's weaknesses.
Introduction
The rapid advancement of deep learning has created a gap between state-of-the-art model performance and the capabilities of resource-constrained hardware, such as IoT sensors and legacy embedded systems. These devices often run "frozen" models that cannot be easily updated. While cloud offloading offers a performance boost, a naive "send-everything" approach is impractical due to per-query costs and unreliable network conditions.
Traditional "Learning to Defer" (L2D) research typically focuses on training a human-in-the-loop or a fixed expert server to assist a trainable client. In contrast, this work explores Learning to Help (L2H), where the client model is treated as a fixed "black box." The goal is to optimize a trainable server model and a decision-making "rejector" to provide help only when the local model is likely to fail.
Methodology
The L2H framework is built around three primary components: a fixed local classifier, a trainable server classifier, and a trainable rejector.
System Architecture
The rejector acts as a gatekeeper on the client device. For every input $x$, the rejector determines whether the local model can provide a reliable prediction or if the task should be offloaded to the remote server.
Stage-Switching Surrogate Loss
Optimizing the decision to offload is challenging because the standard 0-1 loss (measuring correct vs. incorrect classification) is non-differentiable. The authors propose a "stage-switching" surrogate loss function. This function is mathematically consistent, meaning it converges to the optimal Bayes rule as data increases. It allows for the iterative training of the server and the rejector, ensuring they remain synchronized without requiring a constant high-bandwidth connection during the training phase.
Operational Scenarios
The framework is designed to handle three realistic deployment constraints:
- Pay-Per-Request (PPR): Each server query incurs a fixed financial or computational cost.
- Intermittent Availability (IA): The system must account for potential server downtime or network drops.
- Bounded Reject Rate (BRR): The system operates under a strict "data cap," limiting the total number of offloaded queries.
Experimental Evaluation
The researchers evaluated the L2H framework using the CIFAR-10, SVHN, and CIFAR-100 datasets. They utilized a weak LeNet-5 model as the fixed local classifier and paired it with more robust architectures like AlexNet or Vision Transformers (ViT) on the server side.
Key Findings
- Performance Gains: The hybrid system consistently outperformed the local model. On CIFAR-10, the local model achieved approximately 74% accuracy. By offloading only the most difficult 14% of samples, the L2H system significantly boosted overall accuracy while maintaining cost-efficiency.
- Automated Patching: In a controlled experiment, the local model was intentionally "blinded" to a specific class (e.g., trucks). The rejector successfully identified this systematic failure and learned to offload nearly 100% of truck-related queries to the server, effectively patching the local model's knowledge gap.
- Intelligent Filtering: Qualitative analysis of offloaded images revealed that the rejector functions as an effective filter. It retains clear, high-contrast images for local processing while sending blurry, low-resolution, or ambiguous samples to the server.
Conclusion and Future Work
The L2H framework provides a robust solution for extending the lifespan and utility of legacy hardware in the age of massive AI models. By treating the local device as a fixed entity and optimizing the "help" it receives, the system balances accuracy with real-world constraints like cost and connectivity.
Future research directions include addressing the privacy implications of data offloading and extending the framework to multi-party settings where a client can choose between multiple servers with varying costs and expertise levels. Additionally, integrating this framework with Large Language Models (LLMs) could allow small on-device models to act as efficient filters for expensive cloud-based reasoning engines.
Summary updated