Learning to Help in Multi-Class Settings

Yu Wu, Yansong Li, Zeyu Dong, Nitya Sathyavageeswaran, Anand D. Sarwate

Abstract

Modern artificial intelligence relies on increasingly large models that exceed the computational capacity of edge devices. While offloading to a server is a common solution, it is often constrained by cost, latency, and connectivity. This paper presents the "Learning to Help" (L2H) framework, a multi-class classification approach designed to augment "legacy" devices with fixed local models. By training a server-side classifier and a local "rejector" using a novel stage-switching surrogate loss, the system learns to selectively offload difficult queries. Experimental results on standard benchmarks demonstrate significant accuracy improvements and the ability of the system to automatically identify and patch the local model's weaknesses.

Introduction

The rapid advancement of deep learning has created a gap between state-of-the-art model performance and the capabilities of resource-constrained hardware, such as IoT sensors and legacy embedded systems. These devices often run "frozen" models that cannot be easily updated. While cloud offloading offers a performance boost, a naive "send-everything" approach is impractical due to per-query costs and unreliable network conditions.

Traditional "Learning to Defer" (L2D) research typically focuses on training a human-in-the-loop or a fixed expert server to assist a trainable client. In contrast, this work explores Learning to Help (L2H), where the client model is treated as a fixed "black box." The goal is to optimize a trainable server model and a decision-making "rejector" to provide help only when the local model is likely to fail.

Methodology

The L2H framework is built around three primary components: a fixed local classifier, a trainable server classifier, and a trainable rejector.

System Architecture

The rejector acts as a gatekeeper on the client device. For every input $x$, the rejector determines whether the local model can provide a reliable prediction or if the task should be offloaded to the remote server.

Stage-Switching Surrogate Loss

Optimizing the decision to offload is challenging because the standard 0-1 loss (measuring correct vs. incorrect classification) is non-differentiable. The authors propose a "stage-switching" surrogate loss function. This function is mathematically consistent, meaning it converges to the optimal Bayes rule as data increases. It allows for the iterative training of the server and the rejector, ensuring they remain synchronized without requiring a constant high-bandwidth connection during the training phase.

Operational Scenarios

The framework is designed to handle three realistic deployment constraints:

  1. Pay-Per-Request (PPR): Each server query incurs a fixed financial or computational cost.
  2. Intermittent Availability (IA): The system must account for potential server downtime or network drops.
  3. Bounded Reject Rate (BRR): The system operates under a strict "data cap," limiting the total number of offloaded queries.

Experimental Evaluation

The researchers evaluated the L2H framework using the CIFAR-10, SVHN, and CIFAR-100 datasets. They utilized a weak LeNet-5 model as the fixed local classifier and paired it with more robust architectures like AlexNet or Vision Transformers (ViT) on the server side.

Key Findings

  • Performance Gains: The hybrid system consistently outperformed the local model. On CIFAR-10, the local model achieved approximately 74% accuracy. By offloading only the most difficult 14% of samples, the L2H system significantly boosted overall accuracy while maintaining cost-efficiency.
  • Automated Patching: In a controlled experiment, the local model was intentionally "blinded" to a specific class (e.g., trucks). The rejector successfully identified this systematic failure and learned to offload nearly 100% of truck-related queries to the server, effectively patching the local model's knowledge gap.
  • Intelligent Filtering: Qualitative analysis of offloaded images revealed that the rejector functions as an effective filter. It retains clear, high-contrast images for local processing while sending blurry, low-resolution, or ambiguous samples to the server.

Conclusion and Future Work

The L2H framework provides a robust solution for extending the lifespan and utility of legacy hardware in the age of massive AI models. By treating the local device as a fixed entity and optimizing the "help" it receives, the system balances accuracy with real-world constraints like cost and connectivity.

Future research directions include addressing the privacy implications of data offloading and extending the framework to multi-party settings where a client can choose between multiple servers with varying costs and expertise levels. Additionally, integrating this framework with Large Language Models (LLMs) could allow small on-device models to act as efficient filters for expensive cloud-based reasoning engines.

Summary updated

Abstract

Deploying complex machine learning models on resource-constrained devices is challenging due to limited computational power, memory, and model retrainability. To address these limitations, a hybrid system can be established by augmenting the local model with a server-side model, where samples are selectively deferred by a *rejector* and then sent to the server for processing. The hybrid system enables efficient use of computational resources while minimizing the overhead associated with server usage. The recently proposed Learning to Help (L2H) model proposed training a server model given a fixed local (client) model. This differs from the Learning to Defer (L2D) framework which trains the client for a fixed (expert) server. In both L2D and L2H, the training includes learning a rejector at the client to determine when to query the server. In this work, we extend the L2H model from binary to multi-class classification problems and demonstrate its applicability in a number of different scenarios of practical interest in which access to the server may be limited by cost, availability, or policy. We derive a stage-switching surrogate loss function that is differentiable, convex, and consistent with the Bayes rule corresponding to the 0-1 loss for the L2H model. Experiments show that our proposed methods offer an efficient and practical solution for multi-class classification in resource-constrained environments.
BibTeX
@inproceedings{wu2025learning,
  author = {Yu Wu and Yansong Li and Zeyu Dong and Nitya Sathyavageeswaran and Anand D. Sarwate},
  title = {{Learning to Help in Multi-Class Settings}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=NCgTbt2j1F},
}
Figure