TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers

Zhibo Wang, Z. Zhang, Xiaoyi Pang, Qile Zhang, Xuanyi Hao, Shuguo Zhuo, Peng Sun

TL;DR

TAP-ViTs is a novel framework that enables task-specific pruning of Vision Transformers (ViTs) for mobile devices without ever accessing a user's raw private data. By using Gaussian Mixture Models (GMMs) to create a "proxy" dataset on the cloud, the system generates customized, lightweight models that often outperform the original, unpruned versions while using 30% fewer parameters.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision, but they are notoriously "heavy"—requiring massive amounts of memory and compute power. This makes them difficult to deploy on edge devices like smartphones or IoT cameras.

While pruning (removing redundant neurons and layers) is a standard solution, it faces a "one-size-fits-all" problem. Most pruning methods create a single generic model for every device. However, a camera in a self-driving car needs different "knowledge" than a camera in a medical diagnostic tool.

To customize a model for a specific task, you usually need to fine-tune it on local data. This presents two major hurdles:

  1. Privacy: Users don't want to upload their private photos to a cloud server for pruning.
  2. Resource Constraints: Edge devices often lack the hardware to perform the intensive training required for fine-tuning.

TAP-ViTs bridges this gap by enabling privacy-preserving, task-adaptive pruning that happens on the cloud but is tailored to the specific data distribution of the end device.

Methods

The authors introduce two core innovations to make this work:

1. GMM-based Metric Dataset Construction

Instead of sending raw images to the cloud, the edge device extracts feature vectors from its local data and fits them to a Gaussian Mixture Model (GMM). The device only uploads the lightweight GMM parameters (means and covariances). On the cloud side, the server uses these parameters to "scout" through a large public dataset. It selects samples that statistically match the device's GMM to build a proxy metric dataset. This proxy dataset acts as a stand-in for the user's private data, allowing the cloud to understand the "task" without seeing the actual images.

2. Dual-Granularity Importance Evaluation

Once the proxy dataset is ready, TAP-ViTs applies a sophisticated pruning strategy:

  • Composite Neuron Importance: Rather than just looking at weight magnitude, it evaluates neurons based on three dimensions: Activeness (how often they fire), Redundancy (how much they overlap with other neurons), and Relevance (how much they actually contribute to the final prediction).
  • Adaptive Layer Importance: Not all layers are equally important for every task. The framework uses Kullback-Leibler (KL) divergence to measure how much pruning a specific layer would shift the model's output. This allows the system to prune "unimportant" layers more aggressively while preserving "critical" ones.

Datasets

The researchers evaluated TAP-ViTs using several standard backbones (DeiT-Base, DeiT-Small, T2T-ViT, and Swin Transformer) across two primary datasets:

  • TinyImageNet
  • CIFAR-100

To simulate real-world "tasks," they partitioned these datasets by semantic labels, assigning different groups of classes to different simulated "devices."

Key Insights and Ideas

  • The "Aha!" Moment on Task Sensitivity: The authors proved that neurons essential for one task are often useless for another. Their heatmap analysis (Fig. 1) showed near-orthogonal importance patterns across different tasks, confirming that "one-size-fits-all" pruning is fundamentally flawed.
  • Pruning as a Regularizer: In several tests, the pruned model actually outperformed the original unpruned model. This suggests that for specific edge tasks, the extra parameters in a full ViT act as "noise." Removing them helps the model generalize better to the specific local task.
  • Privacy doesn't have to be expensive: By using GMMs, the "privacy tax" (the computational cost of protecting data) is shifted to the cloud. The device-side work is minimal.

Key Results

  • Superior Accuracy: On the DeiT-Base architecture, TAP-ViTs achieved 88.24% accuracy on TinyImageNet while retaining only 70% of parameters—outperforming the original unpruned model by +2.86%.
  • Beating the SOTA: It consistently outperformed state-of-the-art baselines (like MD-ViT and UP-ViT) even when those baselines were given more training time and data.
  • Efficiency: Real-world testing on an NVIDIA Jetson Orin Nano showed that the on-device GMM fitting takes less than one minute for 1,000 samples, making it highly practical for deployment.

Limitations and Future Work

  • Public Data Dependency: The method assumes the cloud has access to a "rich enough" public dataset to find matches for the device's GMM. If the local task is extremely niche (e.g., specific satellite imagery), the cloud might not find good proxy samples.
  • Generative Solutions: The authors suggest that future versions could use Generative Foundation Models to synthesize proxy data from scratch based on the GMM parameters, rather than just searching a public database.
  • Beyond Vision: While this paper focuses on ViTs, the dual-granularity pruning logic could potentially be extended to Large Language Models (LLMs) or multi-modal models.

Summary updated

Abstract

Vision Transformers (ViTs) have demonstrated strong performance across a wide range of vision tasks, yet their substantial computational and memory demands hinder efficient deployment on resource-constrained mobile and edge devices. Pruning has emerged as a promising direction for reducing ViT complexity. However, existing approaches either (i) produce a single pruned model shared across all devices, ignoring device heterogeneity, or (ii) rely on fine-tuning with device-local data, which is often infeasible due to limited on-device resources and strict privacy constraints. As a result, current methods fall short of enabling task-customized ViT pruning in privacy-preserving mobile computing settings. This paper introduces TAP-ViTs, a novel task-adaptive pruning framework that generates device-specific pruned ViT models without requiring access to any raw local data. Specifically, to infer device-level task characteristics under privacy constraints, we propose a Gaussian Mixture Model (GMM)-based metric dataset construction mechanism. Each device fits a lightweight GMM to approximate its private data distribution and uploads only the GMM parameters. Using these parameters, the cloud selects distribution-consistent samples from public data to construct a task-representative metric dataset for each device. Based on this proxy dataset, we further develop a dual-granularity importance evaluation-based pruning strategy that jointly measures composite neuron importance and adaptive layer importance, enabling fine-grained, task-aware pruning tailored to each device's computational budget. Extensive experiments across multiple ViT backbones and datasets demonstrate that TAP-ViTs consistently outperforms state-of-the-art pruning methods under comparable compression ratios.
BibTeX
@misc{wang2026tap,
  author = {Zhibo Wang and Z. Zhang and Xiaoyi Pang and Qile Zhang and Xuanyi Hao and Shuguo Zhuo and Peng Sun},
  title = {{TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers}},
  year = {2026},
  eprint = {2601.02437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.02437},
}
Figure