Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators

Afzal Ahmad, Gaoyu Mao, Shoubo Hu, Hui-Ling Zhen, Mingxuan Yuan, Xinyu Chen, Wei ZhangarXiv 2026 · arXiv

TL;DR

Multi-task neural networks waste significant energy and compute by running their entire backbone even when only a specific sub-task (like braking or lane-following) is active. This paper introduces "Sparse by Command," a hardware/software co-designed system that uses high-level task commands to dynamically skip unnecessary computation tiles at the hardware level. Prototyped on an FPGA, this approach slashes FLOPs by up to 76% and speeds up inference by over 2× with zero runtime overhead.

What Problem Does This Paper Address?

Multi-task models are highly efficient for autonomous systems like self-driving cars, where a single vision backbone serves multiple operational heads (e.g., lane following, turning, or braking). However, standard hardware executes the entire network uniformly, regardless of which task is currently active. For instance, a lane-following command might not need the feature channels dedicated to detecting traffic lights or obstacles for braking, yet traditional accelerators still fetch those weights from memory and compute those channels anyway.

Prior attempts at dynamic sparsity (like skipping residual blocks or dynamic channel pruning) often fail to deliver actual wall-clock speedups on real hardware. This is because their sparsity patterns do not align with the hardware's native tiling boundaries; "skipping" a channel usually just means multiplying its output by zero while still incurring the memory bandwidth cost of fetching the weights. The authors address this gap by exploiting a "free" signal—the high-level task command—which is known before inference begins and remains stable over many frames, allowing the hardware to completely bypass unnecessary tiles.

Methods

The authors propose a full-stack co-design spanning the model, the instruction set, and the hardware architecture:

  1. Task-Conditional Gating Network: A lightweight Multi-Layer Perceptron (MLP), accounting for less than 0.12% of the backbone parameters, maps a one-hot task command to a binary execution mask. This mask dictates which "tiles" (groups of 16 output channels) are needed. The network is trained using a three-phase pipeline: standard dense training, soft masking with a sparsity penalty, and hard binarized masking using a straight-through estimator.
  2. Neural Instruction Set Architecture (NISA): The authors design a custom ISA where each layer's instruction carries a 128-bit field dedicated to tile execution masks.
  3. Tiled Inference Accelerator: Implemented with an INT8 datapath on an FPGA, the accelerator's tile manager reads the NISA bitmask. If a bit is zero, the hardware advances past that tile in a single cycle, completely skipping the DMA weight fetch, activation load, and compute phases. Crucially, this sparsity propagates across layers: skipping an output tile in layer $L$ automatically eliminates the corresponding input tile computation in layer $L+1$.
Overview of task-conditional tile sparsity and its latency impact on an FPGA.

Datasets

The system was evaluated on a closed-loop visuomotor driving task within the CARLA autonomous driving simulator (specifically using the urban environment Town10HD_Opt).

  • Training Data: A dataset of 302K frames was collected using an expert rule-based controller. To prevent the model from collapsing into a simple "drive straight" policy, the dataset was balanced by removing 90% of the redundant straight-driving frames. It covers six distinct commands: follow lane/go straight, turn left, turn right, change lane left, change lane right, and brake.
  • Evaluation: Testing was performed across three challenging routes (Hard 1, Hard 2, and Hard 3) featuring multi-lane merges, tight turns, and intersections.

Key Insights and Ideas

  • The Command is a "Free" Signal: Unlike typical dynamic networks that must process intermediate image activations at runtime to decide what to skip, the task command is known beforehand. Because a command like "turn left" persists for hundreds of frames, the tile mask can be resolved just once when the command changes, reducing runtime gating overhead to near zero.
  • The "Polysemantic Core": To ensure the network retains critical shared representations, the authors designate "Tile 0" of every layer as always-on. The gating network only prunes the remaining tiles.
  • Task Complexity Reflection: The learned masks naturally adapt to the complexity of the maneuver. "Brake"—which relies primarily on proximity detection—is pruned aggressively, leaving only 18.7% of tiles active. Conversely, "Turn Left"—which requires navigating oncoming traffic and wider arcs—retains the most capacity (31.3% active).
  • Hardware Support is Mandatory: When the authors ran their sparse model on an NVIDIA RTX 4090 GPU, it was actually 22% slower than the dense baseline. At batch size 1, the GPU is bottlenecked by kernel launch overhead and still fetches all weights. This underscores why custom hardware like their tiled FPGA accelerator is required to translate tile-level sparsity into actual speedups.

Key Results

  • Compute & Latency Reductions: Task-conditional sparsity reduced FLOPs by 66% to 76% across the driving commands. On an AMD/Xilinx Alveo U50 FPGA, on-device latency dropped by 51–59% (from 9.12 ms down to 3.74–4.44 ms), representing a 2.1–2.4× speedup.
  • Energy Savings: On-device energy consumption per inference dropped by 51–59%, from 263 mJ down to 108–128 mJ.
  • Improved Driving Quality: The sparse model maintained 100% route completion in CARLA. Surprisingly, it reduced Cross-Track Error (lateral deviation from the lane) by 21–59% compared to the dense baseline. The authors attribute this to a "task-specialization effect," where masking irrelevant tiles eliminates cross-command interference in the shared backbone.
  • Transformer Generality: The method successfully generalized to a Vision Transformer (ViT-Base) backbone. By masking the MLP linear layers, the authors achieved a 40–62% per-command FLOP reduction without needing a redesigned accelerator.

Limitations and Future Work

  • The Attention Bottleneck: In the Transformer evaluation, the attention mechanism was left entirely dense. This created a compute floor, limiting the ViT's FLOP reduction compared to the CNN. The authors note that integrating attention head-group pruning would lift this restriction.
  • Static vs. Dynamic Environments: The current gating network conditions purely on the high-level command. A promising future direction is to incorporate input-dependent conditioning (e.g., skipping even more tiles if the camera image shows an entirely empty road).
  • Scaling Up: The prototype was evaluated on a mid-range FPGA. Scaling the architecture to larger models and multi-SLR FPGA platforms remains an item for future engineering work.

Summary updated

Abstract

Multi-task inference models share a single backbone across diverse tasks, yet execute identical computation regardless of which task is active - wasting energy and cycles on task-irrelevant operations. We observe that the task command, typically available before inference begins, provides a free signal that can be exploited to skip unnecessary computation at the hardware level. We present a HW/SW co-designed approach in which a lightweight gating network, trained jointly with the backbone, predicts per-tile binary execution masks conditioned on the task input. Each tile corresponds to a fixed group of output channels (the native scheduling granularity of the accelerator), enabling masked tiles to be skipped with zero overhead. This yields a task-dependent reduction in compute, where each command activates only the subset of the network it requires, without changes to the model architecture or inference pipeline. We co-design the full system stack: a command-conditioned training procedure that learns hardware-aligned tile masks under a sparsity objective; an instruction set architecture whose instructions carry per-tile bitmask fields, allowing the hardware to skip masked tiles without software intervention; and a tiled inference accelerator with configurable parallelism, double-buffered memory, and INT8 datapath that natively supports sparse tile execution. We prototype on an AMD/Xilinx Alveo U50 FPGA and evaluate on a closed-loop visuomotor driving task in CARLA autonomous driving simulator. Task-conditional sparsity reduces FLOPs by 66-76% while maintaining driving quality. On-device latency decreases by 51-59%, from 9.12 ms to 3.74-4.44 ms (2.1-2.4x speedup), with energy per inference dropping from 263 to 108-128mJ.
BibTeX
@misc{ahmad2026sparse,
  author = {Afzal Ahmad and Gaoyu Mao and Shoubo Hu and Hui-Ling Zhen and Mingxuan Yuan and Xinyu Chen and Wei Zhang},
  title = {{Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators}},
  year = {2026},
  eprint = {2607.22038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.22038},
}
Figure