TL;DR
Multi-task neural networks waste significant energy and compute by running their entire backbone even when only a specific sub-task (like braking or lane-following) is active. This paper introduces "Sparse by Command," a hardware/software co-designed system that uses high-level task commands to dynamically skip unnecessary computation tiles at the hardware level. Prototyped on an FPGA, this approach slashes FLOPs by up to 76% and speeds up inference by over 2× with zero runtime overhead.
What Problem Does This Paper Address?
Multi-task models are highly efficient for autonomous systems like self-driving cars, where a single vision backbone serves multiple operational heads (e.g., lane following, turning, or braking). However, standard hardware executes the entire network uniformly, regardless of which task is currently active. For instance, a lane-following command might not need the feature channels dedicated to detecting traffic lights or obstacles for braking, yet traditional accelerators still fetch those weights from memory and compute those channels anyway.
Prior attempts at dynamic sparsity (like skipping residual blocks or dynamic channel pruning) often fail to deliver actual wall-clock speedups on real hardware. This is because their sparsity patterns do not align with the hardware's native tiling boundaries; "skipping" a channel usually just means multiplying its output by zero while still incurring the memory bandwidth cost of fetching the weights. The authors address this gap by exploiting a "free" signal—the high-level task command—which is known before inference begins and remains stable over many frames, allowing the hardware to completely bypass unnecessary tiles.
Methods
The authors propose a full-stack co-design spanning the model, the instruction set, and the hardware architecture:
- Task-Conditional Gating Network: A lightweight Multi-Layer Perceptron (MLP), accounting for less than 0.12% of the backbone parameters, maps a one-hot task command to a binary execution mask. This mask dictates which "tiles" (groups of 16 output channels) are needed. The network is trained using a three-phase pipeline: standard dense training, soft masking with a sparsity penalty, and hard binarized masking using a straight-through estimator.
- Neural Instruction Set Architecture (NISA): The authors design a custom ISA where each layer's instruction carries a 128-bit field dedicated to tile execution masks.
- Tiled Inference Accelerator: Implemented with an INT8 datapath on an FPGA, the accelerator's tile manager reads the NISA bitmask. If a bit is zero, the hardware advances past that tile in a single cycle, completely skipping the DMA weight fetch, activation load, and compute phases. Crucially, this sparsity propagates across layers: skipping an output tile in layer $L$ automatically eliminates the corresponding input tile computation in layer $L+1$.
Datasets
The system was evaluated on a closed-loop visuomotor driving task within the CARLA autonomous driving simulator (specifically using the urban environment Town10HD_Opt).
- Training Data: A dataset of 302K frames was collected using an expert rule-based controller. To prevent the model from collapsing into a simple "drive straight" policy, the dataset was balanced by removing 90% of the redundant straight-driving frames. It covers six distinct commands: follow lane/go straight, turn left, turn right, change lane left, change lane right, and brake.
- Evaluation: Testing was performed across three challenging routes (Hard 1, Hard 2, and Hard 3) featuring multi-lane merges, tight turns, and intersections.
Key Insights and Ideas
- The Command is a "Free" Signal: Unlike typical dynamic networks that must process intermediate image activations at runtime to decide what to skip, the task command is known beforehand. Because a command like "turn left" persists for hundreds of frames, the tile mask can be resolved just once when the command changes, reducing runtime gating overhead to near zero.
- The "Polysemantic Core": To ensure the network retains critical shared representations, the authors designate "Tile 0" of every layer as always-on. The gating network only prunes the remaining tiles.
- Task Complexity Reflection: The learned masks naturally adapt to the complexity of the maneuver. "Brake"—which relies primarily on proximity detection—is pruned aggressively, leaving only 18.7% of tiles active. Conversely, "Turn Left"—which requires navigating oncoming traffic and wider arcs—retains the most capacity (31.3% active).
- Hardware Support is Mandatory: When the authors ran their sparse model on an NVIDIA RTX 4090 GPU, it was actually 22% slower than the dense baseline. At batch size 1, the GPU is bottlenecked by kernel launch overhead and still fetches all weights. This underscores why custom hardware like their tiled FPGA accelerator is required to translate tile-level sparsity into actual speedups.
Key Results
- Compute & Latency Reductions: Task-conditional sparsity reduced FLOPs by 66% to 76% across the driving commands. On an AMD/Xilinx Alveo U50 FPGA, on-device latency dropped by 51–59% (from 9.12 ms down to 3.74–4.44 ms), representing a 2.1–2.4× speedup.
- Energy Savings: On-device energy consumption per inference dropped by 51–59%, from 263 mJ down to 108–128 mJ.
- Improved Driving Quality: The sparse model maintained 100% route completion in CARLA. Surprisingly, it reduced Cross-Track Error (lateral deviation from the lane) by 21–59% compared to the dense baseline. The authors attribute this to a "task-specialization effect," where masking irrelevant tiles eliminates cross-command interference in the shared backbone.
- Transformer Generality: The method successfully generalized to a Vision Transformer (ViT-Base) backbone. By masking the MLP linear layers, the authors achieved a 40–62% per-command FLOP reduction without needing a redesigned accelerator.
Limitations and Future Work
- The Attention Bottleneck: In the Transformer evaluation, the attention mechanism was left entirely dense. This created a compute floor, limiting the ViT's FLOP reduction compared to the CNN. The authors note that integrating attention head-group pruning would lift this restriction.
- Static vs. Dynamic Environments: The current gating network conditions purely on the high-level command. A promising future direction is to incorporate input-dependent conditioning (e.g., skipping even more tiles if the camera image shows an entirely empty road).
- Scaling Up: The prototype was evaluated on a mid-range FPGA. Scaling the architecture to larger models and multi-SLR FPGA platforms remains an item for future engineering work.
Summary updated