Tailor: Altering Skip Connections for Resource-Efficient Inference

Olivia Weng, Gabriel Marcano, Vladimir Loncar, Alireza Khodamoradi, G. Abarajithan, Nojan Sheybani, Andres Meza, Farinaz Koushanfar, Kristof Denolf, Javier Mauricio Duarte, Ryan Kastner

TL;DR

Tailor is a hardware-software codesign framework that optimizes neural networks by gradually removing or shortening skip connections during retraining. By reducing the "lifetime" of data buffers, Tailor achieves up to a 34% reduction in FPGA memory resources and a 30% boost in throughput with minimal to no loss in accuracy.

What Problem Does This Paper Address?

Skip connections (identity functions that bypass one or more layers) are the secret sauce behind the success of architectures like ResNet. they solve the "vanishing gradient" problem, allowing us to train incredibly deep networks.

However, what’s good for the trainer is often a headache for the hardware. In hardware accelerators (like FPGAs), skip connections are "irregular." Because a skip connection might jump over five layers of computation, the hardware must store the input data in a buffer (FIFO) until those layers finish processing. This increases on-chip memory utilization (BRAMs) and off-chip memory bandwidth. Essentially, skip connections force the hardware to keep data alive much longer than necessary, creating a bottleneck for resource-constrained edge devices.

Methods

The authors introduce Tailor, which employs two primary strategies during a specialized retraining phase:

  1. SkipRemover: This algorithm gradually deletes skip connections one by one. To prevent the accuracy from tanking, it uses Knowledge Distillation (KD), where a "teacher" model (the original network) guides a "student" model (the one losing its connections).
  2. SkipShortener: Instead of removing the connection, this method "shortens" its lifespan. For example, a connection skipping five layers is broken into five smaller connections that each skip only one layer. This allows the data to be stored in much smaller, more efficient shift registers (Flip-Flops) rather than bulky memory blocks (BRAMs).

The "secret sauce" is the Hardware-aware Training loop. Rather than stripping the connections all at once, Tailor modifies the topology every few epochs. This gives the network time to "heal" and adapt its weights to the new, more hardware-efficient structure.

Datasets

The authors evaluated Tailor across a diverse set of tasks to prove its versatility:

  • Image Classification: CIFAR-10, CIFAR-100, SVHN, and the massive ImageNet (using ResNet-50).
  • DNA Basecalling: Oxford Nanopore Reads (using QuartzNet-5x5).
  • Automatic Speech Recognition (ASR): LibriSpeech (using QuartzNet-10x5).

Key Insights and Ideas

  • Learning vs. Inference: The paper argues that while skip connections are vital for a network to learn complex features, they aren't always necessary once the weights are fully trained.
  • The "Lifespan" of Data: The core hardware insight is that shortening a connection's lifespan allows it to stay within a single "dataflow stage." This enables a transition from using BRAM (Block RAM) to FFs (Flip-Flops), which are more abundant on FPGAs.
  • Dynamic is Better than Static: The authors show that training a "skip-less" model from scratch results in poor accuracy. However, gradually removing them from a pre-trained model via Tailor preserves performance.
  • Architecture Specificity: Tailor provides different benefits depending on the hardware style. For dataflow architectures (like hls4ml), it saves on-chip area. For 2D Processing Element (PE) arrays, it drastically reduces the need to fetch skip data from off-chip memory, directly increasing speed.

Key Results

  • Resource Savings (hls4ml): On an Alveo U200 FPGA, Tailor reduced BRAM usage by up to 34%, Flip-Flops by 13%, and Look-Up Tables (LUTs) by 16%.
  • Performance Gains (2D PE Array): For a ResNet-50 implementation, removing skip connections increased throughput (FPS) by 30% and reduced memory bandwidth requirements by 45%.
  • Accuracy Preservation: On ImageNet, SkipRemover achieved a ResNet-50 model with 75.36% accuracy—only a 0.49% drop from the baseline, while being significantly leaner.
  • Quantization Robustness: Interestingly, SkipShortener models actually outperformed traditional ResNets by 0.5% when quantized to 4-bit or 8-bit precision, suggesting that shorter connections might be more resilient to bit-width reduction.

Limitations and Future Work

  • Scalability on Deep Networks: While SkipRemover works wonders on shallower models, the authors noted that for very deep networks (like ResNet-110) or complex ASR tasks, removing all skips causes a noticeable accuracy drop. In these cases, SkipShortener is the better, albeit slightly less "efficient," choice.
  • Hybrid Approaches: The current work treats SkipRemover and SkipShortener as separate paths. Future work could explore a hybrid approach that decides which connections to remove and which to shorten based on a specific hardware cost function.
  • Extreme Cases: The authors suggest exploring "partial" removal (e.g., removing only half the connections) to find a finer-grained balance between accuracy and hardware cost.

Summary updated

Abstract

Deep neural networks use skip connections to improve training convergence. However, these skip connections are costly in hardware, requiring extra buffers and increasing on- and off-chip memory utilization and bandwidth requirements. In this article, we show that skip connections can be optimized for hardware when tackled with a hardware-software codesign approach. We argue that while a network’s skip connections are needed for the network to learn, they can later be removed or shortened to provide a more hardware-efficient implementation with minimal to no accuracy loss. We introduce Tailor, a codesign tool whose hardware-aware training algorithm gradually removes or shortens a fully trained network’s skip connections to lower the hardware cost. Tailor improves resource utilization by up to 34% for block random access memories (BRAMs), 13% for flip-flops (FFs), and 16% for look-up tables (LUTs) for on-chip, dataflow-style architectures. Tailor increases performance by 30% and reduces memory bandwidth by 45% for a two-dimensional processing element array architecture.
BibTeX
@article{weng2024tailor,
  author = {Olivia Weng and Gabriel Marcano and Vladimir Loncar and Alireza Khodamoradi and G. Abarajithan and Nojan Sheybani and Andres Meza and Farinaz Koushanfar and Kristof Denolf and Javier Mauricio Duarte and Ryan Kastner},
  title = {{Tailor: Altering Skip Connections for Resource-Efficient Inference}},
  journal = {ACM Trans. Reconfigurable Technol. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=7VoerGucqh},
}
Figure