Tailor: Altering Skip Connections for Resource-Efficient Inference

Olivia Weng, Gabriel Marcano, Vladimir Loncar, Alireza Khodamoradi, Nojan Sheybani, Farinaz Koushanfar, Kristof Denolf, Javier Mauricio Duarte, Ryan Kastner

Abstract

Skip connections are fundamental to training deep neural networks but introduce significant overhead in hardware accelerators due to increased data buffering requirements. We present Tailor, a hardware-software codesign framework that optimizes network topology by gradually removing or shortening skip connections during a specialized retraining phase. By reducing the "lifetime" of data buffers, Tailor achieves up to a 34% reduction in FPGA memory resources and a 30% boost in throughput with minimal impact on accuracy.

Introduction: The Cost of Skip Connections

Skip connections—identity functions that bypass one or more layers—are essential for training deep architectures like ResNet by mitigating the vanishing gradient problem. However, these connections pose a challenge for hardware efficiency, particularly in resource-constrained environments like FPGAs.

In a hardware accelerator, a skip connection that jumps over multiple layers requires the input data to be stored in a buffer (FIFO) until the bypassed layers finish processing. This "long-lived" data increases on-chip memory utilization (BRAMs) and off-chip memory bandwidth, creating a bottleneck for edge devices. Tailor addresses this by optimizing the network's topology to align better with hardware constraints.

Methodology: SkipRemover and SkipShortener

Tailor employs two primary strategies during a hardware-aware retraining phase to optimize skip connections:

  1. SkipRemover: This algorithm gradually deletes skip connections. To maintain accuracy, it utilizes Knowledge Distillation (KD), where a "teacher" model (the original network) guides a "student" model (the network losing its connections).
  2. SkipShortener: Instead of total removal, this method "shortens" the lifespan of a connection. For instance, a connection skipping five layers is decomposed into five smaller connections that each skip only a single layer. This allows data to be stored in efficient shift registers (Flip-Flops) rather than bulky memory blocks (BRAMs).
The SkipRemover and SkipShortener training processes. SkipRemover gradually eliminates connections, while SkipShortener breaks long skips into shorter, hardware-friendly segments over multiple epochs.

The core of Tailor is its Hardware-aware Training loop. Rather than modifying the topology abruptly, Tailor alters the structure every few epochs, allowing the network to "heal" and adapt its weights to the new, more efficient architecture.

Hardware Architectural Impact

The primary hardware insight behind Tailor is managing the "lifespan" of data. Shortening a connection's span allows it to remain within a single "dataflow stage," enabling a transition from using Block RAM (BRAM) to Flip-Flops (FFs), which are more abundant on FPGAs.

Hardware implementation differences between (a) traditional skip connections requiring large FIFOs, (b) no skip connections, and (c) shortened skip connections utilizing efficient registers.

Tailor's benefits vary by hardware style. In dataflow architectures (e.g., hls4ml), it reduces on-chip area. In 2D Processing Element (PE) arrays, it minimizes the need to fetch skip data from off-chip memory, directly increasing throughput.

Experimental Results

The authors evaluated Tailor across diverse tasks, including image classification (CIFAR, ImageNet), DNA basecalling (QuartzNet), and Automatic Speech Recognition (LibriSpeech).

  • Resource Efficiency: On an Alveo U200 FPGA, Tailor reduced BRAM usage by up to 34%, Flip-Flops by 13%, and Look-Up Tables (LUTs) by 16%.
  • Performance Gains: For ResNet-50, removing skip connections increased throughput by 30% and reduced memory bandwidth requirements by 45%.
  • Accuracy Preservation: On ImageNet, SkipRemover produced a ResNet-50 model with 75.36% accuracy—a negligible 0.49% drop from the baseline.
  • Quantization Robustness: SkipShortener models outperformed traditional ResNets by 0.5% when quantized to 4-bit or 8-bit precision, suggesting that shorter connections are more resilient to reduced bit-widths.

Conclusion and Future Work

Tailor demonstrates that while skip connections are vital for learning, they are not always necessary for inference. Gradually removing or shortening them via hardware-aware training preserves performance while significantly lowering hardware costs.

Future work may explore hybrid approaches that dynamically decide which connections to remove versus shorten based on specific hardware cost functions. Additionally, investigating partial removal in extremely deep networks could provide a finer balance between accuracy and efficiency.

Summary updated

BibTeX
@article{weng2024tailor,
  author = {Olivia Weng and Gabriel Marcano and Vladimir Loncar and Alireza Khodamoradi and Nojan Sheybani and Farinaz Koushanfar and Kristof Denolf and Javier Mauricio Duarte and Ryan Kastner},
  title = {{Tailor: Altering Skip Connections for Resource-Efficient Inference}},
  journal = {ACM Trans. Reconfigurable Technol. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=7VoerGucqh},
}
Figure