Cross Layer Co Design for Deep Learning Accelerators

760 concepts471 papers16 proposals

For decades, computer systems operated under a clean separation of concerns. Software engineers designed algorithms to minimize theoretical computational complexity, assuming the underlying hardware was a black box that would execute instructions as fast as physics allowed. Meanwhile, hardware engineers designed general-purpose processors to maximize raw throughput (FLOPS) for standard mathematical operations.

As transistor scaling hits its physical limits in the post-Moore era, this wall between software and hardware has become a major bottleneck. In modern deep learning, the primary constraint on performance is no longer raw computational capacity, but rather data movement—the energy and latency cost of fetching weights and activations from memory. When optimization happens sequentially—designing a model first, and then attempting to compress or map it onto a fixed hardware architecture—the theoretical gains of the algorithm are frequently lost to inefficient hardware scheduling, memory contention, or mismatched data layouts.

To break through this performance ceiling, researchers have turned to cross-layer hardware-software co-design. The core intuition behind this approach is simple: by optimizing the mathematical structure of the model and the physical architecture of the hardware concurrently, we can eliminate the overhead of data movement and build systems that are orders of magnitude more efficient than the sum of their parts.

Merging the Search Spaces

Traditionally, finding an efficient neural network deployment involves a two-step process: run Neural Architecture Search (NAS) to find a high-accuracy model, and then use a compiler to map that model onto an accelerator. However, this sequential approach explores only a fraction of the possible design space.

True co-design merges these steps into a single, unified search loop. As Zhang et al. demonstrate in their framework for FPGA deployment, concurrently optimizing neural network hyperparameters alongside hardware parallelization factors allows the system to discover optimal trade-offs that sequential methods miss entirely. By integrating hardware resource and latency estimation directly into the architectural search loop, the resulting models are inherently tailored to the physical constraints of the target silicon.

This joint optimization strategy is highly effective across diverse hardware substrates:

  • Memristive Crossbars: Designing accelerators based on emerging non-volatile memory requires balancing model accuracy with physical constraints like device area and energy consumption. Parsa et al. address this by using a multi-objective Bayesian optimization framework that jointly tunes neural network hyperparameters and memristive hardware parameters, preventing the selection of high-accuracy models that are physically unfeasible to deploy.
  • Optical Accelerators: In light-based computing, data movement across optical circuits must be minimized to maintain throughput. Li et al. show that combining a fast analytical cost predictor with a differentiable search engine allows for the simultaneous optimization of physical hardware parameters and data mapping strategies (such as loop-tiling and data flow order), vastly outperforming manual hardware designs.

Aligning Mathematical Structure with Physical Substrates

Why does co-design yield such massive efficiency gains? The answer lies in how mathematical operations are physically executed on silicon. Hardware accelerators are highly sensitive to data locality and regularity; irregular memory access patterns can cause processing units to sit idle while waiting for data. Co-design restructures the underlying mathematics of an algorithm to mirror the physical execution patterns of the hardware.

For instance, standard network pruning often creates irregular, sparse matrices that general-purpose processors struggle to accelerate efficiently. To solve this, Ding et al. utilize block-circulant matrices to compress recurrent neural networks, directly mapping the structured mathematical compression to the parallel execution units of an FPGA.

Similarly, in ReRAM-based architectures, representing signed weights typically introduces significant hardware overhead. Wang et al. resolve this by co-designing an Alternating Direction Method of Multipliers (ADMM) structured pruning algorithm alongside fine-grained hardware sub-arrays and zero-skipping logic, ensuring that the compressed model's structure perfectly aligns with the physical hardware's ability to skip zero values.

Sequential Design:
[ Algorithmic Optimization ] ---> [ Fixed Hardware Substrate ] 
                                      (Mismatched Data Layouts & Idle Compute)

Co-Design Loop:
[ Algorithmic Structure ] <======> [ Hardware Architecture ]
  (e.g., Structured Sparsity)        (e.g., Custom Dataflows)

This principle applies equally to large-scale modern workloads like Large Language Models (LLMs) and recommendation systems. Kong et al. demonstrate that the inference latency of dynamic LLM adapters can be significantly reduced by constraining algorithmic routing frequencies to match efficient GPU kernel execution patterns. For recommendation systems, where massive embedding tables hit a "memory wall," the authors of MP-Rec optimize performance by mapping memory-intensive lookup tables to high-bandwidth memory accelerators while routing compute-intensive generators to processors with large on-chip SRAM.

Even non-standard computational paradigms can be unlocked through clever mapping. Ni et al. show that brain-inspired Hyperdimensional Computing (HDC)—which relies on hyper-wide vectors—can be efficiently deployed on commercial edge accelerators like TPUs by mathematically disguising the HDC operations as standard neural network layers to exploit mass-produced matrix-multiplication units.

Dynamic and Runtime Co-Optimization

Hardware-software co-design is not limited to static configurations at compile time; it also operates dynamically at runtime to handle unpredictable environments. Edge devices frequently experience fluctuations in available power, thermal throttling, or competing background tasks.

To maintain efficiency under these conditions, systems must co-optimize software-level model scaling with hardware-level resource management. Stamoulis et al. demonstrate that jointly designing adaptive neural network architectures alongside their internal selection logic ensures that fallback models work harmoniously within a device's real-time energy constraints.

Taking this further, Lei proposes a holistic framework that integrates software-level dynamic model scaling with hardware-level adjustments like Dynamic Voltage and Frequency Scaling (DVFS) and heterogeneous core mapping. This cross-layer visibility allows the system to navigate a significantly larger trade-off space than standalone optimizations.

In extreme deployment scenarios, such as battery-less, energy-harvesting environments where power is intermittent, static models fail. Mishra demonstrates that by integrating real-time power traces directly into the training loop—dynamically adjusting dropout and quantization—the model is prepared during training for the exact scheduling decisions and power interruptions it will face during runtime execution, preventing catastrophic drops in accuracy.

Full-Stack Toolchains and Compilers

The primary challenge of hardware-software co-design is complexity: developers must understand both high-level algorithmic abstractions and low-level hardware description languages. To bridge this gap, the field relies heavily on full-stack toolchains and specialized compilers that automate the translation between layers.

The integration of software-based training libraries with hardware compilers allows algorithmic choices to translate directly into predictable hardware performance. For example, Aarrestad highlights how the hls4ml library enables ultra-low-latency FPGA deployment by allowing developers to tune bit-precision and sparsity within standard software frameworks, which the compiler then maps directly to streaming-based hardware implementations.

To accelerate the development cycle, frameworks like the CFU Playground, introduced by Prakash et al., enable an iterative software-profiling and hardware-modification loop. Developers can identify specific software bottlenecks in real time and immediately design targeted custom hardware instructions to accelerate them.

Ultimately, these toolchains are what make next-generation hardware viable. As Agosta et al. note, the physical advantages of exotic, post-CMOS architectures—such as photonics or processing-in-memory—are entirely lost without integrated programming abstractions. Compilers must expose model optimization techniques like sparsification and quantization directly to the underlying hardware fabric, ensuring that the physical benefits of specialized hardware are fully realized by the software application.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 451 papers
  2. Version 1 · 2026-10-04 · 334 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hardware_software_co_design_hardware_sof_e3e4ba91,
  title = {hardware-software co-design (hardware-software co-optimization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-software-co-design-hardware-software-co-optimization/rdr_e3e4ba91599c4490b54e3ed656bc01cf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_hardware_software_co_design_hardware_sof_e3e4ba91_v2,
  title = {hardware-software co-design (hardware-software co-optimization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-software-co-design-hardware-software-co-optimization/rdr_e3e4ba91599c4490b54e3ed656bc01cf/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake