Gradient-Based Optimization for Discrete Hardware Metrics

This direction is merged into another direction: hardware-aware neural architecture search. Its last version stays readable and citable here.

Version 1 of 127 papers33 conceptsfrozen 2026-10-04

Modern deep learning lives in two conflicting worlds. In the world of training, everything is smooth and differentiable; we navigate the landscape of loss functions using the precise compass of backpropagation. In the world of deployment, however, the landscape is "spiky" and discrete. Hardware metrics like latency, energy consumption, and peak memory usage do not have gradients. You cannot ask a GPU, "How much would my latency change if I increased this layer's width by 0.001?"

This "differentiability gap" is the primary obstacle for Hardware-Aware Neural Architecture Search (NAS). If we cannot differentiate through the hardware, we cannot use efficient gradient-based methods like DARTS to find models that fit strict physical constraints. Researchers have developed three primary strategies to bridge this gap: modeling the hardware, steering the gradients, or bypassing them entirely.

Modeling the Hardware with Differentiable Proxies

The most intuitive way to bridge the gap is to build a "digital twin" of the hardware that is differentiable. If we can train a small neural network to predict the latency of an architecture based on its encoding, we can use that predictor as a proxy in our optimization loop.

Bowen Shi and colleagues introduced this in LA-DARTS by integrating a Latency Prediction Module (LPM). By making latency a differentiable term in the loss function, the search process can directly trade off accuracy for speed. Similarly, Qian Jiang et al. proposed EH-DNAS, which moves beyond simple lookup tables to use a differentiable neural network for end-to-end latency prediction, allowing for more complex hardware interactions to be captured during the search.

The challenge with proxies is accuracy. If the proxy is wrong, the search will exploit its flaws. Ke Li and Yu Cui address this in OptiProxy-NAS by using a re-parameterized optimization proxy that smooths the discrete architecture space, allowing gradients to flow more reliably toward efficient designs.

Steering Gradients Toward Feasibility

A second approach avoids modeling the hardware altogether. Instead of adding a hardware term to the loss function, these methods "steer" the architectural gradients based on hardware feedback.

In Constraint-guided Hardware-aware NAS through Gradient Modification, Gregory De Ruyter and his team argue that traditional penalty terms are difficult to tune. If the penalty is too low, the model violates constraints; if it's too high, accuracy collapses. Their framework, ConNAS, modifies the gradients directly: if a candidate architecture exceeds a memory budget, the gradient is projected to move the search back toward the "feasible" region.

This philosophy is shared by the HDX framework, which uses gradient manipulation to guarantee that hard constraints—like a strict 60 FPS target—are met in a single search run. By decoupling the hardware constraint from the loss function, these methods can use non-differentiable, real-world measurements (like actual on-device latency) to guide a differentiable search.

Bypassing Gradients with Zero-Order Optimization

If the hardware landscape is too rugged for gradients to be useful, some researchers choose to bypass them. Xiaoxing Wang and colleagues developed ZARTS, which replaces gradient descent with zero-order (gradient-free) optimization. This makes the search process robust to the "biased approximations" that often cause traditional differentiable NAS to collapse.

Alternatively, Bo Lyu proposed TND-NAS, which uses Reinforcement Learning (RL) to optimize architecture parameters. While RL is typically slower than gradient descent, Bo Lyu integrates it into a weight-sharing supernetwork, combining the flexibility of RL for non-differentiable objectives (like energy and parameter counts) with the speed of weight-sharing.

Beyond Search: Co-Design and Pareto Fronts

The ultimate goal of this research is not just to find one good model, but to understand the entire trade-off space. Rhea Sanjay Sukthanker and Frank Hutter introduced MODNAS, which uses a hypernetwork to find the entire Pareto front of optimal architectures across multiple devices in a single run. This allows for "zero-shot" transfer to new hardware without re-running the search.

Finally, some work suggests that we shouldn't just adapt the model to the hardware, but design them together. Hongjiang Chen et al. demonstrated this with HQNAS, which jointly optimizes architecture and bit-precision (quantization). In the most extreme cases, like LL-ViT by Shashank Nag, the model is co-designed with custom FPGA logic, replacing standard arithmetic with learnable Look-Up Tables (LUTs) to achieve massive gains in energy efficiency.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_based_optimization_for_discrete_62e0dac1,
  title = {Gradient-Based Optimization for Discrete Hardware Metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-optimization-for-discrete-hardware-metrics/rdr_62e0dac1bb3e4f7b946173c931936fb6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_based_optimization_for_discrete_62e0dac1_v1,
  title = {Gradient-Based Optimization for Discrete Hardware Metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-optimization-for-discrete-hardware-metrics/rdr_62e0dac1bb3e4f7b946173c931936fb6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}