Deep learning is notoriously power-hungry, largely due to the von Neumann bottleneck—the constant shuttling of weights and activations between separate memory and processing units. To break this bottleneck, researchers have increasingly turned to physical computing, where neural network operations are embedded directly into the hardware substrate. For instance, processing-in-pixel architectures (Gourav Datta et al.) and optical encoders (Pedram Pad et al.) perform the first layer of convolutional feature extraction using the physical properties of transistors or light itself before data is even digitized.
While this approach yields massive improvements in energy efficiency and latency, it introduces a profound challenge: architectural rigidity. In traditional computing, changing a neural network is as simple as updating a software file. In physical computing, the network's weights or topology are often literally baked into the silicon or glass. Once fabricated, they cannot be easily modified.
Freezing Weights into Matter
To understand why physical hardware is rigid, we must look at how weights are represented. In an in-sensor edge intelligence chip, neural network weights can be implemented through the physical dimensions of transistors within the pixel array (Md Abdullah-Al Kaiser et al.). Similarly, in optical neural networks, weights are defined by static nanophotonic components or amplitude masks (Kaiyu Cui et al.).
This physical mapping means that the front-end layers become completely non-programmable. If the deployment environment shifts—such as encountering a new dataset or requiring a different feature extraction strategy—the hardware cannot adapt. As Minho Choi et al. point out, this lack of transferability severely restricts the utility of optical front-ends compared to flexible digital processors, rendering the physical layers specialized for a single, permanent task. Lu Gan demonstrates a similar tension in electromagnetic neural networks: while stacked metasurfaces can process semantic communications at the speed of light, adapting these physical structures to changing real-time requirements remains a significant engineering hurdle.
Structural and Topological Lock-In
The rigidity problem extends beyond static weights to the very structure of the hardware. Many domain-specific accelerators achieve high throughput by hardwiring specific dataflows, memory access patterns, and buffer sizes. Akshay Dua et al. observe that many accelerators are tied to fixed layer dimensions, preventing them from supporting diverse model topologies without a complete redesign of the underlying logic.
When hardware is optimized for a narrow set of established models, it creates a brittle architecture. Robert Adolf et al. argue that designing hardware around a limited workload suite risks immediate obsolescence as AI models rapidly evolve. A digital accelerator optimized strictly for a specific kernel size, like those discussed by Geethan Karunaratne et al., struggles to execute complex, modern architectures that feature skip connections or branching topologies.
Algorithmic Adaptation Without Weight Updates
Because modifying physical hardware post-fabrication is either impossible or highly energy-intensive, researchers are forced to develop alternative paradigms to achieve adaptability. If you cannot change the weights, you must change how the network processes inputs or handles internal states.
One prominent strategy is activation and threshold modulation. In neuromorphic hardware where synaptic weights are fused and immutable, Kejie Zhao et al. introduce a method to adapt spiking neural networks to out-of-distribution data by dynamically adjusting the firing thresholds of neurons rather than updating the weights. At the algorithmic level, Shuaicheng Niu et al. propose adapting quantized edge models using only forward passes, shifting internal activations and tuning external prompts rather than computing gradients to alter frozen parameters.
Another path is leveraging reconfigurable fabrics like FPGAs as a middle ground, as reviewed by Yiming Li, or developing platform-independent partial reconfiguration methodologies (Jürgen Teich et al.) to dynamically swap hardware logic blocks. However, for extreme edge devices where every nanojoule counts, navigating the tension between physical efficiency and algorithmic rigidity remains one of the defining challenges of modern hardware design.