Recent work shows that letting a network learn which earlier layers feed each later layer, instead of fixing a chain of residual shortcuts by hand, speeds up training substantially. Those learned shortcuts are treated as free, but they are only free on a GPU, where every intermediate result is written to memory anyway. On a spatial dataflow chip the cost is real. On such a chip each layer is pinned to its own patch of processing elements, and activations stream from patch to patch without returning to main memory. A shortcut from layer $i$ to layer $j$ then has to be held in on-chip memory for as long as it takes the stream to pass through every layer in between, and it has to be physically routed across the chip. Dense learned connectivity therefore turns into a buffer bill that grows roughly with the cube of depth, and it also congests the wiring that spatial compilers already struggle with.
This note proposes to make the learned-connectivity method aware of that bill. During training we add a closed-form estimate of the on-chip buffer and routing cost of every active shortcut. The network can keep long-range connections only where they pay for themselves in convergence, and the resulting topology still fits in a single on-chip pipeline. The claim is specific and checkable. Most of the training-speed benefit of learned shortcuts can be kept while the shortcut memory footprint drops by an order of magnitude or more relative to naive deployment. A cheap physical cost term should do better here than either post-hoc pruning or a hand-picked locality rule.
Background and the gap
The connectivity-learning side comes from ANCRe. It gives every forward shortcut a learnable scalar weight, normalizes the incoming weights at each layer with a softmax, and reports 34.3% fewer iterations to target perplexity on C4, a 1.85x training speedup on LLaMA-1B, and DiT-B/2 FID-50K improving from 22.41 to 20.53, all at under 1% overhead. ANCRe supplies our method, and it also motivates the problem. Its softmax weights are never exactly zero, so a literal streaming deployment would need all $L(L-1)/2$ shortcut streams. ANCRe targets the manual-design bottleneck in its narrowest and most consequential form, the hand-fixed residual chain. It solves that problem only for temporal hardware.
The hardware side comes from three sources, each playing a different role. Data Processing with FPGAs on Modern Architectures and Transformations of High-Level Synthesis Codes for High-Performance Computing describe the execution model we assume, in which pipelined stages exchange data through local line buffers and shift registers. That model is what makes shortcut buffering proportional to pipeline delay. The next two sources constrain the design. MaxEVA gets its efficiency on Versal AI Engines by restricting transfers to neighboring cores that share memory, which shows that locality is what spatial fabrics reward. Evaluating Emerging AI/ML Accelerators documents steep performance cliffs once tensors exceed on-chip SRAM, along with compiler-driven instability on the SambaNova dataflow unit. Learned Cost Model for Placement on Reconfigurable Dataflow Hardware shows that throughput on such hardware depends on graph-level placement. Its graph neural network cost model beats hand heuristics by 31–52% in mean percentage error, but it optimizes placement only for fixed, human-designed models such as BERT and GPT-2.
Existing co-design work sits nearby but answers a different question. NAAS jointly searches convolutional networks, accelerator sizing, and compiler mapping using the MAESTRO analytical model. CODEBench searches convolutional networks with ASIC parameters under a fixed output-stationary dataflow. HW-NAS-Bench provides per-device latency tables and shows that FLOPs predict latency poorly, with a Kendall correlation of 0.18 on the Edge TPU. All three search per-layer block choices for convolutional networks and estimate cost layer by layer. None of them treats inter-layer connectivity as the object being designed, and none models the delay-proportional buffering that streaming pipelines impose. AgentNAS removes much of the hand-built search space by having language-model agents write candidate modules, but it evaluates candidates on a single GPU and has no notion of hardware cost.
Core Claim
The physical cost of a shortcut on a streaming fabric has a simple, nearly closed form. Adding it to the training loss lets learned connectivity find topologies that keep most of ANCRe's convergence gain while fitting a fixed on-chip memory budget. Consider a balanced pipeline in which each stage processes one token interval per step. A shortcut from layer $i$ to layer $j$ that carries $s_i$ bytes per token must buffer about $s_i (j-i)$ bytes, because that is how many token-intervals the stream spends in flight between the two layers. With a gate $g_{ij}\in[0,1]$ marking whether the shortcut is kept, we train on
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{i<j} g_{ij}\,\big(s_i\,(j-i) + \mu\, h_{ij}(\pi)\, s_i\big),$$
where $\mathcal{L}_{\text{task}}$ is the language-modeling or diffusion loss, $\lambda$ sets the overall weight of hardware cost, $h_{ij}(\pi)$ is the number of network-on-chip hops between the tile regions that a placement $\pi$ assigns to layers $i$ and $j$, and $\mu$ converts link traffic into buffer-equivalent cost. The first term depends only on depth gap. The second depends on where layers land on the 2D grid. For example, a snake layout puts layer $i$ physically next to layer $i+k$ at fold points, so a long shortcut can be short in wire length. The gates are hard-concrete relaxations multiplied onto ANCRe's softmax weights, so connections can actually switch off. With all gates open the buffer term grows as about $L^3/6$ times the activation size. That figure is the naive-deployment reference point.
Research Program
The work has three parts. First, we build the cost estimator. Layer and stage latencies come from MAESTRO-style per-layer analysis. The hop and congestion term comes from a simple mesh model with per-link bandwidth. We then check this estimator against a small set of real placements on an AMD Versal AI Engine board, built in the shared-neighbor-memory style of MaxEVA, and against FPGA streaming builds produced with high-level synthesis. The aim is to show that the estimator ranks topologies correctly. Exact cycle accuracy is not required.
Second, we train ANCRe with the gated cost term at several values of $\lambda$. Each run traces a curve of training efficiency against shortcut memory. We alternate training with re-optimizing the placement $\pi$ by simulated annealing on the current gate pattern, which follows the spirit of graph-aware placement in the learned-cost-model paper.
Third, we analyze which shortcuts survive. We test whether the surviving set is mostly short-range, or whether a few long "highway" connections are consistently kept. The second outcome would be the interesting one, because it would mean the convergence benefit ANCRe proves theoretically comes from a small number of long edges.
Baselines
The naive reference point is the standard cascaded residual chain. It has minimal shortcut buffering and no learned connectivity, and it will be trained with the same token budget, learning-rate schedule, and model sizes. The strongest existing competitor on training efficiency is unconstrained ANCRe, which reports 34.3% fewer iterations to target perplexity on C4, a 1.85x speedup on LLaMA-1B, and FID-50K of 20.53 on DiT-B/2 (ANCRe). We will reproduce it in our codebase rather than compare against the published figures. Its hardware cost will be computed with our estimator. No reported result was found on Lacuna for ANCRe's buffer footprint or throughput on any spatial dataflow hardware, so that number is ours to establish.
The strongest cheap competitor is post-hoc pruning: train unconstrained ANCRe, keep the top-$k$ shortcuts by learned weight per layer to meet the same memory budget, and fine-tune briefly. It gets the same total number of training tokens as our method, fine-tuning included. A second cheap competitor is a hand-chosen locality rule in which shortcuts are allowed only within a window of $w$ layers, with $w$ swept.
Two ablations isolate our key ingredients. The first keeps only the depth-gap buffer term ($\mu=0$), which tests whether placement awareness adds anything. The second replaces the physical cost with a uniform per-edge L1 penalty, which tests whether the specific shape of the cost matters. For every method, $\lambda$, $k$, or $w$ gets the same tuning budget of five settings.
We count the proposal as beating these baselines if two things hold. At equal estimated on-chip shortcut memory, it must reach target perplexity in fewer tokens than post-hoc pruning and the window rule. It must also keep at least half of unconstrained ANCRe's iteration savings while using at most a tenth of ANCRe's naive shortcut buffer.
Why not just…?
Why not just train ANCRe normally and prune the weak shortcuts afterward? Learned weights measure how much a connection matters given that all other connections exist, not what it is worth per byte of buffer. A long edge with a modest weight may be irreplaceable, while a short edge with a large weight may be redundant. Pruning after training also removes connections the optimizer relied on. This is a serious threat, because ANCRe's weights may already be sparse in practice. The equal-budget comparison above decides the question directly. If pruning matches the gated training across the memory curve, we will report that the simpler recipe suffices.
Why not just restrict shortcuts to a fixed local window? That is the manual design choice the project is meant to examine. ANCRe's theory says shortcut layout can change convergence rates exponentially, so a window may cut exactly the long edges that carry the benefit. The window sweep shows this directly. If windowed models match ours at every budget, the honest conclusion is that long-range shortcuts are not worth their cost on streaming hardware, which is itself a useful negative result for chip designers.
Why not just use per-layer latency lookup tables, as hardware-aware search usually does? Tables built one layer at a time, as in HW-NAS-Bench, assign shortcuts zero cost, because shortcut cost appears only between layers. The learned-cost-model results (Learned Cost Model for Placement on Reconfigurable Dataflow Hardware) show that summed, layer-isolated estimates miss placement effects. Our depth-gap-only ablation is the closest table-like alternative that still sees shortcuts.
Why not just penalize index distance and skip placement entirely? For buffering this is exactly right, and the depth-gap term already does it. Placement matters only if routing, rather than SRAM, is the binding constraint. If the $\mu=0$ ablation performs as well as the full objective on hardware-validated throughput, we will drop the placement term and present the method as buffer-aware connectivity learning.
Proposed Experiments
Training-efficiency experiments follow ANCRe's settings so the numbers are comparable. We pretrain LLaMA-style models from 60M to 1B parameters on C4 and measure tokens to reach the baseline's final validation perplexity, and we train DiT-S/2 and DiT-B/2 on ImageNet-1K and measure FID-50K. Hardware experiments place models of up to roughly 12 layers fully on chip. On the Versal AI Engine board and an FPGA we measure sustained tokens per second, buffer bytes actually allocated, and whether the build fits in a single on-chip pipeline at all. On the same placements we also report the estimator's rank correlation with measured throughput.
The decisive checks are three. The first is the shape of the training-efficiency versus shortcut-memory curve for each method at matched budgets. The second is whether the full objective beats depth-gap-only on measured throughput, which would mean placement matters. The third is whether the cost-aware topologies keep a small, stable set of long edges across random seeds. If unconstrained ANCRe topologies cannot be built on chip at the small scales while ours can at similar perplexity, the central claim is supported. If pruning or the window rule matches us, the main result becomes the measured cost curve itself.
References
- ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
- Data Processing with FPGAs on Modern Architectures
- Transformations of High-Level Synthesis Codes for High-Performance Computing
- MaxEVA: Maximizing the Efficiency of Matrix Multiplication on Versal AI Engine
- Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs
- Learned Cost Model for Placement on Reconfigurable Dataflow Hardware
- NAAS: Neural Accelerator Architecture Search
- MAESTRO: A Data-Centric Approach to Understand Reuse, Performance, and Hardware Cost of DNN Mappings
- CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- Agentic Neural Architecture Search