Abstract
Tiled spatial architectures have proved to be an effective solution to build large-scale DNN accelerators. In particular, interconnections between tiles are critical for high performance in these tile-based architectures. In this work, we identify the inefficiency of the widely used traditional on-chip networks and the opportunity of software-hardware co-design. We propose METRO with the basic idea of decoupling the traffic scheduling policies from hardware fabrics and moving them to the software level. METRO contains two modules working in synergy: METRO software scheduling framework to coordinate the traffics and METRO hardware facilities to deliver the data based on software configurations. We evaluate the co-design using different flit sizes for synthetic study, illustrating its effectiveness under various hardware resource constraints, in addition to a wide range of DNN models selected from real-world workloads. The results show that METRO achieves 56.3% communication speedup on average and up to 73.6% overall processing time reduction compared with traditional on-chip network designs.
Introduction: The NoC Bottleneck in Spatial Accelerators
As spatial accelerators—such as Google’s TPU or Cerebras’ Wafer-Scale Engine—scale to integrate thousands of cores, the Network-on-Chip (NoC) connecting these cores becomes a critical performance bottleneck. Traditional NoCs rely on hardware-based reactive scheduling. While effective for general-purpose computing, this approach suffers from "nearsightedness" and "rigidity" in the context of deep learning. Individual routers only observe local traffic, leading to global hotspots and "tree saturation," where a single blocked message causes a ripple effect of congestion, idling expensive compute resources.
The METRO Architecture
METRO addresses these inefficiencies through a holistic software-hardware co-design that offloads traffic scheduling from rigid hardware logic to a global software framework. By leveraging the deterministic nature of tensor applications, METRO plans communication patterns at compile-time.
The architecture consists of a compute core array where each core contains local buffers, functional units, and a network interface. Dataflow graphs are mapped onto this array, creating predictable communication requirements that METRO exploits to optimize routing and timing.
Software-Defined Traffic Scheduling
The core of METRO is its software framework, which "looks into the future" to optimize data movement across the chip.
1. Evolutionary Global Scheduling
METRO uses an Evolutionary Algorithm (EA) to explore the vast design space of possible paths and message priorities. This allows the system to find an optimal global schedule that avoids the hotspots common in local hardware-based routing.
2. Injection Time Control
A key innovation is the "wait at the source" philosophy. Instead of allowing packets to collide and cause backpressure within the network, the software calculates precise injection times.
As shown in the figure above, traditional backpressure leads to Head-of-Line (HOL) blocking, where a stalled packet prevents others from moving. METRO’s injection management ensures that if a path is predicted to be busy, the message waits at the source core, keeping the network channels clear for other traffic and maximizing global throughput.
Simplified Hardware Design
By moving the "intelligence" of scheduling to software, METRO enables significantly simpler and faster hardware:
- 3-Stage Pipeline: Removing the need for hardware-based Virtual Channel (VC) allocation allows the router pipeline to be shortened from 4 stages to 3, reducing latency.
- Hybrid Routing: METRO uses a mix of algorithmic routing for critical nodes and small local tables for final delivery, keeping packet headers compact.
- Chunk Granularity: Data is sent as large chunks rather than many small packets, maximizing bandwidth efficiency by reducing header overhead.
Performance and Energy Results
The researchers evaluated METRO across several multi-model serving tasks and individual benchmarks like MobileNet-v3. Key results include:
- Communication Speedup: METRO achieved a 56.3% average communication speedup.
- Overall Performance: Execution performance improved by up to 52% for traffic-intensive models, while overall processing time was reduced by up to 73.6%.
- Energy Efficiency: The system achieved a 74% reduction in communication energy and a 61% reduction in total energy compared to traditional NoC baselines.
- Utilization: Channel utilization improved by up to 8x, as the software balanced loads across previously under-utilized links.
Conclusion
METRO demonstrates that for deterministic workloads like DNNs, reactive hardware scheduling is often a waste of resources. By shifting complexity to a compile-time software framework, METRO achieves a "win-win": higher performance and lower energy consumption through hardware simplification. This approach paves the way for more efficient large-scale spatial architectures.
Summary updated