TL;DR
SPLICE-CORE is a hybrid hardware architecture for AI acceleration that combines the speed of binary arithmetic with the energy-saving "skip" logic of temporal-unary computing. By treating low-order and high-order activation bits differently, it slashes energy consumption by over 75% and improves throughput by 44% compared to traditional systolic arrays used in chips like Google's TPU.
What Problem Does This Paper Address?
Modern AI models, especially Large Language Models (LLMs) and Mixture-of-Experts (MoE) architectures, rely heavily on General Matrix Multiplication (GEMM). To make these models run faster and cooler, engineers use quantization (reducing precision to 4 or 8 bits) and sparsity (skipping multiplications by zero).
Traditional systolic arrays (the industry standard) are great for dense math but aren't very good at exploiting sparsity—they often spend energy "multiplying by zero" or processing padded data. On the other hand, temporal-unary designs are great at skipping zeros because they represent numbers as a series of pulses over time; if a number is small, it finishes early. However, these designs are usually slow because they have to "count" through every possible value.
The authors identify a specific bottleneck: low-order bits are rarely zero, making temporal counting inefficient, while high-order bits are frequently zero, making them perfect candidates for temporal skipping.
Methods
The authors introduce SPLICE-CORE, a "tub" (temporal-unary/binary) hybrid design. It splits an activation value into two parts:
- Low-order bits ($a_{lo}$): Processed using a fast, fixed-cycle binary approach.
- High-order bits ($a_{hi}$): Processed using a value-dependent temporal-unary approach.
They propose two Processing Element (PE) variants:
- Splice-BP (Bit-Plane): Updates one low-order bit plane per cycle.
- Splice-LUT (Look-Up Table): Uses a shared weight-multiple generator for each column. This allows the PE to select the correct product (e.g., $1 \times \text{weight}$, $2 \times \text{weight}$, or $3 \times \text{weight}$) and complete the low-order update in just one cycle.
Both variants share a common accumulator and inherently "gate" (turn off) the hardware when inputs are zero, saving massive amounts of switching power.
Datasets and Workloads
The researchers evaluated their design across 26 different workloads to ensure it handles real-world AI diversity:
- CNNs: MobileNetV2, MobileNetV3, and ResNet-50.
- LLMs: Llama (both prefill and decode phases) and OPT (ReLU projections).
- Mixture-of-Experts (MoE): OLMoE and Qwen1.5-MoE, tested with various batch sizes to simulate different server loads.
The evaluation covered both INT4 and INT8 precisions, synthesized in a 45nm process.
Key Insights and Ideas
- The "Splicing" Strategy: The core "aha!" moment is realizing that the "cost" of unary computing is only worth it for the bits that are likely to be zero. By "splicing" a binary phase for the noisy low-order bits and a unary phase for the sparse high-order bits, you get the best of both worlds.
- Shared Multiples: Instead of putting a full multiplier in every single PE (which is area-expensive), SPLICE-CORE generates weight multiples once per column and broadcasts them. This keeps the individual PEs lean and efficient.
- Value-Dependent Latency: In a SPLICE-CORE array, the time it takes to finish a calculation depends on the actual values being processed. If a tile contains mostly small numbers, the hardware finishes early and moves on, naturally boosting throughput.
- MoE Optimization: MoE models often produce "skinny" or sparse matrices that underutilize standard hardware. SPLICE-CORE’s ability to gate inactive rows makes it uniquely suited for the irregular workloads of modern transformer models.
Key Results
The quantitative gains over a standard binary systolic array are significant:
- Energy Efficiency: Achieved a 77.4% (Splice-BP) and 76.4% (Splice-LUT) reduction in geometric-mean tile energy across all INT4 workloads.
- Throughput: Splice-LUT increased throughput by 44.4% for INT4.
- Area: Despite the added logic, the designs are actually 9–14% smaller than a systolic array because they eliminate the need for complex general-purpose multipliers in every PE.
- INT8 Performance: Even at higher 8-bit precision, Splice-LUT reduced energy by 77.3% for ResNet-50 and 88.1% for OLMoE expert projections.
Limitations and Future Work
- Synchronized Bottlenecks: While rows with small values finish early, the entire group must wait for the row with the largest magnitude to finish before the next step. One "outlier" high-magnitude activation can limit the speedup for the whole tile.
- INT8 Scaling: While the paper shows great results for INT8, the "one-cycle" low-order update is harder to maintain as bit-widths increase without significantly increasing the complexity of the weight-multiple generators.
- Memory Hierarchy: The study focuses on the compute core. Future work could explore how these faster, more efficient cores interact with the high-bandwidth memory systems required by LLMs to ensure the compute gains aren't lost to memory bottlenecks.
Summary updated