This paper provides a rare, direct look under the hood of the Apple Neural Engine (ANE), a ubiquitous but notoriously opaque AI accelerator. By bypassing Apple’s standard CoreML framework, the author establishes a predictive energy "roofline model" that reveals the ANE’s true performance characteristics across multiple hardware generations.
TL;DR
Researchers developed a custom tool to bypass CoreML and measure the Apple Neural Engine's performance directly, discovering it has two distinct memory bandwidths rather than one. Their new "corrected roofline" model accurately predicts when the ANE beats the GPU in energy efficiency, highlighting that operator fusion is the secret to unlocking the engine's full potential.
What Problem Does This Paper Address?
The Apple Neural Engine (ANE) is inside over a billion devices, yet it remains a "black box" for developers. Apple only allows access through CoreML, which uses opaque placement policies—deciding behind the scenes whether to run a task on the CPU, GPU, or ANE. This makes it nearly impossible for researchers to know the hardware's actual limits or energy profile.
Standard "roofline models"—which graph a processor's peak compute vs. its memory bandwidth—fail to describe the ANE. This paper addresses this gap by providing the first direct, per-engine energy accounting for the ANE, allowing engineers to predict which workloads actually belong on the engine versus the companion GPU.
Methods
The author utilized aneforge, a custom compiler and runtime that bypasses CoreML to dispatch operators directly to the ANE via Apple’s lower-level e5rt runtime. This allowed for:
- Direct Measurement: Timing arbitrary operators with controlled batch sizes and data types (fp16, int8, int4).
- Energy Accounting: Measuring power via
powermetricsrails to isolate the ANE's draw from the CPU and GPU. - Comparative Analysis: Benchmarking the ANE against the Apple GPU (using the MLX framework) and the CPU (using Accelerate/AMX) on an Apple M5 Pro SoC.
- Cross-Generation Validation: Testing the findings across M1 Max, M2 Pro, M4, and M5 Pro architectures to ensure the results weren't specific to one chip.
Datasets
The study evaluated performance using both synthetic benchmarks and real-world model architectures:
- Synthetic Primitives: Square GEMM (matrix multiplication) and 3x3 convolutions.
- Vision: ResNet-18 and ViT-B/16 (Vision Transformer).
- NLP/Encoder: MiniLM.
- LLM Decode: A TinyLlama-class model (8-layer transformer) used to test autoregressive decoding.
- Accuracy Validation: Models were scored against the ILSVRC2012 (ImageNet) validation set and Wikitext-2.
Key Insights and Ideas
- The Two-Tiered Bandwidth: The ANE doesn't have one memory speed; it has two. There is a slow "activation path" (~24 GB/s) for standalone operations and a fast "weight-stream path" (~143 GB/s) for compiled, fused programs.
- The "Falling" Compute Roof: Unlike most processors that reach a performance plateau, the ANE’s throughput for large matrix multiplications actually drops by over 50% once the operand size exceeds the engine's internal working-set window.
- Fusion is Mandatory: Standalone operations (like a single Softmax or LayerNorm) are extremely inefficient on the ANE because they fall onto the slow activation path. Fusing them into a larger graph moves them to the fast weight-stream path, where they run at negligible marginal cost.
- Energy vs. Throughput: The ANE is often the "energy winner" even when it is the "speed loser." For single-stream inference (batch=1), the ANE can be significantly more efficient than the GPU, even if the GPU has higher raw TOPS.
Key Results
- Efficiency Peaks: The ANE's compute floor is roughly 1.16 pJ/FLOP, compared to 1.66 pJ/FLOP for the GPU.
- Real-Model Gains: For a ResNet-18 workload, the ANE demonstrated a 14.6x energy advantage over a compiled GPU implementation.
- LLM Decoding: In LLM tasks, the ANE is "weight-stream bound." Using int8 compression saved ~20% energy per token at batch=1, though it didn't significantly improve latency because the bottleneck shifted to the hardware dequantizer's element rate.
- Predictive Accuracy: The proposed energy roofline model achieved a median error of only 14% near the hardware's peak performance limits.
Limitations and Future Work
- Path Limits: The author notes that the measured 24 GB/s activation ceiling might be a limit of the
aneforgedispatch path rather than a hard silicon limit of the DMA itself. - Host Overhead: At very small shapes, the energy cost is dominated by the fixed "dispatch floor" (host CPU overhead), which the basic roofline model does not account for without the added "dispatch term" introduced in Section 5.
- Thermal Variables: The measurements were taken on an actively cooled MacBook Pro; performance might differ on fanless devices (like iPhones or MacBook Airs) due to thermal throttling.
- Future Work: The author suggests that future system schedulers should place work based on arithmetic intensity and utilization rather than just operator names.
Summary updated