Jack Unit: An Area- and Energy-Efficient Multiply-Accumulate (MAC) Unit Supporting Diverse Data Formats

Seock-Hwan Noh, Sungju Kim, Seohyun Kim, Daehoon Kim, Jaeha Kung, Yeseong KimIslped 2025

Abstract

Modern AI workloads demand support for a wide array of data formats, including bfloat16, INT8, and emerging microscaling (MX) formats. Traditional hardware accelerators often employ dedicated compute units for each format, resulting in significant silicon area waste and power leakage. This paper presents the Jack Unit, a reconfigurable Multiply-Accumulate (MAC) unit that shares hardware resources across diverse formats. By integrating significand alignment within the multiplier and leveraging 2D sub-word parallelism, the Jack Unit achieves up to a 2x reduction in area and a 5.4x boost in energy efficiency over conventional designs.

Introduction

The landscape of AI data formats is increasingly fragmented. While high-precision formats like bfloat16 are essential for training, low-precision formats like INT8 and INT4 are preferred for efficient edge inference. Furthermore, new microscaling (MX) formats are gaining traction for their ability to balance range and precision.

The "brute force" approach of packing separate, dedicated compute units for every format onto a chip is inefficient. When a model operates in INT8 mode, the floating-point hardware remains idle, consuming valuable area and power. The Jack Unit addresses this by providing a single, flexible MAC unit capable of on-the-fly reconfiguration.

Figure 1: Comparison of area and power consumption for different operations and the concept of a flexible MAC unit.

Architectural Innovations

The Jack Unit's efficiency stems from three core architectural strategies:

1. Precision-Scalable Carry-Save Multiplier (CSM)

The unit utilizes a modular multiplier design that can function as a single 8x8-bit multiplier (for bfloat16) or be fractured into four 4x4-bit sub-multipliers. This ensures high hardware utilization regardless of the input data's bit-width.

2. In-Multiplier Significand Adjustment

A major bottleneck in floating-point addition is the need for complex shifters to align exponents. The Jack Unit innovatively moves this alignment logic inside the multiplier. By shifting the results of sub-multiplications before summation, the design replaces bulky floating-point adders with streamlined integer adder trees.

3. 2D Sub-word Parallelism

To further reduce overhead, the Jack Unit groups sub-multipliers that require identical shifts. This 2D sub-word parallelism allows multiple units to share barrel shifters, reducing the total shifter count by 75% compared to naive flexible implementations.

Figure 6: Detailed architecture of the Jack Unit showing bfloat16 accumulation steps and the use of sub-word parallelism with integer adders.

Results and Evaluation

The Jack Unit was integrated into a systolic array accelerator and evaluated against five diverse AI benchmarks, including ImageNet (ConvNeXt-T), WMT14 (BERT), and Wikitext-2 (GPT2-Small).

Compared to baseline designs similar to Google TPU or IBM RaPiD, the Jack Unit is 1.17x to 2.01x smaller and consumes 1.05x to 1.84x less power. Across the tested benchmarks, the Jack-based accelerator improved energy efficiency by 1.32x to 5.41x. Notably, when running the MXFP8 format, the unit achieved 4.98x higher energy efficiency than standard FP8 baselines.

Conclusion

The Jack Unit demonstrates that hardware-software co-design can significantly mitigate the area and power overheads of multi-format AI acceleration. While it introduces a modest 6.65% increase in inference latency due to its deep pipelining, the gains in area and energy efficiency make it a compelling choice for modern, polyglot AI hardware. Future work will focus on optimizing the normalization stage to further reduce silicon footprint.

Summary updated

BibTeX
@inproceedings{noh2025jack,
  author = {Seock-Hwan Noh and Sungju Kim and Seohyun Kim and Daehoon Kim and Jaeha Kung and Yeseong Kim},
  title = {{Jack Unit: An Area- and Energy-Efficient Multiply-Accumulate (MAC) Unit Supporting Diverse Data Formats}},
  booktitle = {Islped},
  year = {2025},
}
Figure