Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection

Zhen Gao, Xiao Yin, Kexin Liang, Pedro Reviriego, Shanshan Liu, Fabrizio LombardiarXiv 2026 · arXiv

TL;DR

This paper introduces PIES (Permutation Invariance Enabled Selective Protection), a novel scheme that allows binary Transformers to run reliably on "approximate memory" with high error rates. By exploiting the mathematical property of permutation invariance, the authors group critical weights together and protect only a small fraction of the model, enabling massive power savings through aggressive voltage scaling with minimal hardware overhead.

What Problem Does This Paper Address?

Deploying massive Transformer models like BERT on resource-constrained edge devices is a major challenge due to their high memory and power demands. While binarization (compressing weights to just 1 bit) helps significantly, further power can be saved by using approximate memories—hardware that operates at lower voltages or refresh frequencies.

The catch? Lowering voltage introduces random bit-flip errors. While traditional floating-point models have "Most Significant Bits" (MSBs) that are obviously more critical to protect, binary weights are all "equal" in significance. Standard error correction codes (ECC) for the entire memory are too power-hungry, and existing selective protection schemes don't translate well to 1-bit weights. The authors set out to find a way to identify and protect only the most "critical" binary weights without the overhead of tracking their individual locations.

Methods

The researchers proposed a three-step workflow called PIES:

  1. Identification of Critical Weights: They used a Genetic Algorithm (GA) to search for the optimal protection ratio for each weight matrix across the 12 layers of BERT. This identifies which specific weights, if flipped, would cause the largest drop in model accuracy.
  2. Gathering via Permutation Invariance: This is the "secret sauce." Transformers are mathematically invariant to the order of rows and columns in their weight matrices (as long as corresponding rows/columns in connected layers are swapped together). The authors used this property to physically move the "critical" weights identified in step 1 into the first few rows or columns of each matrix.
  3. Selective Protection: Once the critical weights are "gathered" at the front of the matrix, they apply selective protection—either Triple Modular Redundancy (TMR) or a Block ECC (BECC)—only to that specific "prefix" area.
Fig. 1. From Ternary BERT to Binary BERT [21]

Datasets

The authors evaluated their approach using two popular binarized BERT variants: BinaryBERT and BiT. They tested these models on tasks from the GLUE (General Language Understanding Evaluation) benchmark:

  • QNLI (Question Natural Language Inference): Used to evaluate BinaryBERT.
  • SST-2 (Stanford Sentiment Treebank): Used to evaluate BiT.

Key Insights and Ideas

  • Binary Resilience: One surprising finding is that binarized Transformers are remarkably robust. They operate reliably even at Bit Error Rates (BER) where floating-point models would completely fail. Significant accuracy loss only starts appearing at BERs higher than 5%.
  • The Power of Permutation: The most elegant insight is using Permutation Invariance (PI) to solve the "bookkeeping" problem. Normally, protecting specific bits requires a "map" of where those bits are, which takes up memory. By reordering the matrix so the important bits are always at the start, the hardware simply needs to know to protect the first $N$ rows.
  • Layer Sensitivity: Not all layers are equal. The authors found that weights in the first fully connected layer (F1) of the Feed-Forward Network are significantly more sensitive to errors than others, as errors there can flip many positive activations to zero (due to the ReLU function).
Illustration of how permutation invariance holds across different circuits in a Transformer.

Key Results

  • Performance Recovery: At a very high Bit Error Rate of 10%, the PIES scheme recovered almost all performance loss. For example, in the BiT-W1A1 model, protecting just 5% of the critical weights brought accuracy back to near-error-free levels.
  • Overhead Efficiency: PIES-BECC (using Block ECC) achieved similar dependability to full Hamming Code protection but reduced the memory overhead from 75% down to just 7.5% for BinaryBERT.
  • Voltage Scaling Feasibility: The results confirm that it is feasible to run binary Transformers on SRAM with a BER as high as 10%, which corresponds to a power saving of roughly 56% per SRAM access compared to standard operating voltages.

Limitations and Future Work

While the paper demonstrates success with BERT-base models, the authors note that the approach relies on the specific architecture of Transformers. Future work could explore:

  • Scaling to LLMs: Testing PIES on much larger models (LLMs) with billions of parameters.
  • Non-Binary Models: Applying the "gathering" principle to 4-bit or 8-bit quantized models where MSB protection is already common.
  • Hardware Implementation: Moving from simulations to actual silicon implementation of the selective TMR/BECC controllers to measure real-world power savings.

Summary updated

Abstract

Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (https://github.com/YangZhutian/binary-transformer-prefix-repair).
BibTeX
@misc{gao2026implementing,
  author = {Zhen Gao and Xiao Yin and Kexin Liang and Pedro Reviriego and Shanshan Liu and Fabrizio Lombardi},
  title = {{Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection}},
  year = {2026},
  eprint = {2611.23358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23358},
}

Loading related content...

Figure