Neural network pruning has long been haunted by a "sparsity paradox." On one hand, unstructured pruning—where any individual weight can be zeroed out—retains high model accuracy even at extreme compression levels. However, because these zeroed weights are scattered randomly, standard hardware like GPUs and CPUs cannot skip them efficiently, leading to zero actual speedup. On the other hand, structured pruning—removing entire rows, columns, or filters—is perfectly compatible with hardware but often causes catastrophic drops in accuracy because it removes too much essential information.
To resolve this, researchers have converged on semi-structured sparsity. This approach seeks to find "fine-grained" patterns that are regular enough for hardware to accelerate but flexible enough to preserve the model's representational power.
The N:M Pattern: A Hardware-Native Compromise
The most prominent form of semi-structured sparsity is the N:M pattern, where for every block of $M$ consecutive weights, exactly $N$ must be non-zero. The most famous instance is the 2:4 sparsity pattern, which was baked into the hardware of NVIDIA’s Ampere GPU architecture.
As Aojun Zhou and colleagues demonstrated in Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch, this pattern provides a "sweet spot." It allows the hardware to use specialized metadata to skip zero values during matrix multiplication, effectively doubling throughput, while the $N$ weights can be chosen from any position within the $M$ block to minimize error.
However, simply forcing a model into an N:M pattern often hurts performance. To fix this, Amir Yazdanbakhsh and others introduced Pruning Mask Decay, which gradually transitions the model from "soft" weights to "hard" binary N:M masks during training. This allows the network to slowly adapt its internal representations to the rigid hardware constraints.
Blocked and Shuffled Structures
Beyond N:M patterns, researchers have explored grouping weights into small, manageable blocks. The intuition is that while a single weight is too small for hardware to track, a $4 \times 1$ or $8 \times 8$ block is large enough to trigger vectorized instructions on modern CPUs and GPUs.
In An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs, Haihao Shen and his team utilized a 4x1 structured pruning pattern. By aligning the sparsity with the register width of standard CPUs (using AVX-512 instructions), they achieved massive speedups that unstructured sparsity could never reach.
To further improve the accuracy of these blocked methods, Peng Jiang introduced the concept of shuffled blocks in Exposing and Exploiting Fine-Grained Block Structures for Fast and Accurate Sparse Training. By shuffling the indices of weights before grouping them into blocks, the algorithm can capture the high-accuracy benefits of unstructured placement while still presenting a hardware-friendly blocked interface to the execution engine.
Recovering Accuracy with Error Correction
Even with clever patterns, the act of zeroing out weights introduces "pruning noise." Recent work has focused on mathematically compensating for this loss. Alexander Liu proposed ARMOR, which factorizes a weight matrix into a sparse N:M core surrounded by very small, block-diagonal "error-correcting" matrices. This allows the model to "patch" the errors introduced by the hardware-friendly sparsity, sometimes even allowing sparse models to outperform their dense counterparts on complex reasoning tasks.
Similarly, Connor Holmes and the NxMTransformer team used the Alternating Direction Method of Multipliers (ADMM) to treat hardware constraints as a formal optimization problem. Instead of just "cutting" weights, they mathematically pull the weights toward a state that satisfies the N:M constraint while minimizing the impact on the model's output.
Why It Works: The Hardware-Software Co-Design
The success of semi-structured sparsity isn't just about better math; it's about co-design. As Geng Yuan argues in his work on Unified Network Pruning and Architecture Search, the "best" sparsity pattern depends entirely on the compiler and the target hardware.
By defining patterns like pSConv (periodic sparse kernels) as seen in work by Souvik Kundu 0002, researchers can pre-calculate memory addresses and storage offsets. This eliminates the "bookkeeping overhead" (the extra memory needed to store where the non-zero weights are) that usually makes sparse models slower than dense ones. When the hardware knows exactly where the next non-zero weight will be—because it follows a semi-structured rule—it can maintain a full pipeline, turning theoretical compression into real-world wall-clock speed.