Most neural network compression algorithms—whether they prune, quantize, or binarize parameters—make a convenient but deeply flawed assumption: they treat network weights as independent, identically distributed (i.e.d.) scalars. When a standard quantizer maps a 32-bit floating-point weight to a 2-bit integer, it evaluates that weight in isolation, ignoring its neighbors.
However, a trained neural network is not a random collection of isolated parameters. Weights work in concert to extract features, meaning they possess rich structural, spatial, and statistical correlations. Ignoring these dependencies leads to a severe accumulation of quantization errors, causing model accuracy to drop precipitously at ultra-low bitwidths.
To overcome this limitation, recent research shifts the paradigm by treating weight spaces as highly structured signals. By explicitly modeling the dependencies between parameters, these methods achieve extreme compression with minimal loss in performance.
Spatial Redundancy and Filter-Level Geometry
Why does treating weights independently fail? Consider a convolutional filter. It acts as a localized spatial feature detector, such as an edge or texture sensor. If we quantize or binarize each weight independently, we distort the geometric pattern of the filter, destroying its ability to extract clean features.
To exploit this spatial cohesion, the authors of Training Binary Neural Networks through Learning with Noisy Supervision propose processing entire convolutional filters as single entities rather than independent scalars. By treating filters holistically during binarization, the model preserves the internal spatial correlations and structural context that element-wise quantization typically destroys.
Independent Quantization:
[ 0.85, 0.12, -0.43 ] --> Quantize each in isolation --> [ +1, 0, -1 ] (Pattern distorted)
Holistic/Vector Quantization:
[ 0.85, 0.12, -0.43 ] --> Evaluated as a joint vector --> Preserves directional geometry
This geometric perspective scales across layers and architectures:
- Kernel Clustering: In Clustering Convolutional Kernels to Compress Deep Neural Networks, Sanghyun Son 0002 et al. demonstrate that thousands of unique 2D spatial kernels share underlying geometric patterns. By clustering these kernels into a small codebook of learned centroids and applying transform-invariant weight sharing (such as flips and rotations), they compress networks by exploiting structural redundancy.
- Weights as Images: For Large Language Models (LLMs), Louis Leconte et al. extend this intuition in ReALLM: A general framework for LLM compression and fine-tuning by treating entire weight matrices like images. They utilize neural autoencoders and a column-shuffling technique to group outlier weights together, capturing structural textures that standard compression algorithms miss.
Weights as a Structured Signal
If weights are highly correlated, we can treat the weight space as a continuous, compressible signal rather than an unstructured black box. This allows researchers to borrow powerful tools from classical signal processing and video coding.
In image compression (like JPEG), pixels are highly correlated with their neighbors. Instead of quantizing raw pixels, engineers apply a transform (like the Discrete Cosine Transform) to decorrelate the data, isolating the core signal from high-frequency noise.
The authors of Transform Quantization for CNN Compression apply this exact principle to neural network parameters. By introducing a decorrelating transform—such as the Karhunen-Loève Transform (KLT) or an end-to-end learned transform—to the weights prior to quantization, Sean I. Young et al. eliminate statistical redundancy. This allows models to be squeezed down to 1–2 bits per weight with minimal accuracy loss.
Similarly, other signal-based approaches exploit these underlying distributions:
- Entropy Coding: In DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks, Simon Wiedemann et al. adapt Context-Based Adaptive Binary Arithmetic Coding (CABAC)—a state-of-the-art video compression technique—to deep networks. By rejecting the i.i.d. assumption and adapting to local statistical correlations within layers, they achieve massive compression ratios.
- Compressed Sensing: In SINR: Sparsity Driven Compressed Implicit Neural Representations and CoINR: Compressed Implicit Neural Representations, Dhananjaya Jayasundara et al. discover that the weights of coordinate-based networks naturally follow a Gaussian distribution. By treating these weights as a structured signal, they leverage compressed sensing and pseudo-random dictionaries to compress the network without needing to transmit heavy, unstructured parameter tables.
- Implicit Representations: This is echoed in NVRC: Neural Video Representation Compression, where Ho Man Kwan et al. show that treating weights as unstructured data leads to higher bitrates, whereas accounting for correlations across different dimensions of network layers yields superior reconstruction quality.
High-Dimensional and Cross-Layer Interactions
When a single weight is quantized, it introduces a small error. If weights are treated as independent, these errors accumulate blindly across the network, compounding until the model's output is corrupted. However, if we model the dependencies between weights, we can use the remaining unquantized parameters to compensate for the errors introduced by their neighbors.
In GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance, Jinuk Kim et al. demonstrate the importance of preserving cross-weight dependencies within output channels. Instead of simplifying the optimization Hessian matrix into a diagonal matrix—which assumes weight independence—the authors preserve the full off-diagonal interactions. This allows the quantization process to track how a change in one weight impacts its neighbors, offering a far more accurate approximation of the model's true loss surface.
Diagonal Hessian (Independence Assumption):
[ Error 1 0 0 ] --> Errors accumulate independently
[ 0 Error 2 0 ]
[ 0 0 Error 3 ]
Full Hessian (Dependency Modeling):
[ Error 1 Cov12 Cov13 ] --> Neighboring weights adjust
[ Cov21 Error 2 Cov23 ] to compensate for errors
[ Cov31 Cov32 Error 3 ]
To manage these complex interactions at scale, researchers have developed several multi-dimensional frameworks:
- Graph Neural Networks: In Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models, Wanlong Liu et al. employ Graph Neural Networks (GNNs) to model the intricate, non-linear dependencies between weights, intelligently navigating the cumulative quantization errors across ultra-low-bit LLMs.
- Vector Loss: In VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization, Cheng Gong et al. replace scalar distance minimization with a high-dimensional "Vector Loss." By decomposing weights into independent orientation and modulus optimization steps, they preserve the collective directional information of the weight vectors, maintaining feature extraction quality.
- Cross-Layer Transformations: Finally, dependencies exist not just within a single layer, but hierarchically across the network. In BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation, Geon Park et al. show that the optimal value of a weight is mathematically linked to adjacent layers. By introducing learned transformation matrices during quantization-aware training, they explicitly model intra-layer and cross-layer dependencies, ensuring that subsequent layers can effectively process the quantized signal.
Key Foundational Work
Clustering Convolutional Kernels to Compress Deep Neural Networks (2018) — contributed 1 concept elements to this direction.
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks (2019) — contributed 1 concept elements to this direction.