Block-Wise Output Reconstruction for Model Compression

35 concepts22 papers1 proposal

When compressing a deep neural network—whether through quantization, pruning, or low-rank factorization—the traditional approach minimizes the difference between the original weights $W$ and the compressed weights $\hat{W}$. This is known as weight-space optimization:

$$\min_{ \hat{W} } \|W - \hat{W}\|_2^2$$

However, neural networks are functional mappings, not static databases of numbers. A small change in a weight parameter can cause a catastrophic drop in accuracy if it amplifies an outlier activation. Conversely, a large change might have zero impact if it occurs along a redundant activation channel.

To preserve model accuracy under aggressive compression, a powerful insight has emerged: we must minimize the reconstruction error of the network's local outputs, not the weights themselves. By optimizing the compressed parameters to match the original activations over a small calibration dataset, the parameters can cooperatively adapt to absorb compression noise:

$$\min_{ \hat{W} } \|f(X; W) - f(X; \hat{W})\|_2^2$$

This core mechanism shifts the compression paradigm from parameter fidelity to functional fidelity.

Why Weight Matching Fails

As noted by Wang in Towards Accurate Post-training Network Quantization via Bit-Split and Stitching, conventional quantization treats compression as an isolated rounding or clipping exercise. This ignores how parameters interact to reconstruct the original signal. When weights are compressed in a vacuum, the optimization cannot account for the distribution of the incoming activations $X$.

In Outlier Suppression+, Wei demonstrates that optimizing for input activation error alone is insufficient because it fails to capture how errors propagate through matrix multiplications. Instead, focusing on the Mean Squared Error (MSE) of the layer's final output preserves the functional signal, ensuring that critical downstream information is maintained.

Early foundational work in convolutional networks by Wu et al. in Quantized Convolutional Neural Networks for Mobile Devices and Shoot to Know What: An Application of Deep Networks on Mobile Devices proved that prioritizing the estimation error of layer responses over weight discrepancies allows for aggressive compression with minimal accuracy drops. This principle holds true across other compression modalities, such as energy-aware pruning by Yang et al. and weight binarization by Hu et al..

The Local Proxy: Making Optimization Tractable

If global end-to-end retraining is too computationally expensive or data-intensive, how can we optimize for output reconstruction? The solution lies in using a local proxy loss.

In their work on AdaRound, Nagel et al. showed that minimizing the MSE of a single layer's pre-activations serves as a mathematically grounded substitute for minimizing the global task loss. Instead of running full backpropagation across the entire network, we can pass a small calibration dataset through the original model, capture the intermediate target activations, and optimize the compressed layer's parameters locally.

This localized strategy has been successfully adapted to modern architectures. For instance, Krishnamoorthi et al. utilized this approach in LLM-FP4: 4-Bit Floating-Point Quantized Transformers to optimize 4-bit floating-point parameters layer-by-layer. Similarly, Wang et al. applied per-layer MSE minimization to compress KV caches in CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios. By treating the full-precision outputs as a static target, local optimization allows parameters to find a flatter loss landscape that generalizes well from tiny calibration sets to real-world test data, as explored by Wei et al. in QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization.

From Layers to Blocks: Capturing Inter-Layer Dependencies

While layer-wise optimization is powerful, modern architectures like Transformers consist of tightly coupled functional units (e.g., Self-Attention followed by a Feed-Forward Network). Optimizing each matrix in isolation can miss the broader picture.

To address this, recent methods have shifted from layer-wise to block-wise reconstruction. As argued by Panda et al. in TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction, optimizing an entire transformer block simultaneously accounts for the functional dependencies and interactions between attention and feed-forward layers. This block-level perspective ensures that the quantized output closely mimics the internal activations of the original model.

Layer-Wise Optimization (Isolated):
[Linear Q] -> [Linear Q] -> [Activation] -> [Linear Q]

Block-Wise Optimization (Joint):
┌───────────────────────── Transformer Block ─────────────────────────┐
│  [Attention Q] ──> [FFN Q] ──> Joint Output Matching vs. Original   │
└─────────────────────────────────────────────────────────────────────┘

This block-wise paradigm is highly versatile across different optimization strategies:

Functional Fidelity Beyond Quantization

The philosophy of output-centric reconstruction extends far beyond quantization. Any structural modification that alters a model's weights can benefit from matching functional outputs.

For example, in low-rank approximation, Wong et al. (A3 : an Analytical Low-Rank Approximation Framework for Attention) show that minimizing the error of functional components like attention scores preserves performance better than minimizing the reconstruction error of individual weight matrices. In sparse dictionary learning, Shopkhoev et al. (COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning) highlight that standard weight-distance metrics do not correlate with actual model performance; instead, using a calibration set to minimize functional reconstruction error ensures the model's original logic is preserved.

Even when transforming dense models into sparse architectures, as seen in the MoEfication framework proposed by Bamba et al. (DOT-MoE: Differentiable Optimal Transport for MoEfication), optimizing based on final block outputs protects the integrity of the residual stream far better than matching internal activation patterns. While some methods explore integrating global metrics—such as Liu et al. (PD-Quant: Post-Training Quantization based on Prediction Difference Metric), which uses a global prediction difference metric to guide local adjustments—the foundational mechanism remains the same: treating the original model's local outputs as the ground truth target.

By shifting the optimization target from what the weights are to what the weights do, block-wise output reconstruction allows compressed parameters to collectively compensate for noise, maintain the integrity of the residual stream, and preserve the model's decision-making capacity with minimal data and compute.

Key Foundational Work

Shoot to Know What: An Application of Deep Networks on Mobile Devices (2016) — contributed 2 concept elements to this direction.
Quantized Convolutional Neural Networks for Mobile Devices (2016) — contributed 2 concept elements to this direction.
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_block_wise_reconstruction_621946f0,
  title = {block-wise reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/block-wise-reconstruction/rdr_621946f08e8a442bbd3197e46459b4b0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_block_wise_reconstruction_621946f0_v1,
  title = {block-wise reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/block-wise-reconstruction/rdr_621946f08e8a442bbd3197e46459b4b0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake