Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention

Saad Wazir, Daeyoung Kim

Abstract

Segmenting biomarkers in medical images is a critical task for diagnostics and biotech research, yet current methods often struggle with variations in staining and morphology. While recent state-of-the-art (SOTA) approaches focus on scaling pre-trained encoders, this paper identifies the decoder as a significant bottleneck. We propose the Multiscale Convolution Attention with Depth-to-Space (MCADS) decoder, a novel architecture that effectively integrates multiscale features, emphasizes critical regions through attention, and utilizes lossless spatial reconstruction. Our method achieves absolute IoU gains of up to 4.03% over existing SOTA models across four diverse datasets, demonstrating superior efficiency and generalization, even in zero-shot scenarios.

Introduction: The Decoder Bottleneck

In medical image segmentation, the effective transfer of rich multiscale features from encoders to decoders is paramount. However, standard architectures often rely on lossy upsampling techniques (like bilinear interpolation) and simple skip connections that fail to refine features before integration. Furthermore, while Transformers excel at capturing long-range dependencies, they frequently overlook the fine-grained local details necessary for precise boundary delineation in biomarkers like nuclei or mitochondria.

This research addresses these limitations by rethinking decoder design. We argue that optimizing the decoder's ability to reconstruct spatial dimensions and refine skip-connection features is a more efficient path to high-accuracy segmentation than simply increasing encoder size.

Methodology: The MCADS Architecture

The proposed MCADS decoder introduces three specialized components designed to preserve information and focus attention where it matters most:

  1. Depth-to-Space Upsampling Block (DSUB): Standard upsampling "guesses" pixel values. DSUB instead uses a pixel-shuffle technique, rearranging information from the channel dimension into the spatial dimension to preserve the rich feature maps learned by the encoder.
  2. Residual Linear Attention Block (RLAB): Skip connections often introduce noise. RLAB uses an iterative Residual Block to clean encoder features before applying linear attention, ensuring the decoder focuses on relevant spatial regions.
  3. Channel and Spatial Attention Block (CASAB): This module combines multiple pooling operations (Max, Mean, Min, and Sum) to capture both global context and local prominence, selectively emphasizing important channels and regions.
The MCADS Decoder architecture and its constituent sub-blocks, including DSUB, RLAB, and CASAB.

Experimental Results

The model was evaluated on four public datasets: MoNuSeg (multi-organ nuclei), DSB (2018 Data Science Bowl), Electron Microscopy (mitochondria), and TNBC (Triple-negative Breast Cancer).

Quantitative Performance

MCADS consistently outperformed existing SOTA models across all metrics. Specifically, it achieved absolute IoU gains of 2.76% on MoNuSeg, 3.12% on DSB, and 2.87% on EM. Notably, on the TNBC dataset—used for zero-shot evaluation—the model achieved a 4.03% gain, proving its ability to generalize to unseen morphologies and staining conditions without retraining.

Qualitative Analysis

Visual comparisons demonstrate that MCADS produces sharper boundaries and fewer false positives than competing models like TransUNet or EMCAD. By treating upsampling as a rearrangement of data rather than an estimation, the model retains the structural integrity of small biomarkers.

Qualitative comparison of segmentation results across MoNuSeg, DSB, EM, and TNBC datasets, showing MCADS (Ours) superior alignment with Ground Truth (GT).

Conclusion

The MCADS architecture demonstrates that sophisticated decoder design can significantly enhance segmentation accuracy without the massive computational overhead of giant Vision Transformers. By prioritizing lossless upsampling through Depth-to-Space restoration and active refinement of skip connections via Residual Linear Attention, we provide a robust framework for high-precision biomarker segmentation in clinical and research settings.

Summary updated

Abstract

Segmenting biomarkers in medical images is crucial for various biotech applications. Despite advances, Transformer and CNN based methods often struggle with variations in staining and morphology, limiting feature extraction. In medical image segmentation, where datasets often have limited sample availability, recent state-of-the-art (SOTA) methods achieve higher accuracy by leveraging pre-trained encoders, whereas end-to-end methods tend to underperform. This is due to challenges in effectively transferring rich multiscale features from encoders to decoders, as well as limitations in decoder efficiency. To address these issues, we propose an architecture that captures multi-scale local and global contextual information and a novel decoder design, which effectively integrates features from the encoder, emphasizes important channels and regions, and reconstructs spatial dimensions to enhance segmentation accuracy. Our method, compatible with various encoders, outperforms SOTA methods, as demonstrated by experiments on four datasets and ablation studies. Specifically, our method achieves absolute performance gains of 2.76% on MoNuSeg, 3.12% on DSB, 2.87% on Electron Microscopy, and 4.03% on TNBC datasets compared to existing SOTA methods. Code: https://github.com/saadwazir/MCADS-Decoder
BibTeX
@inproceedings{wazir2025rethinking,
  author = {Saad Wazir and Daeyoung Kim},
  title = {{Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=yci3OD0NrC},
}
Figure