Abstract
Recent medical image segmentation methods apply implicit neural representation (INR) to the decoder for achieving a continuous coordinate decoding to tackle the drawback of conventional discrete grid-based data representations. However, the INR-based decoder cannot well handle the feature misalignment problem brought about by the naive latent code acquisition strategy in INR. Although there exist many feature alignment works, they all adopt a progressive multi-step aligning paradigm on a discrete feature pyramid, which is incompatible with the continuous one-step characteristics of INR-based decoder, and thus fails to be the solution. Therefore, we propose Q2A, a novel one-step query-based aligning paradigm, to solve the feature misalignment problem in the INR-based decoder. Specifically, for each target coordinate, Q2A first generates several queries depicting the spatial offsets and the cell resolutions of the contextual features aligned to the coordinate, then calculates the corresponding aligned features by feeding the queries into a novel implicit fully continuous feature pyramid (FCFP), finally fuses the aligned features to predict the class distribution. In FCFP, we further propose a novel universal partition-and-aggregate strategy (P&A) to replace the naive interpolation strategy for latent code acquisition in INR, which mitigates the information loss problem that occurs when the query cell resolution is relatively large and achieves an effective feature decoding at arbitrary continuous resolution. We conduct extensive experiments on two medical datasets, i.e. Glas and Synapse, and a universal dataset, i.e. Cityscapes, and they show the superiority of the proposed Q2A.
Introduction: The Misalignment Problem in INR
Modern medical image segmentation has traditionally relied on discrete, grid-based representations. While effective, these methods are limited by the fixed resolution of the underlying grid. To overcome this, researchers have recently turned to Implicit Neural Representations (INR), which model images as continuous functions. By mapping coordinates to signals, INR-based decoders allow for smooth, high-resolution predictions regardless of the input size.
However, a significant challenge remains: feature misalignment. When an INR decoder retrieves "latent codes" (features) from multi-scale feature maps using standard interpolation, it often captures mismatched context. A coordinate might represent a foreground object in a high-resolution map but background in a low-resolution map due to slight spatial shifts. Existing solutions typically use progressive, layer-by-layer alignment, which is fundamentally incompatible with the one-step, direct decoding nature of INR. This paper introduces Q2A (Querying to Align), a novel one-step paradigm designed to bridge this gap.
The Q2A Framework
The Q2A framework addresses misalignment through a query-based system that maintains the continuous benefits of INR while ensuring spatial accuracy across scales.
1. Query Generator
For any target coordinate, the Query Generator produces $K$ "quadruple queries." These queries encapsulate not only the spatial location but also the spatial offsets and cell resolutions (the size of the area being queried) required to align contextual features accurately.
2. Implicit Fully Continuous Feature Pyramid (FCFP)
Standard feature pyramids are restricted to fixed scales. In contrast, the FCFP can decode features at any arbitrary continuous resolution, providing the flexibility needed for high-fidelity segmentation across different scales.
3. Partition-and-Aggregate (P&A) Strategy
The core innovation of Q2A is the Partition-and-Aggregate (P&A) strategy. Standard INR decoders use point interpolation at the center of a query, which leads to significant information loss if the query area is larger than the feature map's pixels.
P&A solves this by dividing the query cell into $s \times s$ subcells. Each subcell is interpolated individually, and the results are aggregated. This approach acts like standard interpolation at fine resolutions but shifts to an adaptive pooling-like behavior at coarser resolutions, preventing information loss.
To handle coordinates effectively, the authors employ a voting-based aggregation function. This function projects high-dimensional feature distances back to the 2D plane, ensuring the final coordinate is representative of the most relevant features in the local neighborhood.
Experimental Results
The Q2A model was evaluated against state-of-the-art (SOTA) methods on three benchmarks: Glas (histology), Synapse (clinical CT), and Cityscapes (urban scenes).
- Accuracy: On the Glas dataset, Q2A achieved a 93.29% Dice score, outperforming the previous best alignment method, AlignSeg (92.31%). On the Synapse multi-organ dataset, it reached 78.99% Dice, surpassing several Transformer-based models like TransUNet.
- Efficiency: Q2A is remarkably lean. On the Glas dataset, it utilizes only 11.49M parameters and 18.44 GFLOPs. In comparison, SFNet requires 22.40M parameters and 51.51 GFLOPs for lower performance.
- Qualitative Performance: Visualizations demonstrate that Q2A effectively reduces false positives and negatives at object boundaries, where contextual mismatch is typically most severe.
Discussion and Conclusion
Q2A represents a significant step forward in continuous coordinate-based decoding for medical image segmentation. By shifting from progressive alignment to a one-step query-based paradigm, it resolves the long-standing issue of feature misalignment in INR decoders.
While the current implementation uses a stop_grad function in the voting mechanism to manage training costs, ablation studies suggest that removing this constraint could further improve performance (reaching 93.85% on Glas). Future work may extend this one-step alignment paradigm to other dense prediction tasks, such as depth estimation or optical flow, where continuous representations offer similar advantages.
Summary updated