TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation

Huaiyuan Peng, Jisheng Dang, Wencan Zhang, Jingwen Zhao, Hong Peng, Wenjun Ma, Bin Hu, Qi Tian, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

TopoDPO introduces a spatially dense approach to preference alignment for aerial referring segmentation, moving beyond standard image-level objectives that often ignore fine-grained structural details. By mining preference pairs based on topological priors and applying a pixel-wise penalty, the framework significantly reduces connectivity errors (Betti-0) and improves geometric fidelity in complex remote sensing imagery.

What Problem Does This Paper Address?

Referring segmentation—the task of mapping a natural language query to specific pixels in an image—is particularly challenging in aerial photography. Targets like roads, narrow water channels, or building complexes are often small, elongated, or fragmented by shadows and occlusions.

The authors identify a "granularity mismatch" in how current models are aligned with human preferences. Standard Direct Preference Optimization (DPO) typically calculates a single reward for an entire image. This leads to global-margin coupling: a model might get a "positive" reward because most pixels are correct, effectively "canceling out" the penalty for a critical structural failure, such as a broken bridge in a road network. To a standard IoU metric, a few missing pixels don't matter; to a navigator, that broken connection is a total failure.

Methods

The TopoDPO framework consists of two main stages:

  1. Topo-Prior (Offline Mining): Instead of relying on human labeling, the authors automatically mine "chosen" (ground truth) and "rejected" (model prediction) pairs from a supervised fine-tuning (SFT) model. They focus on the "IoU Golden Zone" (0.40 to 0.95 IoU) to find samples that are semantically correct but structurally flawed. They use two "oracles" to detect these flaws:
  2. Connectivity Oracle: Uses Betti-0 discrepancy (counting connected components) to find fragmentation or erroneous fusions.
  3. Geometric Oracle: Uses the Douglas-Peucker algorithm to simplify contours and calculate "macro-corner recall," identifying where the model rounds off sharp polygonal corners.
  4. Dense Preference Objective: This is the core mathematical shift. While vanilla DPO averages spatial margins before applying a nonlinear penalty (like a sigmoid), TopoDPO applies the penalty to each pixel's margin before averaging.
Fig. 1. Image-level DPO compresses a spatial preference signal into one scalar, whereas TopoDPO applies the nonlinear preference penalty to the spatial margin before aggregation. The masks illustrate representative localized structural errors addressed by the proposed dense alignment objective.

By moving the nonlinearity inside the spatial sum, the model can no longer "hide" local structural errors behind a sea of easy, correct pixels. The authors prove via Jensen's inequality that this dense objective is strictly more sensitive to spatial heterogeneity.

The TopoDPO Pipeline

Datasets

The models were trained and evaluated using:

  • Aerial-D & RISORS: Large-scale datasets for aerial referring segmentation.
  • EarthVQA: Used to ensure the model retains general multimodal reasoning capabilities.
  • Topo-RS-Hard: A new "stress set" introduced by the authors, consisting of 4,590 instances specifically filtered for high structural complexity (non-compactness and topological errors).

Key Insights and Ideas

  • Aggregation Order is Critical: The most striking insight is that simply changing when you average your loss across an image can fundamentally change what the model learns. Localized errors require localized penalties.
  • Modular Topology: Most topology-aware losses require complex, differentiable surrogates (like persistent homology) that are computationally expensive during training. TopoDPO avoids this by using discrete topological measures offline for data selection, while the online training objective remains a simple, efficient pixel-wise logit comparison.
  • The "Golden Zone" Matters: Training on samples where the model is completely lost (<0.4 IoU) or nearly perfect (>0.95 IoU) provides poor signal for structural refinement. Focusing on the middle ground is where the most informative "preference" signals live.

Key Results

On the Topo-RS-Hard benchmark, TopoDPO outperformed vanilla DPO and several specialized baselines:

  • Connectivity: Reduced Betti-0 error from 1.34 to 0.90 (a ~33% improvement in maintaining correct component counts).
  • Accuracy: Improved IoU from 50.91% to 51.24%. While the IoU gain seems modest, the qualitative results show much more coherent masks for roads and buildings.
  • Geometry: Improved macro-corner recall, showing better preservation of artificial structures' sharp edges.
  • Generalization: The model maintained a 72.30% accuracy on EarthVQA, proving that structural alignment doesn't degrade general vision-language performance.

Limitations and Future Work

  • Betti-0 is a Proxy: While Betti-0 counts components, it doesn't guarantee "route connectivity" (e.g., two masks could have the same number of components but different internal pathing).
  • Corner Sensitivity: The macro-corner recall metric rewards finding annotated corners but doesn't explicitly penalize the model for adding extra spurious corners.
  • Domain Specificity: The current structural oracles (like Douglas-Peucker for corners) are highly tuned for man-made structures in aerial views. Their utility in organic medical imaging or general natural scenes remains an open question.

The authors suggest that future work could extend these dense objectives to other "granularity-mismatched" tasks, such as video preference alignment or complex document layout analysis.

Summary updated

Abstract

Preference optimization for multimodal segmentation faces a granularity mismatch: dense masks contain spatially heterogeneous preference signals, whereas image-level objectives assign one shared nonlinear weight to all spatial margins. We introduce TopoDPO, a topology-aware dense preference alignment framework for structure-sensitive aerial referring segmentation. Topo-Prior mines informative preference pairs using connected-component discrepancy and macro-corner recall, while the dense objective applies the preference penalty to each local margin before spatial aggregation against a frozen supervised reference. On Topo-RS-Hard, a 4590-instance structural stress set, TopoDPO improves intersection over union (IoU) from 50.91% to 51.24% over image-level direct preference optimization (DPO) and reduces Betti-0 error from 1.34 to 0.90; paired image-cluster bootstrap intervals against image-level DPO exclude zero for all five reported metrics. Matched supervised controls and a complete factorial analysis separate the effects of topology-aware pair mining, localized correction, and dense reference-relative preference weighting. An unfiltered stratification over all 39,244 validation records further shows the clearest structural gains on samples with moderate or severe pre-alignment component-count errors. The code is publicly available at: https://github.com/phy709/TopoDPO.
BibTeX
@misc{peng2026topodpo,
  author = {Huaiyuan Peng and Jisheng Dang and Wencan Zhang and Jingwen Zhao and Hong Peng and Wenjun Ma and Bin Hu and Qi Tian and Tat-Seng Chua},
  title = {{TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation}},
  year = {2026},
  eprint = {2611.01551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01551},
}
Figure