TL;DR
CoDeRNet is a new unified framework for remote sensing that jointly performs Change Detection (localizing changes) and Change Captioning (describing them). By treating the interaction between these tasks as a selective routing problem—transferring information only when one task needs support and the other is reliable—it achieves state-of-the-art performance, particularly in generating semantically accurate descriptions.
What Problem Does This Paper Address?
In remote sensing, understanding changes between two images taken at different times (bi-temporal) usually involves two distinct tasks: Change Detection (CD), which produces a pixel-level map of where things changed, and Change Captioning (CC), which generates a natural language sentence describing what happened.
While these tasks are complementary, they suffer from a "supervision mismatch." Detection is densely supervised at the pixel level, making it great at spatial precision but poor at global semantic reasoning. Captioning is weakly supervised at the sentence level, making it good at context but often spatially imprecise.
Existing unified models often just share features or use simple joint optimization. The authors argue this is suboptimal because not all information is equally reliable across different tasks and spatial locations. They ask: How can we selectively route information between these heterogeneous branches so they actually help each other?
Methods
The authors introduce CoDeRNet, which features a novel Confidence-Decomposed Routing (CoDeR) mechanism. Instead of a simple "gate" that lets information through, CoDeR decomposes the routing decision into two specific cues:
- Receiver-side Need ($1-C_t$): Does the target task (e.g., captioning) lack sufficient information at this specific location?
- Sender-side Reliability ($C_s$): Does the source task (e.g., detection) have high-confidence evidence to offer?
The routing mask $M$ is the product of these two: $M = (1-C_t) \cdot C_s$. Information is only transferred via a residual connection when the receiver needs help and the sender is reliable.
The architecture uses a shared backbone to extract features, followed by task-specific "interfaces." The CoDeR module operates bidirectionally between these interfaces before sending the refined features to task-specific decoders: an FPN-style decoder for detection and a Transformer-based decoder for captioning.
Datasets
The model was evaluated on two primary benchmarks for unified Change Detection & Captioning (CDC):
- LEVIR-MCI: A large-scale dataset containing over 10,000 image pairs with multi-class masks (buildings and roads) and five human-written captions per pair.
- WHU-CDC: A dataset focused on post-earthquake urban reconstruction, containing over 7,000 image pairs with binary building-change masks and descriptive captions.
Key Insights and Ideas
- Selective > Uniform: The core "aha moment" is that unconditional feature sharing can introduce noise. By explicitly modeling "need" and "reliability," the model preserves the unique strengths of each branch while filling in their gaps.
- Asymmetric Scale Requirements: The authors found that routing works best when applied at different resolutions for each task. Detection (which needs detail) benefits from captioning cues at higher resolutions, while captioning (which needs semantics) benefits from detection cues at lower, more abstract resolutions.
- Decomposition is Crucial: Ablation studies showed that the two-part routing mask (need + reliability) outperformed a single-gate approach. This proves that the model isn't just "gating" information, but actually reasoning about the relationship between the two tasks.
- Factual Grounding: Beyond standard metrics, the authors used human evaluation to show that CoDeRNet produces captions that are more "factually grounded"—meaning the words in the sentence actually match the changes visible in the predicted mask.
Key Results
CoDeRNet achieved state-of-the-art results across both datasets, showing significant gains in captioning quality while maintaining or slightly improving detection accuracy.
- LEVIR-MCI: Reached a CIDEr-D score of 141.79 and an mIoU of 87.79, outperforming the previous best unified model, MCINet.
- WHU-CDC: Achieved an IoU of 90.25 and a CIDEr-D of 163.37, showing a substantial lead in consensus-based caption quality.
Qualitatively, the model is much better at identifying specific counts and spatial relationships (e.g., "on both sides of the road") compared to baseline models that often miss small details or hallucinate objects.
Limitations and Future Work
- Asymmetric Benefits: While captioning benefits significantly from detection cues, the reverse effect (captioning helping detection) is more modest. The "semantic gap" remains harder to bridge in that direction.
- Ambiguity: The model still struggles with very subtle visual changes or linguistically ambiguous scenes where human annotators might disagree.
- Future Directions: The authors suggest exploring explicit region-language alignment (e.g., using CLIP-like losses) to further strengthen the link between specific pixels and the words used to describe them.
Summary updated