A Multi-Task Learning Framework with Enhanced Cross-Level Semantic Consistency for Multi-Level Land Cover Classification

Shilin Tao, Haoyu Fu, Ruiqi Yang, Leiguang WangRemote 2025 · DOI

TL;DR

This paper introduces MTL-SCH, a novel multi-task learning framework for land cover classification that tackles semantic inconsistencies between coarse and fine-level classifications. By employing a shared encoder, feature cascade, and a hierarchical loss function, MTL-SCH significantly improves both classification accuracy and cross-level semantic consistency, outperforming existing methods.

What Problem Does This Paper Address?

Land cover classification is crucial for various applications, but accurately classifying the Earth's surface is challenging due to its complex, hierarchical nature. Traditional methods often treat different levels of classification (e.g., broad categories like "forest" vs. specific types like "coniferous forest") as independent tasks. This leads to "semantic inconsistency," where a region classified as "forest" at a coarse level might be incorrectly labeled as "farmland" at a fine level, creating conflicting and unreliable results. The paper aims to bridge this gap by developing a unified framework that leverages the inherent hierarchical structure of land cover to ensure semantic consistency across all classification levels.

Methods

The core of the proposed method, MTL-SCH, is a deep multi-task learning (MTL) framework built upon an encoder-decoder architecture, specifically extending DeepLabv3+.

  • Shared Encoder: A single encoder (ResNet50 backbone with atrous convolutions) extracts hierarchical features from the input VHR imagery. This allows for efficient learning of common low-level features across tasks.
  • Task-Specific Decoders: Two decoders are employed: one for coarse-level classification and another for fine-level classification.
  • Feature Cascade: To promote information flow and consistency, the output features from the coarse-level decoder are concatenated with the high-level features from the shared encoder and fed into the fine-level decoder. This allows the fine-level task to benefit from the coarse-level semantic priors.
  • Hierarchical Loss Function (Focal Tree-Min Loss): This novel loss function explicitly models the dependencies between semantic levels. It penalizes predictions that violate the hierarchical structure (e.g., a fine-level class not belonging to its predicted coarse-level superclass), thereby enforcing semantic coherence.
  • New Evaluation Metrics: To better quantify semantic consistency, the authors introduce Semantic Alignment Deviation (SAD) and Enhancing Semantic Alignment Deviation (ESAD).

Datasets

  • GID (Global Land Cover Classification Dataset): This dataset is used for training and evaluation. It's divided into GID-5 (coarse-grained) and GID-15 (fine-grained) and features a two-layer classification system.
  • STB Dataset: This dataset, from an AI competition, is also used for experiments and includes a similar hierarchical classification structure with eight primary categories and 17 secondary subclasses.

Key Insights and Ideas

  • Hierarchical Structure is Key: The paper strongly emphasizes that treating land cover classification levels independently is a fundamental flaw. Explicitly modeling the hierarchical relationships is crucial for accurate and consistent results.
  • MTL for Semantic Consistency: Multi-task learning, when designed with hierarchical awareness, is a powerful paradigm for achieving semantic consistency across different classification granularities.
  • Loss Function Matters: The proposed Focal Tree-Min Loss is a critical component, directly enforcing hierarchical constraints and significantly improving semantic coherence beyond standard cross-entropy losses.
  • New Metrics for a New Problem: The introduction of SAD and ESAD provides a much-needed quantitative way to measure semantic consistency, which is often overlooked by traditional accuracy metrics.

Key Results

  • Improved Accuracy: MTL-SCH consistently outperforms baseline methods (STL, JO, HierU-Net) on both coarse and fine-level segmentation tasks. For example, on the GID dataset, it achieved the highest mIoU scores at both levels (82.33% coarse, 76.52% fine), representing significant gains over other methods.
  • Enhanced Semantic Consistency: The proposed method significantly reduces semantic ambiguity. On the STB dataset, MTL-SCH achieved reductions of 9.77% in SAD and 5.09% in ESAD compared to the DeepLabv3+ baseline. Visual results (Figure 6, 7, 8, 9, 11) clearly show fewer label conflicts and more coherent predictions.
  • Generalizability: MTL-SCH demonstrates broad applicability by achieving performance improvements when integrated with various backbone architectures (CNN, Transformer, etc.) and network designs (SegFormer, UNetFormer, etc.).
  • Efficiency: While some MTL-SCH variants have slightly more parameters than STL, the single forward pass for both tasks offers improved inference efficiency.

Limitations and Future Work

  • Backbone Dependency: The model's performance is still influenced by the quality of the underlying backbone architecture. For instance, some artifacts (jagged edges) persist, suggesting that low-level feature extraction limitations can still impact results. Future work could explore structure-aware modules or lightweight hierarchical regularization.
  • Trade-off Between Consistency and Accuracy: There's a potential trade-off where enforcing strict hierarchical consistency might sometimes override pixel-level precision, leading to misclassifications even if semantically consistent. Future research could explore adaptive loss weighting mechanisms to balance these aspects.
  • Parameter Count: Some MTL-SCH adaptations might increase parameter counts, potentially elevating memory and computational demands. Future studies could focus on plug-and-play versions or lightweight modules.
  • Architectural Modifications: Integrating MTL-SCH into existing architectures often requires structural changes, which could limit ease of deployment.

Summary updated

Abstract

The multi-scale characteristics of remote sensing imagery have an inherent correspondence with the hierarchical structure of land cover classification systems, providing a theoretical foundation for multi-level land cover classification. However, most existing methods treat classification tasks at different semantic levels as independent processes, overlooking the semantic relationships among these levels, which leads to semantic inconsistencies and structural conflicts in classification results. We addressed this issue with a deep multi-task learning (MTL) framework, named MTL-SCH, which enables collaborative classification across multiple semantic levels. MTL-SCH employs a shared encoder combined with a feature cascade mechanism to boost information sharing and collaborative optimization between two levels. A hierarchical loss function is also embedded that explicitly models the semantic dependencies between levels, enhancing semantic consistency across levels. Two new evaluation metrics, namely Semantic Alignment Deviation (SAD) and Enhancing Semantic Alignment Deviation (ESAD), are also proposed to quantify the improvement of MTL-SCH in semantic consistency. In the experimental section, MTL-SCH is applied to different network models, including CNN, Transformer, and CNN-Transformer models. The results indicate that MTL-SCH improves classification accuracy in coarse- and fine-level segmentation tasks, significantly enhancing semantic consistency across levels and outperforming traditional flat segmentation methods.
BibTeX
@inproceedings{tao2025multi,
  author = {Shilin Tao and Haoyu Fu and Ruiqi Yang and Leiguang Wang},
  title = {{A Multi-Task Learning Framework with Enhanced Cross-Level Semantic Consistency for Multi-Level Land Cover Classification}},
  booktitle = {Remote},
  year = {2025},
  doi = {10.3390/rs17142442},
}
Figure