Abstract
With the proliferation of diverse remote sensing (RS) data sources, including optical, radar, and LiDAR, the field of Earth observation (EO) has seen a surge in multi-view deep learning models. However, the lack of standardized terminology and architectural frameworks has led to fragmented research. This paper establishes a comprehensive taxonomy for deep multi-view fusion, categorizing methods by where, how, and what to fuse. Through a meta-analysis of over 100 studies, we identify feature-level fusion as a robust strategy for handling heterogeneous satellite data and highlight common practices that improve performance in real-world applications.
Introduction
Remote sensing has entered a "golden age" of data abundance. Modern EO tasks leverage a variety of sensors—optical (RGB), Synthetic Aperture Radar (SAR), LiDAR, and thermal—to capture different physical properties of the Earth's surface. Despite this bounty, the machine learning community often uses terms like "multi-modal" and "multi-sensor" interchangeably, and architectural choices are frequently made without a clear theoretical basis. This work addresses these inconsistencies by providing a unified taxonomy and identifying effective common practices for tasks such as land-cover mapping and crop yield prediction.
A Unified Taxonomy for Fusion
The proposed taxonomy is structured around three fundamental questions: where the fusion occurs, how the data is mathematically merged, and what architectural backbones are employed.
1. Where to Fuse?
Fusion strategies are primarily categorized by their position in the processing pipeline:
- Input-level (Early): Raw data channels are stacked together. This is simple but requires identical spatial resolutions across all views.
- Feature-level (Middle): Independent "view-encoders" (e.g., CNNs, RNNs) extract latent representations before merging. This offers the greatest flexibility and is often considered the optimal approach.
- Decision-level (Late): Independent models are trained for each view, and their final predictions are aggregated.
2. How to Fuse?
Mathematical merge functions range from simple operations like Concatenation and Uniform-summing to sophisticated Gated Units and Attention Mechanisms. Attention-based fusion is particularly effective as it allows the model to dynamically weigh sensors based on their reliability (e.g., ignoring cloud-obscured optical data).
3. What to Focus On?
The choice of backbone architecture is critical and should be tailored to the sensor's physics. For instance, CNNs are standard for spatial patterns, while RNNs (LSTM/GRU) handle temporal sequences, and MLPs are used for tabular metadata.
Advanced and Hybrid Strategies
Beyond the three basic levels, researchers often employ hybrid strategies to manage the extreme heterogeneity of satellite data. These include "Central-feature" strategies that use intermediate shared models and "Hybrid" strategies that combine feature-level and decision-level fusion to capture both low-level correlations and high-level consensus.
Key Insights and Empirical Trends
The meta-analysis reveals several critical findings for practitioners:
- The Feature-Level Advantage: Empirical evidence suggests that feature-level fusion is the "Goldilocks" zone, allowing models to learn cross-sensor correlations without being overwhelmed by the noise of raw input stacking.
- Heterogeneity Requires Specialization: Successful models avoid "one-size-fits-all" approaches, instead using specialized encoders (e.g., SAR-specific CNN geometries) tailored to the specific physics of the sensor.
- Auxiliary Losses as Stabilizers: Adding auxiliary tasks, such as forcing the model to reconstruct the original input view from the fused features, prevents "modality collapse" where the model ignores a specific sensor during training.
- The Curse of Dimensionality: Simply adding more views can degrade performance if labeled data is scarce, leading to over-fitting due to over-parameterization.
Challenges and Future Directions
Despite progress, several frontiers remain:
- Missing Views: Models must become more robust to missing modalities at test time, such as when a sensor fails or is obscured by clouds.
- Uncertainty Quantification: There is a need for models that output confidence levels, especially when sensors provide contradictory information (e.g., radar vs. optical disagreement).
- Explainability: Understanding why a model prioritizes one sensor over another is vital for trust in climate science and policy-making applications.
Conclusion
By standardizing the taxonomy of multi-view fusion, this work provides a roadmap for future research in remote sensing. Feature-level fusion and attention mechanisms emerge as powerful tools for navigating the complexities of heterogeneous Earth observation data, though challenges in uncertainty and explainability remain.
Summary updated