This paper, titled "Landscape of Neural Architecture Search across sensors: how much do they differ?", explores a fundamental question in automated machine learning (AutoML): does the data source you choose change how hard it is to find the "perfect" neural network?
By applying the Fitness Landscape Footprint framework to satellite imagery, the authors provide a rare quantitative look at how different sensors—specifically radar (Sentinel-1) and optical (Sentinel-2)—shape the search space for Neural Architecture Search (NAS).
TL;DR
The researchers demonstrate that the choice of input sensor fundamentally alters the "difficulty" of the neural architecture search space. Optical data (Sentinel-2) creates a smoother, more predictable search landscape than radar data (Sentinel-1), which produces a "rugged" environment prone to local optima, making it significantly harder for search algorithms to navigate.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) has revolutionized how we design deep learning models, but it remains a "black box." We often don't know why a search strategy succeeds on one dataset but fails on another. This is particularly critical in Earth Observation (EO), where researchers frequently switch between different sensors or fuse them together.
The authors address the lack of interpretability in NAS by asking: How does the choice of sensor affect the search process? If we treat the set of all possible neural networks as a "landscape" where height represents accuracy (fitness), does switching from radar to optical data turn a gentle hill into a jagged mountain range?
Methods
The authors utilize Fitness Landscape Analysis (FLA), specifically a framework called the Fitness Landscape Footprint. This method uses eight distinct metrics to characterize the search space, including:
- Ruggedness: How much the fitness changes between neighboring architectures.
- Persistence: The likelihood that a high-performing model at an early stage of training remains high-performing later.
- Fitness Distance Correlation (FDC): The relationship between an architecture's distance from the global optimum and its performance.
They adapted the popular NASBench-101 search space, which contains roughly 423k CNN architectures. To make the analysis tractable, they developed a custom binary vector representation for these architectures, allowing them to define "neighborhoods" based on Hamming distance (how many edges/nodes differ between two models).
Datasets
The study centers on the So2Sat LCZ42 dataset, a benchmark for Local Climate Zone classification. It provides co-registered image patches from two distinct sources:
- Sentinel-1: Synthetic Aperture Radar (SAR) data.
- Sentinel-2: Multi-spectral (optical) data.
- Sensor Fusion: A stacked combination of both Sentinel-1 and Sentinel-2.
Key Insights and Ideas
- Optical is "Easier" for NAS: The search landscape for Sentinel-2 is significantly smoother and has higher overall fitness than Sentinel-1. This suggests that standard CNN architectures (like those in NASBench-101) are inherently better suited for optical data.
- The Ruggedness of Radar: Sentinel-1 produces a highly "rugged" landscape. In a random walk analysis, the fitness values for Sentinel-1 jumped erratically, indicating a landscape full of traps (local optima) that can stall search algorithms.
- Fusion Isn't a Magic Bullet: Surprisingly, simple input-level fusion (stacking both sensors) did not improve the search landscape. In fact, it slightly worsened the fitness distribution compared to using Sentinel-2 alone. This implies that to truly benefit from multi-modal data, we may need search spaces specifically designed for fusion rather than just stacking inputs.
- Training Time Flattens the Landscape: As models are trained for more epochs (moving from 36 to 108), the landscape becomes "flatter" and less rugged. This is a crucial finding for NAS researchers who use "early stopping" or short proxy tasks to speed up search; those proxy tasks might actually be presenting a much harder search problem than the full task.
Key Results
The "Footprint" radar plot (Figure 1) visually summarizes the stark differences between the sensors.
- Fitness Distribution: After 36 epochs, Sentinel-2 achieved a mean fitness ($\mu$) of 0.94 with low deviation ($\sigma=0.03$), while Sentinel-1 lagged at $\mu=0.47$ with high deviation ($\sigma=0.13$).
- Ruggedness ($\tau$): Random walk analysis showed Sentinel-1 was nearly 12 times more rugged ($\tau=17.84$) than Sentinel-2 ($\tau=1.54$).
- Persistence: Sentinel-2 showed much higher "positive persistence" (AuC = 0.14) compared to Sentinel-1 (AuC = 0.04), meaning top-performing architectures are easier to identify early on when using optical data.
Limitations and Future Work
The authors acknowledge that these findings are specific to the NASBench-101 search space and input-level fusion. It is possible that different search spaces (e.g., those including specialized layers for radar data) or more sophisticated fusion techniques (like mid-level feature fusion) would yield different landscape properties.
Future Work: The authors propose using these insights to develop "cross-sensor" search strategies. For example, one could perform an initial, cheap search on a smooth optical landscape to find candidate architectures, then refine them for a more complex radar or fused task.
Summary updated