Landscape of Neural Architecture Search across sensors: how much do they differ ?

Kalifou René Traoré, Andrés Camero, Xiao Xiang ZhuarXiv 2022 · arXiv

This paper, titled "Landscape of Neural Architecture Search across sensors: how much do they differ?", explores a fundamental question in automated machine learning (AutoML): does the data source you choose change how hard it is to find the "perfect" neural network?

By applying the Fitness Landscape Footprint framework to satellite imagery, the authors provide a rare quantitative look at how different sensors—specifically radar (Sentinel-1) and optical (Sentinel-2)—shape the search space for Neural Architecture Search (NAS).

TL;DR

The researchers demonstrate that the choice of input sensor fundamentally alters the "difficulty" of the neural architecture search space. Optical data (Sentinel-2) creates a smoother, more predictable search landscape than radar data (Sentinel-1), which produces a "rugged" environment prone to local optima, making it significantly harder for search algorithms to navigate.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) has revolutionized how we design deep learning models, but it remains a "black box." We often don't know why a search strategy succeeds on one dataset but fails on another. This is particularly critical in Earth Observation (EO), where researchers frequently switch between different sensors or fuse them together.

The authors address the lack of interpretability in NAS by asking: How does the choice of sensor affect the search process? If we treat the set of all possible neural networks as a "landscape" where height represents accuracy (fitness), does switching from radar to optical data turn a gentle hill into a jagged mountain range?

Methods

The authors utilize Fitness Landscape Analysis (FLA), specifically a framework called the Fitness Landscape Footprint. This method uses eight distinct metrics to characterize the search space, including:

  • Ruggedness: How much the fitness changes between neighboring architectures.
  • Persistence: The likelihood that a high-performing model at an early stage of training remains high-performing later.
  • Fitness Distance Correlation (FDC): The relationship between an architecture's distance from the global optimum and its performance.

They adapted the popular NASBench-101 search space, which contains roughly 423k CNN architectures. To make the analysis tractable, they developed a custom binary vector representation for these architectures, allowing them to define "neighborhoods" based on Hamming distance (how many edges/nodes differ between two models).

Datasets

The study centers on the So2Sat LCZ42 dataset, a benchmark for Local Climate Zone classification. It provides co-registered image patches from two distinct sources:

  1. Sentinel-1: Synthetic Aperture Radar (SAR) data.
  2. Sentinel-2: Multi-spectral (optical) data.
  3. Sensor Fusion: A stacked combination of both Sentinel-1 and Sentinel-2.
Sample images from the So2Sat LCZ42 dataset showing both SAR and Multi-spectral patches.

Key Insights and Ideas

  • Optical is "Easier" for NAS: The search landscape for Sentinel-2 is significantly smoother and has higher overall fitness than Sentinel-1. This suggests that standard CNN architectures (like those in NASBench-101) are inherently better suited for optical data.
  • The Ruggedness of Radar: Sentinel-1 produces a highly "rugged" landscape. In a random walk analysis, the fitness values for Sentinel-1 jumped erratically, indicating a landscape full of traps (local optima) that can stall search algorithms.
  • Fusion Isn't a Magic Bullet: Surprisingly, simple input-level fusion (stacking both sensors) did not improve the search landscape. In fact, it slightly worsened the fitness distribution compared to using Sentinel-2 alone. This implies that to truly benefit from multi-modal data, we may need search spaces specifically designed for fusion rather than just stacking inputs.
  • Training Time Flattens the Landscape: As models are trained for more epochs (moving from 36 to 108), the landscape becomes "flatter" and less rugged. This is a crucial finding for NAS researchers who use "early stopping" or short proxy tasks to speed up search; those proxy tasks might actually be presenting a much harder search problem than the full task.

Key Results

The "Footprint" radar plot (Figure 1) visually summarizes the stark differences between the sensors.

The Fitness Landscape Footprint comparing Sentinel-1 (blue), Sentinel-2 (yellow), and Fusion (green).
  • Fitness Distribution: After 36 epochs, Sentinel-2 achieved a mean fitness ($\mu$) of 0.94 with low deviation ($\sigma=0.03$), while Sentinel-1 lagged at $\mu=0.47$ with high deviation ($\sigma=0.13$).
  • Ruggedness ($\tau$): Random walk analysis showed Sentinel-1 was nearly 12 times more rugged ($\tau=17.84$) than Sentinel-2 ($\tau=1.54$).
  • Persistence: Sentinel-2 showed much higher "positive persistence" (AuC = 0.14) compared to Sentinel-1 (AuC = 0.04), meaning top-performing architectures are easier to identify early on when using optical data.

Limitations and Future Work

The authors acknowledge that these findings are specific to the NASBench-101 search space and input-level fusion. It is possible that different search spaces (e.g., those including specialized layers for radar data) or more sophisticated fusion techniques (like mid-level feature fusion) would yield different landscape properties.

Future Work: The authors propose using these insights to develop "cross-sensor" search strategies. For example, one could perform an initial, cheap search on a smooth optical landscape to find candidate architectures, then refine them for a more complex radar or fused task.

Summary updated

Abstract

With the rapid rise of neural architecture search, the ability to understand its complexity from the perspective of a search algorithm is desirable. Recently, Traoré et al. have proposed the framework of Fitness Landscape Footprint to help describe and compare neural architecture search problems. It attempts at describing why a search strategy might be successful, struggle or fail on a target task. Our study leverages this methodology in the context of searching across sensors, including sensor data fusion. In particular, we apply the Fitness Landscape Footprint to the real-world image classification problem of So2Sat LCZ42, in order to identify the most beneficial sensor to our neural network hyper-parameter optimization problem. From the perspective of distributions of fitness, our findings indicate a similar behaviour of the search space for all sensors: the longer the training time, the larger the overall fitness, and more flatness in the landscapes (less ruggedness and deviation). Regarding sensors, the better the fitness they enable (Sentinel-2), the better the search trajectories (smoother, higher persistence). Results also indicate very similar search behaviour for sensors that can be decently fitted by the search space (Sentinel-2 and fusion).
BibTeX
@misc{traore2022landscape,
  author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
  title = {{Landscape of Neural Architecture Search across sensors: how much do they differ ?}},
  year = {2022},
  eprint = {2201.06321},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.06321},
}
Figure