Core-Structures-Guided Multi-Modal Classification Neural Architecture Search

Pinhan Fu, Xinyan Liang, Tingjin Luo

TL;DR

CSG-NAS (Core-Structures-Guided Neural Architecture Search) drastically accelerates the discovery of high-performing multi-modal classification models. By first identifying "core structures"—high-quality combinations of features and fusion operators—and then expanding the search locally, it achieves state-of-the-art accuracy while being up to four times faster than existing search methods.

What Problem Does This Paper Address?

In the world of AI, "multi-modal" means combining different types of data—like images, text, and sensor readings—to make a single decision. While powerful, designing the "fusion" architecture (the way these modalities are mashed together) is a nightmare for human engineers because the number of possible combinations grows exponentially as you add more features.

Neural Architecture Search (NAS) was invented to automate this design process. However, standard NAS methods often struggle with "vast search spaces." They spend a massive amount of time and computational power evaluating thousands of mediocre or "underperforming" architectures before finding a winner. The authors of this paper wanted to find a way to "shrink" the search space to focus only on the most promising regions from the start.

Methods

The authors propose a two-stage "shrink-and-expansion" strategy called CSG-NAS:

  1. Core Structure Search (CSS): Instead of looking at every possible combination, the algorithm first evaluates individual features and fusion operators (like addition, concatenation, or multiplication) at a low cost. It selects the top performers to create a "shrunk" search space. An evolutionary algorithm then searches this small space to find "core structures"—the foundational building blocks of a great model.
  2. Core Structures-Guided Optimal Fusion Architectures Search (CSG-OFSS): Once the core structures are found, the search space is "expanded." The algorithm takes these high-quality cores and performs a local search, adding the remaining features and operators around them to fine-tune the final architecture.
  3. Adaptive Knowledge Inheritance (AKI): This is the "secret sauce" for speed. Usually, every time a NAS algorithm generates a new "child" model, it has to be trained from scratch. AKI uses Tree Edit Distance to measure how structurally similar a child is to its parent. If they are similar enough, the child "inherits" the parent's weights, allowing it to reach peak performance with just a little bit of fine-tuning rather than a full training cycle.

Datasets

The researchers validated CSG-NAS across five diverse multi-modal benchmarks:

  • ChemBook-10k (CB): Chemical structure image recognition.
  • NUS-WIDE-128 (NUS): Image and tag-based web data classification.
  • MM-IMDB: Multi-label film genre classification using posters (images) and plots (text).
  • NTU RGB-D: Human action recognition using video and skeleton data.
  • EgoGesture: Egocentric hand gesture recognition using RGB and Depth video.

Key Insights and Ideas

  • The "Core" Hypothesis: The authors found that optimal multi-modal models almost always contain specific high-performing substructures. By finding these "cores" first, you effectively skip the "garbage" parts of the search space.
  • Structural Similarity Matters: Weight sharing in NAS is often unstable. By using the Tree Edit Distance to gate weight inheritance, the authors ensure that weights are only shared between models that actually "look" alike, making the training process much more stable and efficient.
  • Efficiency is a Feature: Most NAS papers focus solely on accuracy. This paper highlights that by narrowing the search space, you don't just get a better model; you get it significantly faster, making NAS more accessible to researchers without massive server farms.

Key Results

CSG-NAS consistently outperformed state-of-the-art competitors (like EDF, BM-NAS, and DC-NAS) in three main areas:

  • Accuracy: It set new benchmarks on all five datasets. For example, it achieved 91.12% on NTU RGB-D and 89.20% on ChemBook-10k.
  • Search Speed: On the ChemBook dataset, CSG-NAS found the optimal model in 24.68 hours, compared to 61.88 hours for DC-NAS and 78.01 hours for EDF. On the NUS dataset, it was nearly 4x faster than the baseline.
  • Model Complexity: Despite the higher accuracy, the discovered models often had fewer parameters (e.g., 0.37M parameters on NUS vs. 0.53M for DC-NAS), making them more efficient for deployment.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Initial Evaluation Cost: The method relies on an initial "low-cost" evaluation of features and operators to define the core space. While effective, the "low-cost" definition is somewhat heuristic.
  • Threshold Sensitivity: The Adaptive Knowledge Inheritance relies on a similarity threshold ($s > 0.5$). The authors chose this based on empirical observation, but a more dynamic or learned threshold might improve results further.
  • Generalization to Non-Classification Tasks: The current work focuses on classification. Extending this "core-guided" approach to multi-modal generative tasks or regression remains an open challenge.

Summary updated

BibTeX
@inproceedings{fu2024core,
  author = {Pinhan Fu and Xinyan Liang and Tingjin Luo},
  title = {{Core-Structures-Guided Multi-Modal Classification Neural Architecture Search}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=XE4qEajoNB},
}
Figure