Abstract
The multi-modal classification methods based on neural architecture search (NAS-MMC) can automatically learn a satisfied classifier from a given multi-modal search space. However, as the number of multi-modal features and fusion operators increases, the complexity of search space has increased dramatically. Rapidly identifying the satisfied fusion model from this vast space is very challenging. In this paper, we propose an efficient NAS-MMC method based on an idea of shrink-and-expansion search space, called core-structure-guided neural architecture search (CSG-NAS). Specifically, an evolutionary algorithm is first used to find core structures from a shrunk space (also called core structure search space) determined by high-quality features and fusion operators. Then a local search algorithm is used to find the optimal MMC model from the expanded space determined by the discovered core structures and the rest features as well as fusion operators. Moreover, a knowledge transfer strategy is introduced to further improve the overall performance and efficiency of the entire search process. Finally, extensive experimental results demonstrate the effectiveness of our CSG-NAS, attaining the superiority of classification performance, training efficiency and model complexity, compared to state-of-the-art competitors on several public benchmark multi-modal tasks.
Introduction
Multi-modal classification (MMC) involves fusing data from diverse sources—such as video, text, and audio—to enhance prediction accuracy. While Neural Architecture Search (NAS) has automated the design of these fusion models, it faces a significant "curse of dimensionality." As the variety of features and fusion operators (e.g., concatenation, addition, multiplication) grows, the search space expands exponentially.
Existing MMC-NAS methods often struggle to navigate this vast space efficiently, frequently evaluating underperforming architectures. CSG-NAS addresses this by introducing a "shrink-and-expansion" strategy. By first identifying "core structures" from high-quality features and then expanding the search locally, it achieves state-of-the-art accuracy with significantly higher search efficiency and lower model complexity than previous methods.
Methodology
The CSG-NAS framework relies on three main components to optimize the search process:
1. Core Structure Search (CSS)
Instead of searching the entire space initially, the algorithm "shrinks" the space. It evaluates individual features and fusion operators at a low cost to identify a high-quality subset. An evolutionary algorithm (ENAS₁) then identifies "core structures"—the foundational building blocks of a high-performing fusion model—within this restricted space.
2. Core Structures-Guided Optimal Fusion Architectures Search (CSG-OFSS)
Once core structures are identified, the search space is "expanded." The algorithm performs a local search (ENAS₂) around these cores, incorporating the remaining features and operators to fine-tune the architecture into an optimal global model.
3. Adaptive Knowledge Inheritance (AKI)
To avoid the inefficiency of random weight initialization, CSG-NAS uses a "tree edit distance" to measure structural similarity between parent and offspring models. If they are sufficiently similar, the offspring inherits the parent's trained weights, allowing for faster convergence through fine-tuning.
Key Insights
- The "Core" Hypothesis: Optimal multi-modal models typically contain specific high-performing features and operators. Identifying this "core" first eliminates the need to explore millions of low-quality combinations.
- Search Space Dynamics: The "shrink-and-expansion" approach treats NAS as a two-stage optimization: first finding the right neighborhood, then locating the best architecture within it.
- Weight Inheritance: The AKI strategy demonstrates that structural similarity in NAS trees is a strong proxy for functional similarity, drastically reducing the computational cost (GPU-hours) of the search.
Experimental Results
The method was evaluated on five diverse multi-modal benchmark datasets: ChemBook-10k (CB), NUS-WIDE-128 (NUS), MM-IMDB, NTU RGB-D, and EgoGesture.
- Superior Accuracy: CSG-NAS outperformed state-of-the-art competitors (e.g., EDF, BM-NAS, DC-NAS) across all datasets, achieving 91.12% accuracy on NTU RGB-D and 95.25% on EgoGesture.
- Drastic Speedup: On the ChemBook dataset, CSG-NAS found the optimal model in 24.68 hours, a 3x speedup compared to the previous SOTA (EDF) at 78.01 hours.
- Model Efficiency: Discovered models are significantly smaller. For instance, on the NTU dataset, CSG-NAS produced a model with only 0.19M parameters, outperforming models with 2.16M parameters (MFAS).
Conclusion and Future Work
CSG-NAS provides a robust solution to the complexity of multi-modal NAS. While highly effective, the initial heuristic for shrinking the search space might overlook synergistic features that only perform well when paired. Future work could explore more complex, learnable fusion cells and extend the core-guided approach to generative multi-modal tasks or dense prediction.
Summary updated