TL;DR
EvoNAS is a novel multi-objective evolutionary neural architecture search framework designed to discover efficient, geometry-aware models for dense 2D and 3D vision tasks. By enforcing dual-domain (spatial and frequency) knowledge distillation and isolating hardware evaluation noise, it produces a family of compact "EvoNets" that achieve Pareto-optimal accuracy-efficiency trade-offs, reducing parameter overhead by up to 88% in 3D rendering applications.
What Problem Does This Paper Address?
Deploying Large Vision Models (LVMs) on resource-constrained edge devices (like autonomous vehicles or mobile systems) is heavily restricted by their massive computational footprints and high inference latencies. While Evolutionary Neural Architecture Search (ENAS) is a powerful tool for balancing competing objectives like accuracy and speed, its practical application faces two major bottlenecks:
- Representation Collapse & Ranking Inconsistency: In weight-sharing supernets, diverse sub-architectures compete for the same parameters. This gradient interference typically favors low-frequency semantic stability over high-frequency geometric precision. As a result, a subnetwork's performance within the supernet often correlates poorly with its standalone performance after independent training, rendering the search process unreliable.
- Hardware Latency Jitter: Evaluating multiple candidate networks concurrently on shared GPUs causes CUDA kernel contention and resource fragmentation. This introduces significant noise into physical latency measurements, misleading the evolutionary algorithm's selection process.
Methods
To overcome these challenges, the authors introduce EvoNAS, which combines a hybrid search space with advanced optimization and evaluation techniques:
- Hybrid VSS-ViT Search Space: The framework designs a multi-scale encoder that integrates the linear-time computational efficiency of Vision State Space (VSS/Mamba) blocks in the early stages with the global reasoning of Vision Transformers (ViT) in the final stage. This is paired with a Spatial-Mamba decoder to preserve fine-grained structural boundaries.
- Cross-Architecture Dual-Domain Knowledge Distillation (CA-DDKD): To fix ranking inconsistency, EvoNAS anchors the supernet to a geometry-aware teacher model (Depth Anything). Crucially, it enforces alignment in both the spatial domain (via feature tokens) and the frequency domain using Discrete Cosine Transform (DCT) constraints. Forcing the supernet to preserve high-frequency spectral components prevents the erosion of sharp boundaries and stabilizes subnetwork ranking.
- Progressive Supernet Training (PST): A curriculum-based training strategy that warm-starts the search space by optimizing high-capacity configurations first before sequentially unlocking smaller dimensions, minimizing gradient interference.
- Distributed Multi-Model Parallel Evaluation (DMMPE): A hardware-isolated evaluation engine that leverages NVIDIA Multi-Process Service (MPS) and asynchronous scheduling. By pooling GPU resources and isolating execution contexts, it eliminates latency jitter and provides a clean, unbiased fitness landscape.
Datasets
The framework was evaluated across a multi-tiered benchmark suite spanning 2D, 2.5D, and 3D complexities:
- COCO: Object detection and instance segmentation.
- ADE20K: Pixel-wise semantic segmentation.
- KITTI & NYU-Depth v2: Outdoor and indoor monocular depth estimation (MDE).
- RealEstate10K: Zero-shot architectural transfer for 3D novel view synthesis via 3D Gaussian Splatting (3DGS).
Key Insights and Ideas
- The Frequency Domain is the Key to Geometry: Traditional NAS methods treat features as generic tensors, ignoring spectral properties. By explicitly supervising the frequency domain via DCT, EvoNAS ensures that even highly pruned, lightweight subnetworks retain the sharp edge and boundary information critical for 3D rendering.
- Hardware Isolation is an Algorithmic Necessity: Latency proxies (like FLOPs) are often poor stand-ins for real-world throughput. However, raw physical measurements are too noisy on shared hardware. True hardware isolation via virtualization (MPS) acts as a high-fidelity sensor that is critical for robust multi-objective Pareto optimization.
- Universal Geometric Priors Transfer to 3D: Models that successfully learn to align spatial and spectral structures for 2D dense prediction tasks possess an intrinsic geometric understanding that allows them to be seamlessly "hot-swapped" into 3D rendering pipelines without massive parameter scaling.
Key Results
- Evaluation Efficiency: The DMMPE framework improved validation throughput by over 70% compared to conventional data-parallel evaluation, achieving a threefold speedup under optimal configurations.
- 2D/2.5D Benchmarks: Discovered architectures, named EvoNets, consistently established new Pareto-optimal frontiers. For instance, EvoNet-C1 (33M parameters) achieved 45.4 AP on COCO, outperforming ResNet-50 by 7.2 points. On NYU-Depth v2, EvoNet-N3 achieved a state-of-the-art Absolute Relative Error (Abs Rel) of 0.085 while remaining highly lightweight.
- 3D Generalization: When transferred zero-shot to the DepthSplat framework on the RealEstate10K dataset, EvoNet-D achieved competitive rendering fidelity (26.41 PSNR) while utilizing only 12% of the parameters of the baseline model (44M vs. 354M) and running 1.6× faster.
Limitations and Future Work
While the framework drastically improves ranking consistency, the authors note that minor local rank shifts remain inevitable due to extreme architectural variability across heterogeneous blocks. Future research directions include extending this dual-domain alignment paradigm to dynamic 3D scenes and exploring its compatibility with other emerging foundation model backbones.
Summary updated