Dual-Domain Representation Alignment: Bridging 2D and 3D Vision via Geometry-Aware Architecture Search

Haoyu Zhang, Zhihao Yu, Rui Wang, Yaochu Jin, Qiqi Liu, Ran ChengarXiv 2026 · arXiv

TL;DR

EvoNAS is a novel multi-objective evolutionary neural architecture search framework designed to discover efficient, geometry-aware models for dense 2D and 3D vision tasks. By enforcing dual-domain (spatial and frequency) knowledge distillation and isolating hardware evaluation noise, it produces a family of compact "EvoNets" that achieve Pareto-optimal accuracy-efficiency trade-offs, reducing parameter overhead by up to 88% in 3D rendering applications.

What Problem Does This Paper Address?

Deploying Large Vision Models (LVMs) on resource-constrained edge devices (like autonomous vehicles or mobile systems) is heavily restricted by their massive computational footprints and high inference latencies. While Evolutionary Neural Architecture Search (ENAS) is a powerful tool for balancing competing objectives like accuracy and speed, its practical application faces two major bottlenecks:

  1. Representation Collapse & Ranking Inconsistency: In weight-sharing supernets, diverse sub-architectures compete for the same parameters. This gradient interference typically favors low-frequency semantic stability over high-frequency geometric precision. As a result, a subnetwork's performance within the supernet often correlates poorly with its standalone performance after independent training, rendering the search process unreliable.
  2. Hardware Latency Jitter: Evaluating multiple candidate networks concurrently on shared GPUs causes CUDA kernel contention and resource fragmentation. This introduces significant noise into physical latency measurements, misleading the evolutionary algorithm's selection process.

Methods

To overcome these challenges, the authors introduce EvoNAS, which combines a hybrid search space with advanced optimization and evaluation techniques:

  • Hybrid VSS-ViT Search Space: The framework designs a multi-scale encoder that integrates the linear-time computational efficiency of Vision State Space (VSS/Mamba) blocks in the early stages with the global reasoning of Vision Transformers (ViT) in the final stage. This is paired with a Spatial-Mamba decoder to preserve fine-grained structural boundaries.
The EvoNAS unified visual architecture pipeline featuring a searchable hybrid VSS-ViT encoder and Spatial Mamba decoder.
  • Cross-Architecture Dual-Domain Knowledge Distillation (CA-DDKD): To fix ranking inconsistency, EvoNAS anchors the supernet to a geometry-aware teacher model (Depth Anything). Crucially, it enforces alignment in both the spatial domain (via feature tokens) and the frequency domain using Discrete Cosine Transform (DCT) constraints. Forcing the supernet to preserve high-frequency spectral components prevents the erosion of sharp boundaries and stabilizes subnetwork ranking.
Overview of the two-stage supernet optimization with dual-domain knowledge distillation.
  • Progressive Supernet Training (PST): A curriculum-based training strategy that warm-starts the search space by optimizing high-capacity configurations first before sequentially unlocking smaller dimensions, minimizing gradient interference.
  • Distributed Multi-Model Parallel Evaluation (DMMPE): A hardware-isolated evaluation engine that leverages NVIDIA Multi-Process Service (MPS) and asynchronous scheduling. By pooling GPU resources and isolating execution contexts, it eliminates latency jitter and provides a clean, unbiased fitness landscape.

Datasets

The framework was evaluated across a multi-tiered benchmark suite spanning 2D, 2.5D, and 3D complexities:

  • COCO: Object detection and instance segmentation.
  • ADE20K: Pixel-wise semantic segmentation.
  • KITTI & NYU-Depth v2: Outdoor and indoor monocular depth estimation (MDE).
  • RealEstate10K: Zero-shot architectural transfer for 3D novel view synthesis via 3D Gaussian Splatting (3DGS).

Key Insights and Ideas

  • The Frequency Domain is the Key to Geometry: Traditional NAS methods treat features as generic tensors, ignoring spectral properties. By explicitly supervising the frequency domain via DCT, EvoNAS ensures that even highly pruned, lightweight subnetworks retain the sharp edge and boundary information critical for 3D rendering.
  • Hardware Isolation is an Algorithmic Necessity: Latency proxies (like FLOPs) are often poor stand-ins for real-world throughput. However, raw physical measurements are too noisy on shared hardware. True hardware isolation via virtualization (MPS) acts as a high-fidelity sensor that is critical for robust multi-objective Pareto optimization.
  • Universal Geometric Priors Transfer to 3D: Models that successfully learn to align spatial and spectral structures for 2D dense prediction tasks possess an intrinsic geometric understanding that allows them to be seamlessly "hot-swapped" into 3D rendering pipelines without massive parameter scaling.

Key Results

  • Evaluation Efficiency: The DMMPE framework improved validation throughput by over 70% compared to conventional data-parallel evaluation, achieving a threefold speedup under optimal configurations.
  • 2D/2.5D Benchmarks: Discovered architectures, named EvoNets, consistently established new Pareto-optimal frontiers. For instance, EvoNet-C1 (33M parameters) achieved 45.4 AP on COCO, outperforming ResNet-50 by 7.2 points. On NYU-Depth v2, EvoNet-N3 achieved a state-of-the-art Absolute Relative Error (Abs Rel) of 0.085 while remaining highly lightweight.
  • 3D Generalization: When transferred zero-shot to the DepthSplat framework on the RealEstate10K dataset, EvoNet-D achieved competitive rendering fidelity (26.41 PSNR) while utilizing only 12% of the parameters of the baseline model (44M vs. 354M) and running 1.6× faster.

Limitations and Future Work

While the framework drastically improves ranking consistency, the authors note that minor local rank shifts remain inevitable due to extreme architectural variability across heterogeneous blocks. Future research directions include extending this dual-domain alignment paradigm to dynamic 3D scenes and exploring its compatibility with other emerging foundation model backbones.

Summary updated

Abstract

Modern computer vision requires balancing predictive accuracy with real-time efficiency, yet the high inference cost of large vision models (LVMs) limits deployment on resource-constrained edge devices. Although Evolutionary Neural Architecture Search (ENAS) is well suited for multi-objective optimization, its practical use is hindered by two issues: expensive candidate evaluation and ranking inconsistency among subnetworks. To address them, we propose EvoNAS, an efficient distributed framework for multi-objective evolutionary architecture search. We build a hybrid supernet that integrates Vision State Space and Vision Transformer (VSS-ViT) modules, and optimize it with a Cross-Architecture Dual-Domain Knowledge Distillation (CA-DDKD) strategy. By coupling the computational efficiency of VSS blocks with the semantic expressiveness of ViT modules, CA-DDKD improves the representational capacity of the shared supernet and enhances ranking consistency, enabling reliable fitness estimation during evolution without extra fine-tuning. To reduce the cost of large-scale validation, we further introduce a Distributed Multi-Model Parallel Evaluation (DMMPE) framework based on GPU resource pooling and asynchronous scheduling. Compared with conventional data-parallel evaluation, DMMPE improves efficiency by over 70% through concurrent multi-GPU, multi-model execution. Experiments on COCO, ADE20K, KITTI, and NYU-Depth v2 show that the searched architectures, termed EvoNets, consistently achieve Pareto-optimal trade-offs between accuracy and efficiency. Compared with representative CNN-, ViT-, and Mamba-based models, EvoNets deliver lower inference latency and higher throughput under strict computational budgets while maintaining strong generalization on downstream tasks such as novel view synthesis. Code is available at https://github.com/EMI-Group/evonas
BibTeX
@misc{zhang2026dual,
  author = {Haoyu Zhang and Zhihao Yu and Rui Wang and Yaochu Jin and Qiqi Liu and Ran Cheng},
  title = {{Dual-Domain Representation Alignment: Bridging 2D and 3D Vision via Geometry-Aware Architecture Search}},
  year = {2026},
  eprint = {2603.19563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.19563},
}
Figure