Bridging Geometric and Semantic Foundation Models for Generalized Monocular Depth Estimation

Sanggyun Ma, Wonjoon Choi, Jihun Park, Jaeyeul Kim, Seunghun Lee, Jiwan Seo, Sunghoon ImarXiv 2025 · DOI · arXiv

Introduction

Monocular Depth Estimation (MDE)—the task of predicting 3D scene depth from a single 2D image—is a cornerstone of robotics, autonomous driving, and augmented reality. While recent "foundation models" like DepthAnything have achieved remarkable results by training on massive geometric datasets, they often struggle with semantic understanding. These models frequently produce blurry or over-smoothed depth maps when encountering complex, thin structures like power lines or tree branches because they lack the context of what an object actually is.

Training a new foundation model from scratch to incorporate semantic data is computationally prohibitive. This paper introduces BriGeS (Bridging Geometric and Semantic), a novel framework that injects semantic awareness into existing depth models efficiently and effectively.

The BriGeS Architecture

The core innovation of BriGeS is the integration of a frozen geometric foundation model (DepthAnything) with a frozen semantic foundation model (SegmentAnything, or SAM). Instead of retraining these massive networks, the authors introduce a lightweight Bridging Gate to fuse their features.

The BriGeS architecture features a Bridging Gate that uses cross-attention to fuse semantic features (fs) with depth features (fd) while keeping the primary encoders frozen.

The Bridging Gate

The Bridging Gate extracts features from both encoders and uses a cross-attention mechanism. In this setup, the depth features act as the query, while semantic features act as the key and value. This allows the model to dynamically refine geometric predictions using semantic boundaries. To maintain efficiency, only the Bridging Gate is trained, while the massive encoders and the depth decoder remain frozen.

Attention Temperature Scaling

A key discovery during development was that cross-modal fusion can cause the attention mechanism to hyper-focus on the centers of objects, neglecting edges. To counteract this, the authors implemented Attention Temperature Scaling. By dividing attention scores by a temperature factor ($\tau = 2.5$) during inference, the model "cools down" its attention, spreading it more evenly across the entire object and its fine-grained boundaries.

Training and Evaluation

By focusing only on the Bridging Gate, BriGeS is remarkably parameter-efficient. It was trained on just 1% of the original DepthAnything training set (~600K samples) in roughly 8–18 hours on a standard 8-GPU setup.

Evaluation was conducted across several zero-shot benchmarks, including KITTI, NYUv2, ETH3D, and DIODE. The model was also tested on the DA-2K Benchmark, which specifically targets high-resolution scenes with intricate details.

Key Results

BriGeS consistently outperforms state-of-the-art models like MiDaS, DPT, and Marigold. Compared to the base DepthAnything V1 and V2 models, BriGeS achieved an average 7.33% reduction in Absolute Relative Error (AbsRel). On the challenging DIODE dataset, the improvement was even more pronounced, with a 15.33% reduction in error.

Qualitative comparison showing how BriGeS (right) captures intricate details like thin branches and statue limbs more accurately than purely geometric models (middle).

Visually, the difference is striking. As shown in the qualitative results, BriGeS prevents depth "bleeding" across edges. It successfully maps intricate fishing nets, thin branches, and power lines that diffusion-based or purely geometric models often blur or ignore.

Conclusion and Future Work

BriGeS demonstrates that semantic awareness is the "missing link" for high-fidelity monocular depth estimation. While the current dual-model approach is computationally heavy during inference, it proves that foundation models can be successfully bridged with minimal retraining. Future research will focus on knowledge distillation to compress this dual-model intelligence into a single, unified, and memory-efficient encoder.

Summary updated

Abstract

We present Bridging Geometric and Semantic (BriGeS), an effective method that fuses geometric and semantic information within foundation models to enhance Monocular Depth Estimation (MDE). Central to BriGeS is the Bridging Gate, which integrates the complementary strengths of depth and segmentation foundation models. This integration is further refined by our Attention Temperature Scaling technique. It finely adjusts the focus of the attention mechanisms to prevent over-concentration on specific features, thus ensuring balanced performance across diverse inputs. BriGeS capitalizes on pre-trained foundation models and adopts a strategy that focuses on training only the Bridging Gate. This method significantly reduces resource demands and training time while maintaining the model's ability to generalize effectively. Extensive experiments across multiple challenging datasets demonstrate that BriGeS outperforms state-of-the-art methods in MDE for complex scenes, effectively handling intricate structures and overlapping objects.
BibTeX
@misc{ma2025bridging,
  author = {Sanggyun Ma and Wonjoon Choi and Jihun Park and Jaeyeul Kim and Seunghun Lee and Jiwan Seo and Sunghoon Im},
  title = {{Bridging Geometric and Semantic Foundation Models for Generalized Monocular Depth Estimation}},
  year = {2025},
  eprint = {2505.23400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.23400},
}
Figure