BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen, Sihao Ding

Abstract

The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency. This separation in visual processing hinders accurate 3D spatial reasoning and fails to maintain geometric coherence across views. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46%, by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance by 29% in safety-critical scenarios.

Introduction

Current "LLM-for-driving" systems typically treat camera feeds as a collection of independent images. This approach suffers from two major flaws. First, it introduces spatial inconsistency, as it is computationally expensive and logically difficult for an LLM to reconcile objects across multiple views—such as matching a vehicle transitioning from a front-left camera to a side-view camera. Second, there is a significant semantic gap. While Bird’s-Eye View (BEV) representations are excellent at capturing geometry and spatial structure, they are often "semantically poor," trained primarily to identify bounding boxes rather than understand complex scene context or "common sense" driving logic.

Standard BEV encoders utilize geometrically annotated driving data to provide spatial structure but often lack high-level semantic reasoning capabilities.

BEVLM addresses these issues by creating a spatially consistent representation that is semantically distilled from powerful foundation language models, effectively giving autonomous vehicles a "common sense" map of their environment.

Methodology

The BEVLM framework utilizes a two-stage approach to bridge the gap between geometric structure and semantic reasoning.

1. Representation Study

The authors first demonstrate that feeding an LLM a unified BEV grid is significantly more effective for spatial reasoning than feeding it individual camera tokens. By using a lightweight MLP projector to align BEV features with the LLM’s input space, the model can reason about the 3D relationships between objects across different views more efficiently than massive Vision Transformers.

2. Semantic Distillation

This core novelty treats a frozen, high-performance LLM (such as InternVL3) as a "semantic teacher." The BEV encoder, acting as the "student," is trained via Visual Question Answering (VQA) tasks to produce features that align with the LLM’s semantic manifold. This process forces the BEV encoder to reshape its internal feature space to satisfy the structural requirements of the LLM's semantic priors.

The BEVLM framework integrates semantically rich image-text data and geometrically annotated driving data to produce a semantic-distilled BEV representation suitable for LLM reasoning.

3. Joint Training

To maintain the model's fundamental perception capabilities, the distillation process is trained jointly with standard object detection tasks. This ensures that the resulting BEV map preserves geometric integrity while being enriched with high-level concepts like "safety-critical behavior."

Experiments and Results

Datasets

The study evaluates performance across several benchmarks:

  • DriveLM-nuScenes: VQA pairs for perception, prediction, and planning.
  • Ego3D: Evaluation of object-centric cross-view reasoning.
  • nuScenes: Standard open-loop evaluation.
  • NeuroNCAP: A NeRF-based simulation for closed-loop evaluation in safety-critical scenarios.

Key Findings

  • Superior Spatial Reasoning: BEVLM improved accuracy in cross-view driving scenes by 46% compared to standard multi-view image inputs. Even a small 44M parameter BEV encoder outperformed 400M parameter Vision Transformers in 3D relationship tasks.
  • Enhanced Safety: In closed-loop NeuroNCAP testing, the distilled model improved safety scores by 29% and decreased collision rates by 11.3% in corner cases.
  • Behavioral Impact: Distilling knowledge from "Behavior" and "Planning" questions was found to be more impactful for safety than simple "Perception" distillation. In unavoidable collision scenarios, the model demonstrated better anticipatory behavior, such as earlier braking.
  • Efficiency: The framework achieves performance comparable to foundation vision encoders that are ten times larger in model size.

Discussion and Future Work

While BEVLM shows significant promise, the semantic distillation process is computationally intensive, requiring substantial GPU hours. Future work will focus on moving toward "human-labor-free" distillation by using pseudo-labels from larger Vision-Language Models (VLMs) and testing the framework across a wider variety of BEV encoder architectures beyond BEVFormer and UniAD. Ultimately, BEVLM demonstrates that distilling high-level semantic knowledge into spatially consistent maps is a vital step toward more robust and "intelligent" autonomous driving systems.

Summary updated

Abstract

The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency. This separation in visual processing hinders accurate 3D spatial reasoning and fails to maintain geometric coherence across views. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46%, by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance by 29% in safety-critical scenarios.
BibTeX
@misc{monninger2026bevlm,
  author = {Thomas Monninger and Shaoyuan Xie and Qi Alfred Chen and Sihao Ding},
  title = {{BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations}},
  year = {2026},
  eprint = {2603.06576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.06576},
}
Figure