Automotive-ENV: Benchmarking Multimodal Agents in Vehicle Interface Systems

Junfeng Yan, Biao Wu, Meng Fang, Ling ChenSubmission to ICLR 2026 · Withdrawn

TL;DR

While vision-language agents excel at desktop and mobile interfaces, in-vehicle operating systems present unique challenges involving driver attention, strict safety constraints, and geo-dependent rules. The authors introduce Automotive-ENV, the first high-fidelity simulation benchmark tailored for vehicle GUIs, alongside ASURADA, a multimodal agent that uses GPS grounding and external geographic context to make safety-aware driving decisions.

What Problem Does This Paper Address?

Existing graphical user interface (GUI) automation benchmarks (like OSWorld, WebArena, or AndroidWorld) evaluate agents in stable, stationary environments like desktop browsers and smartphone apps. In contrast, automotive cockpit interfaces operate under safety-critical, dynamic conditions:

  • Safety and Driving Dynamics: Actions depend heavily on real-time vehicle speed, weather, and local traffic laws.
  • Ambiguous, Brief Instructions: Drivers cannot type extensive instructions; user queries are often brief or indirect (e.g., "I can’t see through the windshield" requires activating the defroster).
  • Location-Specific Norms: An instruction like adjusting climate or checking speed legality depends entirely on external geography and local driving regulations.

Methods

The paper presents both an automotive evaluation platform and a geo-aware agent architecture:

  1. Automotive-ENV Simulation Platform: Built on an emulated Automotive OS across 8 functional cockpit modules (e.g., HVAC, media, maps, road/safety warnings).
  2. State Management & Reproducibility: Runs completely offline, resets system time and state before each run, and runs on standard laptop hardware (<4 GB RAM, <10 GB disk).
  3. System-State Rewards: Instead of flaky screenshot-matching or LLM-as-a-judge evaluations, tasks are verified programmatically by querying low-level OS APIs (checking HVAC target temperatures, media states, or defroster activations directly).
  4. ASURADA (Automotive Multimodal Agent):
  5. Extends multimodal GUI frameworks (like M3A using Set-of-Mark visual grounding and ReAct-style iterative planning).
  6. Treats GPS as an active sensor, retrieving location-specific context (local speed limits, regional road rules, weather) to ground intent interpretation and safety judgments before generating GUI actions or vehicle API calls.

Datasets

  • Automotive-ENV (Newly Introduced): Features 185 parameterized task templates spanning General Tasks (Explicit Control and Implicit Intent) and Safety-Aware Tasks (Driving Alignment and Environment Alerts). Because parameters are dynamically randomized, the benchmark can instantiate virtually infinite unique evaluation scenarios.
  • Compared to existing GUI datasets (e.g., AndroidWorld, OSWorld, WebArena), Automotive-ENV uniquely provides an automotive OS target, programmatic OS-level reward checks, and geographic parameterization.

Key Insights and Ideas

  • Context is Critical for Safety Evaluation: Without location priors, an agent seeing vehicle speed at 80 km/h defaults to treating it as normal. With GPS context indicating the vehicle is in central Paris (where city speed limits are 50 km/h), the agent correctly flags the driving as unsafe and alerts the driver.
  • Geo-Grounding Reduces Reasoning Overhead: Adding environmental priors doesn't complicate agent reasoning—it streamlines it. Agents with geo-context generated significantly shorter, more focused reasoning traces (mostly under 1,500 tokens) compared to context-blind baselines that often drifted up to 3,000 tokens trying to resolve ambiguity.
  • Visual Grounding Boosts General Tasks, While Geo-Context Secures Safety: Adding visual screenshots (M3A) significantly improved explicit GUI control over text-only trees (T3A), but adding GPS context (ASURADA) was the single decisive factor in driving safety-aware tasks toward high reliability.

Key Results

  • Overall Performance: ASURADA achieved a 65% overall success rate, outperforming adapted mobile/web baseline architectures, but still trailing far behind human performance (which nears 100% on safety tasks).
  • Driving Alignment (DA): On Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 2.0 Flash, and Gemini 2.0 Flash-Lite, ASURADA achieved 90.0% success on DA (compared to 42.8%–75.0% for vision-only M3A).
  • Environment Alerts (EA): On Gemini 1.5 Flash, adding GPS context jumped performance from 55.0% to 85.0%.
  • Category-Level Gains: Dynamic modules like Media, Maps, HVAC, and Road saw 15–30% performance jumps when geo-context was provided, whereas deterministic system tasks (e.g., basic device settings) only changed by 2–5%.

Limitations and Future Work

  • Implicit Intent Gap: Models still struggle significantly with ambiguous, indirect user instructions (e.g., GPT-4o-Mini achieves only 14.2% on implicit intent vs. 82.0% human benchmark), frequently failing to translate vague goals into multi-step GUI actions.
  • Simulation Constraints: While Automotive-ENV simulates OS states and sensor inputs offline, real vehicles involve more continuous, multimodal sensor streams (CAN bus telemetry, physical buttons, voice tonality, camera feeds) that remain to be explored.

Summary updated

Abstract

Multimodal agents have demonstrated strong performance in general GUI interactions, but their application in automotive systems has been largely unexplored. In-vehicle GUIs present distinct challenges: drivers' limited attention, strict safety requirements, and complex location-based interaction patterns. To address these challenges, we introduce Automotive-ENV, the first high-fidelity benchmark and interaction environment tailored for vehicle GUIs. This platform defines 185 parameterized tasks spanning explicit control, implicit intent understanding, and safety-aware tasks, and provides structured multimodal observations with precise programmatic checks for reproducible evaluation. Building on this benchmark, we propose ASURADA, a geo-aware multimodal agent that integrates GPS-informed context to dynamically adjust actions based on location, environmental conditions, and regional driving norms. Experiments show that geo-aware information significantly improves success on safety-aware tasks, highlighting the importance of location-based context in automotive environments. We will release Automotive-ENV, complete with all tasks and benchmarking tools, to further the development of safe and adaptive in-vehicle agents.
BibTeX
@misc{yan2025automotive,
  author = {Junfeng Yan and Biao Wu and Meng Fang and Ling Chen},
  title = {{Automotive-ENV: Benchmarking Multimodal Agents in Vehicle Interface Systems}},
  year = {2025},
  eprint = {2509.21143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.21143},
}

Loading related content...

Figure