TL;DR
While vision-language agents excel at desktop and mobile interfaces, in-vehicle operating systems present unique challenges involving driver attention, strict safety constraints, and geo-dependent rules. The authors introduce Automotive-ENV, the first high-fidelity simulation benchmark tailored for vehicle GUIs, alongside ASURADA, a multimodal agent that uses GPS grounding and external geographic context to make safety-aware driving decisions.
What Problem Does This Paper Address?
Existing graphical user interface (GUI) automation benchmarks (like OSWorld, WebArena, or AndroidWorld) evaluate agents in stable, stationary environments like desktop browsers and smartphone apps. In contrast, automotive cockpit interfaces operate under safety-critical, dynamic conditions:
- Safety and Driving Dynamics: Actions depend heavily on real-time vehicle speed, weather, and local traffic laws.
- Ambiguous, Brief Instructions: Drivers cannot type extensive instructions; user queries are often brief or indirect (e.g., "I can’t see through the windshield" requires activating the defroster).
- Location-Specific Norms: An instruction like adjusting climate or checking speed legality depends entirely on external geography and local driving regulations.
Methods
The paper presents both an automotive evaluation platform and a geo-aware agent architecture:
- Automotive-ENV Simulation Platform: Built on an emulated Automotive OS across 8 functional cockpit modules (e.g., HVAC, media, maps, road/safety warnings).
- State Management & Reproducibility: Runs completely offline, resets system time and state before each run, and runs on standard laptop hardware (<4 GB RAM, <10 GB disk).
- System-State Rewards: Instead of flaky screenshot-matching or LLM-as-a-judge evaluations, tasks are verified programmatically by querying low-level OS APIs (checking HVAC target temperatures, media states, or defroster activations directly).
- ASURADA (Automotive Multimodal Agent):
- Extends multimodal GUI frameworks (like M3A using Set-of-Mark visual grounding and ReAct-style iterative planning).
- Treats GPS as an active sensor, retrieving location-specific context (local speed limits, regional road rules, weather) to ground intent interpretation and safety judgments before generating GUI actions or vehicle API calls.
Datasets
- Automotive-ENV (Newly Introduced): Features 185 parameterized task templates spanning General Tasks (Explicit Control and Implicit Intent) and Safety-Aware Tasks (Driving Alignment and Environment Alerts). Because parameters are dynamically randomized, the benchmark can instantiate virtually infinite unique evaluation scenarios.
- Compared to existing GUI datasets (e.g., AndroidWorld, OSWorld, WebArena), Automotive-ENV uniquely provides an automotive OS target, programmatic OS-level reward checks, and geographic parameterization.
Key Insights and Ideas
- Context is Critical for Safety Evaluation: Without location priors, an agent seeing vehicle speed at 80 km/h defaults to treating it as normal. With GPS context indicating the vehicle is in central Paris (where city speed limits are 50 km/h), the agent correctly flags the driving as unsafe and alerts the driver.
- Geo-Grounding Reduces Reasoning Overhead: Adding environmental priors doesn't complicate agent reasoning—it streamlines it. Agents with geo-context generated significantly shorter, more focused reasoning traces (mostly under 1,500 tokens) compared to context-blind baselines that often drifted up to 3,000 tokens trying to resolve ambiguity.
- Visual Grounding Boosts General Tasks, While Geo-Context Secures Safety: Adding visual screenshots (M3A) significantly improved explicit GUI control over text-only trees (T3A), but adding GPS context (ASURADA) was the single decisive factor in driving safety-aware tasks toward high reliability.
Key Results
- Overall Performance: ASURADA achieved a 65% overall success rate, outperforming adapted mobile/web baseline architectures, but still trailing far behind human performance (which nears 100% on safety tasks).
- Driving Alignment (DA): On Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 2.0 Flash, and Gemini 2.0 Flash-Lite, ASURADA achieved 90.0% success on DA (compared to 42.8%–75.0% for vision-only M3A).
- Environment Alerts (EA): On Gemini 1.5 Flash, adding GPS context jumped performance from 55.0% to 85.0%.
- Category-Level Gains: Dynamic modules like Media, Maps, HVAC, and Road saw 15–30% performance jumps when geo-context was provided, whereas deterministic system tasks (e.g., basic device settings) only changed by 2–5%.
Limitations and Future Work
- Implicit Intent Gap: Models still struggle significantly with ambiguous, indirect user instructions (e.g., GPT-4o-Mini achieves only 14.2% on implicit intent vs. 82.0% human benchmark), frequently failing to translate vague goals into multi-step GUI actions.
- Simulation Constraints: While Automotive-ENV simulates OS states and sensor inputs offline, real vehicles involve more continuous, multimodal sensor streams (CAN bus telemetry, physical buttons, voice tonality, camera feeds) that remain to be explored.
Summary updated