Abstract
Multimodal agents have demonstrated strong performance in general GUI interactions, but their application in automotive systems has been largely unexplored. In-vehicle GUIs present distinct challenges: drivers' limited attention, strict safety requirements, and complex location-based interaction patterns. To address these challenges, we introduce Automotive-ENV, the first high-fidelity benchmark and interaction environment tailored for vehicle GUIs. This platform defines 185 parameterized tasks spanning explicit control, implicit intent understanding, and safety-aware tasks, and provides structured multimodal observations with precise programmatic checks for reproducible evaluation. Building on this benchmark, we propose ASURADA, a geo-aware multimodal agent that integrates GPS-informed context to dynamically adjust actions based on location, environmental conditions, and regional driving norms. Experiments show that geo-aware information significantly improves success on safety-aware tasks, highlighting the importance of location-based context in automotive environments.
Introduction
While AI agents have become proficient at navigating standard web and mobile interfaces, the automotive "cockpit" presents a significantly more dangerous and dynamic environment. Existing Graphical User Interface (GUI) benchmarks typically rely on static screenshots and stable environments. In contrast, an in-car context changes rapidly. A command such as "I can't see through the windshield" requires the agent to infer environmental conditions (fog) and activate the appropriate hardware (defroster), rather than simply locating a labeled button.
Furthermore, automotive tasks are safety-critical; errors can lead to driver distraction or hazardous maneuvers. Current agents often lack "geo-context"—the situational awareness to know, for instance, that a specific speed is appropriate on a highway but illegal in a residential zone.
Methodology: Automotive-ENV and ASURADA
To address these gaps, we propose two major contributions:
- Automotive-ENV: A high-fidelity simulation environment built on a production-grade in-car operating system. Unlike synthetic benchmarks, it utilizes real software and provides programmatic reward signals. This allows for precise verification of the system state (e.g., confirming the AC is active) rather than relying on visual heuristics.
- ASURADA (Automotive Multimodal Agent): A Vision-Language Model (VLM) agent that treats GPS as a "virtual sensor." Upon receiving a command, ASURADA queries external data for local weather, traffic regulations, and regional norms based on its coordinates. It operates in an iterative loop of perception, planning, execution, and reflection.
Dataset and Task Design
The Automotive-ENV dataset comprises 185 parameterized tasks across eight functional modules, including Maps, HVAC, and Media. These tasks are categorized into:
- General Tasks: Covering explicit control and implicit intent understanding.
- Safety-Aware Tasks: Focusing on driving alignment and environmental alerts.
The parameterized nature of these tasks allows for the generation of millions of unique scenarios, preventing agents from succeeding through simple memorization of static datasets.
Key Insights and Results
Our evaluation of ASURADA against standard multimodal baselines (e.g., GPT-4o, Gemini 1.5) yielded several key findings:
- The Impact of Context: GPS-informed context is the most significant factor in improving safety performance. On "Driving Alignment" tasks, Gemini 1.5 Flash's success rate jumped from 55% to 90% when geo-aware context was provided.
- Implicit Reasoning Challenges: While agents handle explicit commands well, they struggle with implicit requests like "It feels stuffy," which requires reasoning about air quality and ventilation.
- Efficiency: Agents with geo-context produced shorter reasoning traces. By understanding local rules a priori, they avoid exploring incorrect or unsafe options.
- Performance Gap: ASURADA achieved an overall success rate of 65%. While this outperforms web-adapted baselines, it remains far below the near-100% human performance level.
Conclusion and Future Work
While geo-awareness significantly enhances agent performance, a 65-90% success rate is insufficient for safety-critical deployment. Future work must focus on robust "guardrails" to prevent actions that contradict safety protocols. Additionally, addressing GPS reliability in "urban canyons" and improving long-term multi-step planning for ambiguous goals remain critical challenges for the next generation of automotive AI agents.
Summary updated