MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction With Automation

Lex Fridman, Daniel E. Brown, Michael Glazer, William Angell, Spencer Dodd, Benedikt Jenik, Jack Terwilliger, Aleksandr Patsekin, Julia Kindelsberger, Li Ding, Sean Seaman, Alea Mehler, Andrew Sipperley, Anthony Pettinato, Bobbie D. Seppelt, Linda Angell, Bruce Mehler, Bryan ReimerIEEE Access 2019 · OpenReview

TL;DR

This paper introduces the MIT Advanced Vehicle Technology (MIT-AVT) study, a large-scale naturalistic driving dataset aimed at understanding human-automation interaction in semi-autonomous vehicles. By collecting extensive real-world driving data from instrumented vehicles, the study aims to fuel the development of robust AI systems and improve how humans and automation collaborate to enhance safety.

What Problem Does This Paper Address?

The paper addresses the fundamental challenge that fully autonomous driving, especially in unconstrained real-world scenarios, remains a complex problem. Despite advancements, current autonomous systems still require human oversight due to the inherent variability of driving, including unpredictable human behavior, complex scene perception, and system limitations. This necessitates a deep understanding of how humans interact with and monitor these systems, especially during the transition period where humans remain an integral part of the driving task. The MIT-AVT study aims to bridge this gap by collecting and analyzing massive amounts of naturalistic driving data to better characterize this human-automation interaction.

Methods

The MIT-AVT study employs a multi-faceted approach to data collection and analysis:

  • Naturalistic Driving Study (NDS): Data is collected from participants driving their own vehicles (long-term) or study-provided vehicles (medium-term) under normal, unconstrained conditions. This contrasts with controlled experiments where drivers perform specific tasks.
  • Comprehensive Instrumentation: Vehicles are equipped with a sophisticated data logging system called RIDER, which captures:
  • High-definition video from multiple cameras (driver face, cabin, forward view, instrument cluster).
  • Vehicle state data via the CAN bus (e.g., speed, acceleration).
  • Inertial Measurement Unit (IMU) and GPS data for motion and location tracking.
  • Deep Learning for Analysis: The study leverages deep learning techniques for automated analysis of the massive dataset. This includes:
  • Driver State Detection: Analyzing facial expressions, eye gaze, and head pose.
  • Body Pose Estimation: Understanding driver posture and position.
  • Semantic Scene Perception: Segmenting and understanding the driving environment.
  • Driving State Prediction: Inferring vehicle and system status.
  • Data Pipeline: A robust pipeline is described for processing the raw sensor data, including cleaning, synchronization, and knowledge extraction.
  • Qualitative Data Collection: Questionnaires and interviews are used to gather insights into participants' mental models, trust, and experiences with automation.

Datasets

The paper introduces the MIT Advanced Vehicle Technology (MIT-AVT) dataset, a newly collected, large-scale naturalistic driving dataset. Key statistics include:

  • Participants: 122
  • Participant Days: 15,610
  • Vehicles: 29 (including Tesla Model S/X, Volvo S90, Range Rover Evoque, Cadillac CT6)
  • Miles Driven: 511,638
  • Video Frames: 7.1 billion

This dataset differs from previous NDS studies by focusing on automated, deep learning-based analysis of the "long-tail" of driving data (i.e., data outside of specific crash or near-crash epochs), enabling a more holistic understanding of human-automation interaction over extended periods.

Key Insights and Ideas

  • The "Long-Tail" is Crucial: The paper emphasizes the importance of analyzing the vast majority of driving data that isn't an "event" (like a crash). Understanding how drivers interact with automation during routine driving, when systems are engaged for long periods, is key to improving safety and user experience.
  • Human-Automation Collaboration is the Future: The study operates under the premise that humans will remain integral to driving for the foreseeable future, acting as supervisors or collaborators with AI systems. This necessitates understanding the nuances of this partnership.
  • Scale is Necessary for Robustness: The sheer volume of data collected (7.1 billion video frames) is essential for training deep learning models that can generalize to the vast number of edge cases encountered in real-world driving.
  • Holistic Understanding is Key: Combining video data with vehicle telemetry, driver characteristics, and self-reported experiences provides a richer, more comprehensive view of human-automation interaction than any single data modality alone.

Key Results

The paper primarily focuses on the study's design, methodology, and data collection. While specific quantitative results from the analysis of the dataset are not detailed in this paper, the authors highlight the following:

  • Successful Data Collection: The study has successfully instrumented 29 vehicles and collected over 511,000 miles of naturalistic driving data, generating 7.1 billion video frames.
  • Development of RIDER System: A robust hardware and software system (RIDER) has been developed for reliable, timestamped data logging in diverse vehicle environments.
  • Foundation for Future Analysis: The collected dataset and the described data pipeline provide a strong foundation for future research into driver behavior, automation interaction, and the development of advanced AI systems.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the analysis of the collected data is ongoing, and this paper focuses on the study's design and data collection. Specific findings from the analysis are deferred to future work.
  • The paper mentions the complexity of decoding CAN bus messages due to proprietary information and varying vehicle implementations.
  • Future Work:
    • The authors plan to conduct individual analysis efforts on the collected data to extract actionable knowledge.
  • Future work will involve leveraging the dataset to propose, design, and build systems that improve shared autonomy and prevent unintended consequences.
  • Ongoing hardware development aims to upgrade the computing platform (e.g., NVIDIA Jetson TX2) for enhanced processing capabilities, enabling real-time systems for detection, annotation, and driver monitoring.
  • The study is actively growing, with data collection ongoing.

Summary updated

Abstract

Today, and possibly for a long time to come, the full driving task is too complex an activity to be fully formalized as a sensing-acting robotics system that can be explicitly solved through model-based and learning-based approaches in order to achieve full unconstrained vehicle autonomy. Localization, mapping, scene perception, vehicle control, trajectory optimization, and higher-level planning decisions associated with autonomous vehicle development remain full of open challenges. This is especially true for unconstrained, real-world operation where the margin of allowable error is extremely small and the number of edge-cases is extremely large. Until these problems are solved, human beings will remain an integral part of the driving task, monitoring the AI system as it performs anywhere from just over 0% to just under 100% of the driving. The governing objectives of the MIT Advanced Vehicle Technology (MIT-AVT) study are to 1) undertake large-scale real-world driving data collection that includes high-definition video to fuel the development of deep learning-based internal and external perception systems; 2) gain a holistic understanding of how human beings interact with vehicle automation technology by integrating video data with vehicle state data, driver characteristics, mental models, and self-reported experiences with technology; and 3) identify how technology and other factors related to automation adoption and use can be improved in ways that save lives. In pursuing these objectives, we have instrumented 23 Tesla Model S and Model X vehicles, 2 Volvo S90 vehicles, 2 Range Rover Evoque, and 2 Cadillac CT6 vehicles for both long-term (over a year per driver) and medium-term (one month per driver) naturalistic driving data collection. Furthermore, we are continually developing new methods for the analysis of the massive-scale dataset collected from the instrumented vehicle fleet. The recorded data streams include IMU, GPS, and CAN messages, and high-definition video streams of the driver's face, the driver cabin, the forward roadway, and the instrument cluster (on select vehicles). The study is on-going and growing. To date, we have 122 participants, 15610 days of participation, 511638 mi, and 7.1 billion video frames. This paper presents the design of the study, the data collection hardware, the processing of the data, and the computer vision algorithms currently being used to extract actionable knowledge from the data.
BibTeX
@inproceedings{fridman2019mit,
  author = {Lex Fridman and Daniel E. Brown and Michael Glazer and William Angell and Spencer Dodd and Benedikt Jenik and Jack Terwilliger and Aleksandr Patsekin and Julia Kindelsberger and Li Ding and Sean Seaman and Alea Mehler and Andrew Sipperley and Anthony Pettinato and Bobbie D. Seppelt and Linda Angell and Bruce Mehler and Bryan Reimer},
  title = {{MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction With Automation}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=AiWk3S1YmQ},
}
Figure