TL;DR
This study compares how well three large language models (LLMs) – GPT-3.5, GPT-4, and Google-Gemini – can analyze driver interview data for psychological assessment. While GPT-4 offers the most in-depth analysis, it's slow. Google-Gemini strikes a good balance between analysis quality and speed, making it the overall best performer for this task.
What Problem Does This Paper Address?
Traditional methods for gathering subjective feedback, like questionnaires, have limitations. They can be rigid, fail to capture nuanced feelings, and may not allow respondents to fully express themselves. Interviews offer richer data but are time-consuming and labor-intensive to analyze. This paper explores whether LLMs can automate and improve the analysis of interview data for driver psychological assessment, a crucial area in transportation research for understanding driver behavior and improving safety.
Methods
The researchers recruited 29 participants for a simulated tunnel driving experiment. After the experiment, participants completed questionnaires and then participated in interviews. The interview transcripts were then fed into three LLMs: GPT-3.5, GPT-4, and Google-Gemini. A specific set of instructions (prompts) was designed for each LLM to analyze the interview content and extract key viewpoints and information. An evaluation index system was developed with six parameters: clarity and precision of key points, logical coherence and flow, fluency and natural language use, emotion recognition and handling, analytical speed, and stability of multiple responses. These indicators were weighted using a combination of the Entropy Weight Method and the CRITIC method to derive a comprehensive score for each LLM's performance.
Datasets
The study utilized interview data from 29 participants who underwent a simulated tunnel driving experiment. This resulted in a dataset of 173 interview data points (after excluding one response). This dataset was used to evaluate the performance of the LLMs. No new benchmark datasets were introduced.
Key Insights and Ideas
- Interviews offer richer data than questionnaires: Even with structured questions, interviews allowed drivers to express a wider range of feelings and factors influencing their experience, which questionnaires couldn't capture.
- LLMs can automate interview analysis: The study demonstrates that LLMs can process and summarize complex interview data, offering a potential solution to the time-consuming nature of manual analysis.
- Trade-offs exist between LLM performance and speed: GPT-4 provides the most detailed and accurate analysis but is significantly slower than GPT-3.5.
- Google-Gemini offers a balanced solution: It achieves analysis quality comparable to GPT-4 while maintaining a much faster processing speed, making it a practical choice for many research tasks.
- LLMs can sometimes misunderstand: Despite their capabilities, LLMs occasionally misinterpret nuances or add extraneous information, highlighting the need for human oversight.
Key Results
- GPT-4 vs. GPT-3.5: GPT-4 generally outperformed GPT-3.5 in clarity, logical coherence, emotion recognition, and stability. However, GPT-4's analysis speed was 7-15 times slower than GPT-3.5, significantly impacting its overall weighted score. GPT-3.5 was found to be superior in terms of analytical speed.
- Google-Gemini vs. ChatGPT: Google-Gemini demonstrated superior performance in clarity, emotion recognition, and stability compared to both GPT-3.5 and GPT-4. Its analytical speed was slightly slower than GPT-3.5 but significantly faster than GPT-4. Overall, Google-Gemini was found to be more stable and offered a better balance of quality and speed.
- Overall Ranking: Google-Gemini was identified as the best overall performer, followed by GPT-4, and then GPT-3.5.
Limitations and Future Work
- Sample Size: The study used a relatively small sample size of 29 participants.
- Targeted Sampling: For the comparison between Google-Gemini and ChatGPT, a targeted sampling strategy was used, focusing on the top and bottom-scoring interview samples. This means the findings are illustrative rather than a comprehensive representation of the entire dataset.
- Language Specificity: The study focused on Chinese language interviews. Further research is needed to assess LLM performance across different languages and domains.
- Human Oversight: The authors emphasize the need for a "human-in-the-loop" validation step, especially in safety-critical research, to ensure the reliability and contextual accuracy of AI-generated summaries.
- Future Work: The authors suggest further research into LLM performance across different languages and domains, as well as exploring the integration of multimodal data and optimizing evaluation criteria systems.
Summary updated