Abstract
Traditional AI evaluations often prioritize technical benchmarks over human-centered factors. This research introduces a novel methodology for studying human-AI interaction in naturalistic settings by analyzing live-streamed interviews between human streamers and GPT-3.5. By examining real-time audience reactions in chat, the study reveals that users are predominantly curious and amused rather than apprehensive. Findings indicate that the volume of interaction is a more significant predictor of engagement than the specific nature (cognitive vs. emotional) of the AI's discourse.
Introduction
The rapid advancement of Large Language Models (LLMs) necessitates a shift toward Human-Centered AI (HCAI). While existing research often relies on controlled lab environments or surveys, these methods can introduce observer bias or lack ecological validity. This paper addresses the need for "in the wild" observation by leveraging live streaming platforms—dynamic environments where communities interact spontaneously—to gauge authentic human perceptions of AI agents in real-time.
Methodology
The researchers developed a custom pipeline to facilitate and analyze live human-AI dialogues. Volunteer streamers utilized a web application providing a voice interface (Speech-to-Text and Text-to-Speech) to interact with OpenAI’s GPT-3.5. Questions were categorized into two main types: Cognitive, focusing on skills and performance, and Emotional, focusing on ethics and abstract reasoning.
Data Analysis Pipeline
The analysis involved several stages to process the "crowd's pulse":
- Translation: Italian streams were translated via the DeepL API to facilitate the use of standard NLP models.
- Segmentation: Sessions were divided into 15-second windows to correlate specific AI outputs with audience chat messages.
- Sentiment and Emotion Analysis: A
roberta-basemodel was used to identify 28 emotion labels, while atwitter-roberta-basemodel classified general sentiment. - Topic Modeling: BERT embeddings and Latent Dirichlet Allocation (LDA) were employed to cluster the themes of the conversations.
Results
The study analyzed 311 specific interactions and a corresponding corpus of audience comments. Sentiment analysis showed a distribution of 57.7% neutral, 21.6% positive, and 20.7% negative. Notably, positive "strong reactions" (853) significantly outnumbered negative ones (444), suggesting that curiosity and amusement are the dominant audience responses to AI agents.
Temporal analysis confirmed that 89.1% of comments occurred after the AI's response, indicating that the audience was reacting specifically to the AI's content rather than the streamer's prompt. Furthermore, linear regression revealed that the total number of comments was the only significant predictor of engagement; the topic type (Cognitive vs. Emotional) did not statistically influence the intensity of the audience's reaction.
Conclusion
This "crowd-based" approach provides a more authentic lens into human-AI interaction than traditional lab studies. While the study is limited by sample size and cultural specificity (Italian-speaking communities), it highlights a general trend of positive engagement and curiosity toward AI. Future research will expand to broader linguistic and cultural contexts to compare reactions across diverse AI models like GPT-4 and Claude.
Summary updated