Abstract
Web surveys completed on smartphones open novel ways for measuring respondents’ attitudes, behaviors, and beliefs that are crucial for social science research and many adjacent research fields. In this study, we make use of the built-in microphones of smartphones to record voice answers in a smartphone survey and extract non-verbal cues, such as amplitudes and pitches, from the collected voice data. This allows us to predict respondents’ level of interest (i.e., disinterest, neutral, and high interest) based on their voice answers, which expands the opportunities for researching respondents’ engagement and answer behavior. We conducted a smartphone survey in a German online access panel and asked respondents four open-ended questions on political parties with requests for voice answers. In addition, we measured respondents’ self-reported survey interest using a closed-ended question with an end-labeled, seven-point rating scale. The results show a non-linear association between respondents’ predicted level of interest and answer length. Respondents with a predicted medium level of interest provide longer answers in terms of number of words and response times. However, respondents’ predicted level of interest and their self-reported interest are weakly associated. Finally, we argue that voice answers contain rich meta-information about respondents’ affective states, which are yet to be utilized in survey research.
Introduction: The Challenge of Measuring Engagement
In social science research, understanding a respondent's level of engagement is crucial for ensuring data quality. Traditionally, researchers have relied on two primary, yet flawed, methods: self-reported interest and indirect proxies like "speeding" (answering questions too quickly). Self-reports are often burdensome, typically only asked once at the end of a survey, and may not reflect question-specific engagement. Speeding, while useful, is a coarse metric that fails to capture the nuanced, affective states of a respondent.
This study investigates whether the built-in sensors of smartphones—specifically microphones—can provide a passive "meta-data" stream. By analyzing the sound of a respondent's voice rather than just the text of their answer, researchers can infer interest levels directly and dynamically throughout the survey process.
Methodology and Data Collection
The researchers integrated a recording tool called SurveyVoice (SVoice) into a browser-based smartphone survey. To analyze the resulting audio, they employed OpenEAR (Open Emotion and Affect Recognition), an open-source toolkit.
The primary dataset was collected from the Forsa Omninet Panel in Germany between December 2019 and January 2020. The final analysis included 925 panelists who provided voice answers to four open-ended questions regarding German political parties: the CDU/CSU, SPD, Greens, and AfD.
To categorize speech into levels of interest (disinterest, neutral, and high interest), the team used the Audiovisual Interest Corpus (AVIC) model set. This pre-trained Support Vector Machine (SVM) classifier extracts acoustic features such as signal energy, voice quality, pitch, and spectral features. To compare these vocal cues against textual content, the researchers used the Google Transcribe API for speech-to-text conversion and performed sentiment analysis using the SentiWS vocabulary.
Key Insights: The Sound of Interest
The "Inverse U-Shape" of Engagement
The most striking finding was a non-linear association between predicted interest and answer length. Contrary to the assumption that higher interest always leads to more effort, respondents with "medium" interest levels actually provided the longest answers. Those classified with very low or very high predicted interest tended to be more concise.
Voice vs. Text for Behavior Prediction
Vocal cues—the actual sound of the answer—proved to be significantly stronger predictors of answer behavior (word count and response time) than the sentiment of the spoken words. While the interest model explained approximately 12% of the variance in answer length, text sentiment explained almost none. This suggests that non-verbal acoustic features capture dimensions of respondent behavior that text analysis misses.
The Self-Report Disconnect
There was only a weak association between the interest predicted from voice data and the interest respondents reported themselves at the end of the survey. This suggests a distinction between "survey interest" (a global, stable trait) and "question interest" (a fleeting, affective state). Voice data appears uniquely suited to capturing the latter.
Limitations and Future Directions
While promising, the study highlights several challenges in "cross-corpus" application—using a model trained on product demonstrations (AVIC) to analyze political survey responses. Future research should focus on building "in-corpus" models specifically trained on survey-response data to improve predictive accuracy.
Furthermore, since vocal intonation and interest cues can be culturally and linguistically specific, these findings (based on a German sample) may not immediately generalize to other regions. However, the speed at which voice data can be processed opens the door for real-time survey interventions. In the future, surveys could use interest recognition to dynamically change question flow or offer encouragement to prevent respondent dropout when engagement flags.
Summary updated