TL;DR
This paper introduces a novel protocol and the resulting Trust-UBA corpus for studying how trust in a virtual assistant (VA) is expressed through speech. By manipulating participants' initial perceptions of a VA's competence, researchers can collect speech data reflecting varying trust levels, paving the way for automatic detection of trust from vocal cues.
What Problem Does This Paper Address?
As virtual assistants become more sophisticated and integrated into our lives, understanding and responding to a user's mental state is crucial for effective human-computer interaction. One key aspect of this mental state is the user's trust in the VA's abilities. If a VA can detect cues of distrust, it could potentially adapt its behavior to regain the user's confidence. However, there's a significant lack of annotated speech corpora that capture different degrees of trust, making it difficult to develop and evaluate automatic trust detection systems. This paper aims to fill that gap by creating such a corpus.
Methods
The core of this work is a carefully designed protocol to induce varying levels of trust in participants. The protocol involves an interactive speech-based session where participants ask a VA factual questions. Before starting, participants are "primed" with information about the VA's supposed performance: either a very high rating (4.9/5) or a very low rating (1.4/5) from previous users. This initial "bias" is intended to anchor their perception.
The VA's responses are then tailored to reinforce this initial bias. In the "high-score" condition, the VA answers all difficult questions correctly, while in the "low-score" condition, it makes mistakes on some difficult questions. Easy questions are always answered correctly to avoid unnecessary frustration. Participants then provide their own answers and rate their confidence in the VA's responses. This entire interaction is recorded.
To further analyze the effectiveness of the protocol, participants also complete evaluation surveys at different points during the session, rating their confidence in the VA and providing qualitative feedback.
Datasets
The primary contribution is the Trust-UBA Database, a newly collected speech corpus in Argentine Spanish. This corpus consists of 8493 short audio recordings from 160 participants. The data was collected using the described protocol, resulting in speech samples from participants who were intentionally led to trust or distrust the virtual assistant. The paper notes that the dataset will be made publicly available.
Key Insights and Ideas
- Anchoring Bias is Effective: The protocol successfully leveraged the anchoring bias to influence participants' trust levels. Participants' self-reported trust scores were significantly different between the high-score and low-score conditions, indicating the manipulation was effective for the majority.
- Speech Rate as a Potential Trust Indicator: Preliminary analysis suggests that speech rate might be a feature that varies with the degree of trust. Participants in the high-trust condition tended to exhibit higher syllable rates, hinting that vocal characteristics could indeed reflect a user's trust state.
- Human Perception of Trust is Difficult: Even expert annotators struggled to consistently identify the degree of trust from speech alone, achieving only "slight" agreement above chance. This highlights the subtlety of trust cues in speech and the challenge of automatic detection.
Key Results
- The protocol effectively induced different degrees of trust in participants, as evidenced by significantly different self-reported trust scores between the high-score (mean trust ≈ 1.5) and low-score (mean trust ≈ -0.5) conditions after normalization.
- A preliminary study on speech rate showed statistically significant differences in syllable rate distributions for 4 out of 16 subjects between the high- and low-score conditions, with higher rates in the high-score condition.
- Inter-annotator agreement among psychology experts for identifying trust from speech was low (Fleiss' κ = 0.116), though statistically significant, indicating a faint but detectable signal.
Limitations and Future Work
- Limited Annotation: The current corpus lacks explicit annotations for trust levels within the speech data itself. The trust levels are inferred from the experimental conditions and participant self-reports.
- Expert Agreement: The low agreement among expert listeners suggests that identifying trust from speech is a challenging task, and further research is needed to pinpoint reliable acoustic and prosodic features.
- Future Work: The authors plan to use the Trust-UBA corpus for machine learning experiments to automatically detect the degree of trust from speech. They also aim to explore other acoustic and prosodic features beyond speech rate and investigate how trust influences other aspects of speech, such as intonation and voice quality. The public release of the corpus is also a key next step.
Summary updated