Abstract
For both individual politicians and political parties the internet has become a vital tool for self-promotion and the distribution of ideas. The rise of streaming has enabled political debates and speeches to reach global audiences. In this paper, we explore the nature of charisma in political speech, with a view to automatic detection. To this end, we have collected a new database of political speech from YouTube and other on-line resources. Annotation is performed both by native listeners, and Amazon Mechanical Turk (AMT) workers. Detailed analysis shows that both label sets are equally reliable. The results support the use of crowd-sourced labels for speaker traits such as charisma in political speech, even where cultural subtleties are present. The impact of these different annotations on charisma prediction from political speech is also investigated.
Introduction
Charisma is a vital yet elusive trait for political success. While often described as a "know it when you see it" quality, quantifying charisma for machine learning remains a significant challenge. Existing speech databases are often limited in size or restricted to specific events. Furthermore, a central question in paralinguistics is whether culturally nuanced traits like charisma can be reliably crowdsourced. This study investigates whether global Amazon Mechanical Turk (AMT) workers can serve as a reliable substitute for local "native" listeners when evaluating the charisma of an Irish politician.
The Irish Political Speech Database
The researchers developed a new audio-visual database consisting of 1,456 sentences from 47 recordings of Enda Kenny, the Irish Prime Minister at the time. The data spans from 2006 to 2012 and includes diverse settings such as parliamentary addresses, election rallies, and media interviews. This longitudinal approach allows for the study of how a politician's speaking style evolves over a career.
Annotation and Cross-Cultural Reliability
To validate crowdsourcing, the study compared ratings from a "native" cohort (Irish university students) against AMT workers (primarily US-based). Raters used a 5-point Likert scale to evaluate four attributes: charisma, enthusiasm, inspiration, and likeability. Inter-rater agreement was found to be comparable between the two groups, with native raters achieving a Cohen's kappa of 0.48 and AMT raters achieving 0.44.
As illustrated in the rating distributions, the perception of charisma and enthusiasm was remarkably consistent between the two groups. This suggests that these traits are signaled through universal vocal cues rather than purely local cultural context.
Key Results: The "Likeability" Gap
The analysis revealed a high correlation between native and AMT ratings for most traits, supporting the validity of crowdsourcing for speaker trait annotation. However, a significant discrepancy emerged regarding "likeability."
While charisma (rho = 0.76) and enthusiasm (rho = 0.84) showed strong cross-cultural agreement, likeability dropped to rho = 0.49. This "likeability gap" suggests that native listeners may be biased by prior knowledge of the politician's policies or persona, whereas AMT workers provided a more "pure" assessment of vocal pleasantness.
Charisma Prediction
The researchers trained a Multi-Layer Perceptron (MLP) to predict charisma scores using prosodic features (pitch, rhythm, intensity) and Mel-frequency cepstral coefficients (MFCCs). Prosodic features were found to be superior predictors, achieving a correlation of rho = 0.66 with true labels, which is comparable to state-of-the-art results on other charismatic speech corpora.
Conclusion
This work demonstrates that crowdsourced labels are as reliable as native listener ratings for most speaker traits in political speech. While "likeability" remains highly subjective and culturally dependent, traits like charisma and enthusiasm appear to be perceived universally through vocal prosody. Future work will expand the database to include more speakers and explore noise-robust feature extraction methods.
Summary updated