The Value of Nothing: Multimodal Extraction of Human Values Expressed by TikTok Influencers

Alina Starovolsky-Shitrit, Alon Neduva, Naama Appel Doron, Ella Daniel, Oren TsurarXiv 2025 · DOI · arXiv

Abstract

As social media influencers increasingly shape the moral and social landscape for youth, understanding the values they transmit is critical. This paper introduces the first TikTok dataset annotated for human values using the Schwartz Theory of Personal Values. We evaluate two primary extraction strategies: direct multimodal analysis using Large Language Models (LLMs) and a two-step "video-to-script-to-classification" pipeline. Our results demonstrate that supervised Masked Language Models (MLMs) operating on detailed textual scripts significantly outperform direct multimodal LLM extraction, particularly in identifying nuanced or contradicted values.

Introduction

In the contemporary digital age, children and adolescents are increasingly turning to social media influencers as primary sources of societal and personal values, often superseding traditional influences like parents or educators. While previous computational social science research has successfully extracted values from text-centric platforms like Reddit or Twitter, short-form video platforms like TikTok present a unique multimodal challenge. Identifying implicit values—such as "Achievement," "Hedonism," or "Tradition"—requires interpreting a complex interplay of visual cues, audio, dialogue, and on-screen text. This study addresses the gap in automated value extraction for video content, exploring whether AI should engage with raw video directly or utilize intermediate textual representations.

Methodology

The authors frame value extraction as a multi-label classification task based on the 19 core values defined by the Schwartz Theory of Personal Values. To determine the most effective approach, two distinct computational pipelines were developed and tested:

  1. Direct Extraction: This approach feeds the raw video file (.mp4) directly into a state-of-the-art multimodal LLM (Gemini 1.5 Pro). The model is prompted in a few-shot manner to identify whether specific values are present or contradicted within the content.
  2. 2-Step Extraction (Indirect): This pipeline involves an intermediate representation. First, Gemini 1.5 Pro converts the video into a comprehensive textual script that captures dialogue, visual actions, facial expressions, and on-screen text. Subsequently, this script is passed to either a supervised Masked Language Model (MLM), such as RoBERTa-large or DeBERTaV2, or a few-shot LLM (Mistral 7B, Gemini) for final value classification.
Overview of the direct and 2-step extraction pipelines.

Dataset and Annotation

The study introduces a novel, manually annotated dataset comprising 890 TikTok videos. These videos were curated from 89 influencers across seven youth-oriented genres, including gaming, beauty, pranks, and DIY. Human raters annotated each video for the presence (1), absence (0), or contradiction (-1) of the 19 Schwartz values, resulting in a total of 2,968 value labels. This dataset represents the first of its kind for video-based social media and is intended to be open-sourced to facilitate further research in the field.

Results and Key Insights

The experimental results reveal several critical insights into the current state of multimodal value extraction:

  • Textual Mediation is Superior: The 2-step pipeline, which distills noisy multimodal signals into a rich text script, currently outperforms direct multimodal analysis. The intermediate script appears to provide a more digestible format for classification models.
  • Supervision vs. Few-Shot: Supervised MLMs, specifically RoBERTa-large, achieved the best overall performance with a weighted F-score of 0.47. This suggests that for highly nuanced tasks like value detection, explicit training on labeled data remains more effective than few-shot prompting of larger models.
  • The Challenge of Contradictions: Models generally performed better at identifying "present" values (Max F-score 0.71 for Achievement) than "contradicted" values. Detecting when a value is mocked or hindered proved significantly more difficult, with performance dropping across all models.
Radar charts comparing model performance for present and contradicting value manifestations.

Specific values like Achievement (F-score 0.71), Benevolence Dependability (0.61), and Security Personal (0.58) were the most accurately detected, while others remained elusive due to class imbalance and the subtlety of their expression.

Conclusion and Future Work

While the 2-step pipeline shows promise, it is not without limitations. Its success is heavily dependent on the quality of the generated script; missing visual context can lead to cascading errors in value extraction. Furthermore, the Schwartz theory, while robust in psychology, may require adaptation to capture the ironic and trend-driven nature of social media content. Future research will focus on addressing class imbalances and expanding the dataset to cover a broader range of TikTok's algorithmic ecosystem, ultimately providing tools for educators and policymakers to monitor the value landscapes of digital influencers.

Summary updated

BibTeX
@misc{starovolskyshitrit2025value,
  author = {Alina Starovolsky-Shitrit and Alon Neduva and Naama Appel Doron and Ella Daniel and Oren Tsur},
  title = {{The Value of Nothing: Multimodal Extraction of Human Values Expressed by TikTok Influencers}},
  year = {2025},
  eprint = {2501.11770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.11770},
}
Figure