Abstract
People's values provide a decision-making framework that helps guide their everyday actions. Most popular methods of assessing values show tenuous relationships with everyday behaviors. Using a new Amazon Mechanical Turk dataset (N = 767) consisting of people's language, values, and behaviors, we explore the degree to which attaining "ground truth" is possible with regards to such complicated mental phenomena. We then apply our findings to a corpus of Facebook user (N=130,828) status updates in order to understand how core values influence the personal thoughts and behaviors that users share through social media. Our findings suggest that self-report questionnaires for abstract and complex phenomena, such as values, are inadequate for painting an accurate picture of individual mental life. Free response language data and language modeling show greater promise for understanding both the structure and content of concepts such as values and, additionally, exhibit a predictive edge over self-report questionnaires.
Introduction: The Value-Behavior Gap
Psychologists have long relied on the Schwartz Value Survey (SVS) to measure the ten core values—such as Achievement, Hedonism, and Tradition—that supposedly guide human decision-making. However, a persistent "value-behavior gap" exists: what individuals claim to value in a structured survey often correlates poorly with their actual daily actions. Traditional surveys are "top-down" and restrictive, forcing participants to map their complex internal lives onto a predetermined list of abstract concepts. This study explores whether natural language—the words people use to describe their lives—provides a more accurate "ground truth" for human values.
Methodology: The Meaning Extraction Method (MEM)
To capture values from a "bottom-up" perspective, researchers utilized the Meaning Extraction Method (MEM). Unlike standard topic modeling (e.g., LDA), MEM identifies psychologically meaningful themes by analyzing the co-occurrence of content words (nouns, verbs, adjectives) while ignoring function words (e.g., "the", "and"). This allows for the discovery of themes that are naturally salient to the participants rather than imposed by a researcher.
The study compared two data types across two projects (AMT workers and Facebook users):
- Closed-format: Traditional Likert-scale scores from the 57-item SVS.
- Open-ended: Natural language from essays and status updates, quantified by the frequency of words associated with MEM-derived themes like "Honesty," "Family Growth," and "Work."
Key Insights: Language as a Superior Predictor
The research revealed a significant "semantic breadth gap." When writing freely, individuals describe values (e.g., "Certainty" or "Caring") that do not always align neatly with the ten Schwartz categories. Language-based metrics proved to be far more potent predictors of behavior. In the essay-based study, SVS scores predicted only 6 out of 27 behavioral categories, whereas language-derived values predicted 14.
A notable case study, "Participant Z," illustrated this discrepancy. While this individual's SVS scores indicated no strong values, their essay revealed a deeply coherent and moral guiding framework. This suggests that standard "top-down" instruments may fail to register clear values that are otherwise evident in free-form expression.
Results: Behavioral Coverage in Social Media
The findings were further validated using a massive dataset of Facebook status updates. While SVS scores showed minimal correspondence with the behaviors and ideas users shared online, language-derived value themes exhibited "extensive coverage."
As shown in the correlation matrix above, MEM-derived values (bottom section) show a much denser network of significant relationships with behavioral topics (e.g., "Religiosity," "Work," "Family") compared to the traditional Schwartz values (top section). This indicates that the language people use naturally reflects their underlying values and subsequent behaviors more effectively than self-report surveys.
Conclusion
Traditional psychological surveys often fail to connect abstract values to everyday behaviors because they are too restrictive. This research demonstrates that natural language is a more accurate reflection of internal value systems. By moving from "check-the-box" questionnaires to language modeling, researchers can better understand the structure and content of human values and their real-world impact. Future work should explore these links across diverse cultures and move toward even finer-grained models of micro-behaviors.
Summary updated