TL;DR
The paper introduces ABCDE ("Affect, Body, Cognition, Demographics, and Emotion"), a massive dataset containing over 400 million text instances from social media, blogs, books, and AI-generated text. Pre-annotated with 136 lexical and demographic features, ABCDE lowers the technical barrier for interdisciplinary researchers to analyze population-level trends in human behavior, psychology, and large language model (LLM) alignment.
What Problem Does This Paper Address?
In Computational Affective Science (CAS) and Computational Social Science (CSS), researchers use language data to track mental states, public sentiment, and demographic trends. However, the prerequisite steps—collecting massive text corpora, extracting metadata, and annotating linguistic features—are incredibly time- and labor-intensive.
While advanced natural language processing (NLP) toolkits exist, they typically require significant programming expertise, making them inaccessible to researchers in fields like sociology, psychology, and the digital humanities. Furthermore, existing tools rarely provide standardized, pre-computed datasets that allow for direct cross-platform or longitudinal comparisons. ABCDE addresses this gap by providing a massive, ready-to-analyze repository of text mapped to standardized features.
Methods
The authors developed an automated pipeline to extract 136 features across five thematic dimensions. Rather than using complex, black-box machine learning models—which excel at individual sentence classification but lack domain portability and interpretability—the pipeline relies on validated word-association lexicons, curated word lists, regular expressions, and heuristic rules:
- Affect & Emotion: Words are matched against established lexicons (NRC VAD, NRC Emotion Intensity, WCST, and NRC WorryWords) to compute average intensity, counts, and binary presence flags for valence, arousal, dominance, and discrete emotions (e.g., joy, anger, warmth).
- Body: Identifies Body Part Mentions (BPMs) using a curated list of 292 anatomical terms combined with possessive pronouns (e.g., "my heart", "her hand").
- Cognition: Categorizes 98 unigrams into 11 cognitive sub-processes (e.g., analyzing, remembering, problem-solving) based on Bloom’s Taxonomy.
- Demographics: Uses high-precision, first-person regular expressions to extract self-disclosed attributes such as age, occupation, gender, location, and religion.
Datasets
The ABCDE dataset aggregates over 400 million text instances across five distinct domains:
- Twitter (TUSC, 2015–2021): 45.2M geolocated tweets.
- Reddit (Pushshift, 2010–2022): 78.6M filtered posts (excluding adult content, promotions, and extreme lengths).
- Books (Google Books, 1800–2012): 177.1M 5-gram occurrences from English Fiction.
- Blogs (Spinn3r, 2008): 34.2M personal blog entries.
- AI-Generated Texts (2022–2025): 68.9M completions from 15 distinct datasets (e.g., WildChat, LMSYS-Chat, TinyStories), including human-LLM interactions, reasoning traces, and narratives.
Key Insights and Ideas
- The Rise of Embodied Language: By analyzing a 200-year span in fiction books, the authors discovered that mentions of body parts increased by an order of magnitude (from ~0.05% in 1800 to ~0.75% post-1980), while cognitive terms remained stable. This points to a long-term cultural shift toward embodied language in literature.
- The Sterile Nature of AI Text: AI-generated text rarely uses explicit emotion words (~0.01%) and significantly underexpresses cognitive terms (like "problem solving" or "explanation") compared to human-authored text on social media.
- Platform-Specific Demographics: Self-disclosure patterns reveal stark differences between platforms: Reddit users skew younger (late teens to late 20s) and frequently identify as creative or technical professionals (e.g., software engineers, musicians). Twitter users skew older (30s) and occupy public-facing roles (e.g., journalists, executives).
Key Results
- Emotion Word Density: Twitter exhibits the highest rate of explicit emotion words (e.g., ~1.4% joy tokens per instance), followed by Books and Reddit.
- Body Mentions: Reddit (8.16%) and Blogs (4.47%) contain the highest percentage of possessive body part mentions, whereas Books and AI text contain very few.
- Lexical Diversity: Twitter shows the highest lexical diversity (MTLD of 252.31), likely due to its short form and varied slang. Crucially, AI-generated text is substantially less lexically diverse (MTLD of 30.60) than human blogs (107.38) or Reddit posts (91.52), revealing a lack of vocabulary variation in LLM outputs.
Limitations and Future Work
- Language and Geographic Bias: The dataset is exclusively in English, and a large portion of the data originates from North American users, limiting its generalizability to other cultures.
- Static Lexicons: The pipeline relies on static lexicons, which do not account for semantic drift over time or domain-specific slang.
- Lack of Contextual Modeling: The word-level approach does not capture syntax, discourse structure, or compositional effects like negation (e.g., "not happy").
- Future Directions: The authors plan to integrate moral and social perception dimensions (such as the NRC Moral Foundations Lexicon) and expand the dataset to include non-English corpora led by native speakers.
Summary updated