That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets

William Yang Wang, Diyi YangEMNLP 2015 · OpenReview

Abstract

We propose a novel data augmentation approach to enhance computational behavioral analysis using social media text. In particular, we collect a Twitter corpus of the descriptions of annoying behaviors using the #petpeeve hashtags. In the qualitative analysis, we study the language use in these tweets, with a special focus on the fine-grained categories and the geographic variation of the language. In quantitative analysis, we show that lexical and syntactic features are useful for automatic categorization of annoying behaviors, and frame-semantic features further boost the performance; that leveraging large lexical embeddings to create additional training instances significantly improves the lexical model; and incorporating frame-semantic embedding achieves the best overall performance.

Introduction

Traditional behavioral science often operates as "small-data" science, relying on expensive, time-consuming laboratory experiments with limited participant pools. This approach often fails to capture the geographic diversity or the niche behaviors present in the real world. While social media provides a massive repository of human behavior, analyzing it presents significant challenges: tweets are short, noisy, and data for specific behaviors—like "pet peeves"—is often too sparse to train robust machine learning models.

This paper addresses the challenge of the automatic categorization of annoying behaviors. The goal is to bridge the gap between raw social media text and structured behavioral insights by creating a system capable of accurately sorting tweets into 60 distinct categories of annoyance, ranging from "interruption" and "hygiene" to "punctuality."

Methodology

The researchers employed a dual approach combining qualitative analysis to understand the linguistic landscape and quantitative modeling to automate classification.

Qualitative Analysis

The team utilized the Sparse Additive Generative Model (SAGE), a supervised topic model, to identify salient words for different categories and visualize how pet peeves vary by geography. This allowed for the identification of regional linguistic markers associated with annoyance.

Feature Engineering and Modeling

Beyond standard unigram word counts, the researchers extracted several layers of linguistic features:

  • Part-of-Speech (POS) tags to capture shallow syntax.
  • Dependency Triples to identify deeper grammatical relationships (e.g., nsubj(hate, I)).
  • Frame-Semantics using the SEMAFOR parser to identify underlying semantic structures based on FrameNet.

Embedding-Based Data Augmentation

To overcome data sparsity, the researchers introduced a novel data augmentation technique. They used Word2Vec to identify the $k$-nearest neighbors ($knn$) for words in the training set. For example, a tweet like "Being late is terrible" could generate a synthetic instance like "Be behind are bad." This was applied to both raw lexical items and semantic frames, creating a "continuous bag-of-frames" model that significantly expanded the training signal.

Dataset

The study utilized a filtered subset of a 9-million tweet dataset, resulting in 3,375 tweets containing the #petpeeve hashtag. These were manually labeled into 60 fine-grained categories, providing a gold-standard benchmark for behavioral NLP. For the augmentation process, the researchers leveraged pre-trained embeddings from Google News (100 billion words), Twitter (51 million words), and Urban Dictionary (53 million words of slang).

Key Insights and Geographic Variation

A major finding of the qualitative analysis was the significant role of regional culture and slang in expressing annoyance. Standard news-based language models often miss these nuances, which are better captured by specialized embeddings like those from Urban Dictionary.

Top topical words by state showing regional variations in pet peeves.

As illustrated in the table above, geographic "Aha" moments were frequent. For example, the term "dmv" (referring to the Department of Motor Vehicles) was a top annoyance in the DC/Maryland/Virginia area, while terms like "rims" and "daddy" were prominent markers of unique car and social cultures in Florida.

Quantitative Results

The quantitative evaluation demonstrated that while syntax (POS and dependencies) provided minor improvements, frame-semantic features offered a much larger boost. This suggests that the underlying meaning and context of an action are more predictive of annoyance than sentence structure.

Performance comparison of different data augmentation methods.

As shown in the results table, the baseline lexical model achieved an F1 score of 0.341. Data augmentation using Google News embeddings improved this baseline by 6.1%. The best overall performance was achieved by combining all linguistic features with both lexical and frame-semantic data augmentation, reaching a top F1 score of 0.380—a 3.8% improvement over the non-augmented all-features baseline.

Conclusion and Future Work

While the task remains challenging due to the inherent subjectivity of annoyance (reflected in a moderate inter-annotator agreement of 0.445), the proposed augmentation methods provide a necessary "patch" for small-sample behavioral data. Future work will explore applying these techniques to analyze workplace environments or public reactions to social movements, providing low-cost, real-time insights for social and organizational psychology.

Summary updated

Abstract

We propose a novel data augmentation approach to enhance computational behavioral analysis using social media text. In particular, we collect a Twitter corpus of the descriptions of annoying behaviors using the #petpeeve hashtags. In the qualitative analysis, we study the language use in these tweets, with a special focus on the fine-grained categories and the geographic variation of the language. In quantitative analysis, we show that lexical and syntactic features are useful for automatic categorization of annoying behaviors, and frame-semantic features further boost the performance; that leveraging large lexical embeddings to create additional training instances significantly improves the lexical model; and incorporating frame-semantic embedding achieves the best overall performance.
BibTeX
@inproceedings{wang2015that,
  author = {William Yang Wang and Diyi Yang},
  title = {{That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using \#petpeeve Tweets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=r1NRyfzuWB},
}
Figure