Say 'YES' to Positivity: Detecting Toxic Language in Workplace Communications

Meghana Moorthy Bhat, Saghar Hosseini, Ahmed Hassan Awadallah, Paul N. Bennett, Weisheng LiEMNLP 2021 · DOI

Abstract

Workplace toxicity often manifests as subtle, implicit behaviors like negative gossip, sarcasm, and microaggressions, which standard hate-speech detection models frequently miss. We introduce ToxiScope, a new dataset of over 10,000 sentences from real-world corporate emails, categorized by a novel workplace-specific taxonomy. Our research demonstrates that context is crucial for detecting these nuanced harms and that current state-of-the-art models require domain-specific fine-tuning to effectively identify the "mild yet hurtful" interactions that erode organizational morale.

Introduction

Most research into toxic language focuses on social media, where toxicity is often "explicit"—characterized by profanity, slurs, and overt threats. However, workplace toxicity is a different beast. It is frequently "implicit": subtle, hidden behind professional jargon, or embedded in sarcasm and negative gossip. While a single rude email might seem minor, studies show that over 80% of productivity issues stem from negative work environments.

Example of implicit workplace toxicity in an email thread.

Current tools, like Google’s Perspective API, are trained on public forums and often fail to flag workplace toxicity because it lacks the "loud" signals of typical online abuse. There is a massive gap in both data and methodology for identifying the kind of "mild yet hurtful" conversations that erode office morale.

Methodology and Dataset

To bridge this gap, we developed a hierarchical annotation framework and a new dataset, ToxiScope.

Taxonomy Development

We defined a workplace-specific taxonomy categorized into three main toxic types: Impolite, Negative Gossip, and Offensive. Each category includes specific sub-types to capture the nuance of professional interactions.

Workplace-specific toxicity taxonomy with examples.

Data Collection and Modeling

ToxiScope consists of 10,110 sentences derived from the Avocado Research Email Collection (a corpus from a real, defunct IT company). Because toxic emails are rare, we used a two-round approach involving heuristics and an SVM classifier to surface likely toxic candidates for human annotation.

We evaluated several architectures, including linear baselines, context-aware GRU models, and BERT-based classifiers. Notably, we developed a BERT+MLP model that incorporates context by concatenating the target sentence with the preceding email in the thread.

Key Insights and Results

Our analysis revealed that context is essential: in 41% of cases, annotators required the previous email to determine if a sentence was toxic. This is particularly true for gossip and sarcasm, where the words themselves might appear neutral in isolation.

Performance Benchmarks

  • Top Performer: The BERT + MLP model achieved the best results on ToxiScope with an F1-macro score of 0.7696 and an overall accuracy of 71.12%.
  • The Recall Challenge: Even the best models had a relatively low True Positive Rate (TPR) of ~0.43 for toxic sentences, highlighting the difficulty of catching subtle toxicity without excessive false positives.
  • Domain Mismatch: Models trained on Wikipedia or Twitter perform poorly on ToxiScope. For instance, the sentence "Your girlfriend needs a lot of help..." (see Figure 1) was flagged as toxic by humans but received a very low toxicity score (0.15) from the Perspective API.

Conclusion and Future Work

ToxiScope captures a broader, more useful definition of "implicit" toxicity than existing benchmarks. However, toxicity remains highly subjective. Future work will investigate how corporate power dynamics influence the perception of toxicity and explore long-term patterns of toxic behavior beyond single-sentence detection.

Summary updated

BibTeX
@inproceedings{bhat2021say,
  author = {Meghana Moorthy Bhat and Saghar Hosseini and Ahmed Hassan Awadallah and Paul N. Bennett and Weisheng Li},
  title = {{Say 'YES' to Positivity: Detecting Toxic Language in Workplace Communications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.173},
}
Figure