When Do Words Matter? Understanding the Impact of Lexical Choice on Audience Perception Using Individual Treatment Effect Estimation

Zhao Wang, Aron CulottaAAAI 2019 · DOI

Introduction

Word choice is a powerful tool in communication. Swapping "excited" for "stoked" or "cheap" for "inexpensive" does more than change the literal meaning; it shifts how an audience perceives the speaker's identity or the desirability of the subject. However, quantifying these subtle shifts—the Lexical Substitution Effect (LSE)—has historically required expensive, small-scale human trials. This paper introduces a framework to automate LSE estimation by adapting causal inference techniques, allowing for scalable analysis of how specific word changes affect perception in context.

The Challenge of Scalability

The gold standard for measuring causal effects in linguistics is the Randomized Controlled Trial (RCT). In an RCT, human participants rate two versions of a sentence that differ by only one word. While effective, this method cannot scale to the millions of possible word substitutions across diverse contexts. Previous research often relied on global word associations, which fail to capture context-specific effects. For instance, the impact of changing "boyfriend" to "buddy" depends heavily on whether the sentence is personal or professional.

Examples of word substitutions designed to shift desirability and gender perception.

Methodology: Individual Treatment Effect (ITE)

The authors frame LSE as an Individual Treatment Effect (ITE) estimation problem. Borrowing from medical literature—where ITE asks how a specific patient reacts to a drug—this framework asks how a specific sentence's perception changes when word $w_i$ (control) is replaced by $w_j$ (treatment).

Two primary approaches were evaluated:

  1. Adapted ITE Estimators: Four causal inference algorithms were adapted for text:
  2. K-Nearest Neighbor (KNN) Matching: Estimates counterfactuals using similar sentence contexts.
  3. Virtual Twins Random Forest (VT-RF): Predicts outcomes for both original and substituted "twin" sentences using a single forest.
  4. Counterfactual Random Forest (CF-RF): Uses separate forests for control and treatment words.
  5. Causal Forest (CSF): A specialized forest that optimizes for the variance in treatment effects.
  6. Causal Perception Classifier: A meta-classifier trained on a small RCT dataset. It leverages features like "context probability" (the degree to which surrounding words already signal an attribute) to predict the magnitude of perceptual shifts.

Experimental Setup

The framework was tested across three datasets representing different perceptual attributes:

  • Yelp & Twitter: Perceived author gender.
  • Airbnb: Neighborhood desirability, using FBI crime statistics as a proxy.
  • Human Validation: A benchmark of 687 sentence pairs rated by 720 Amazon Mechanical Turk workers served as the ground truth.

Key Results and Insights

The study demonstrates that automated models can closely mirror human perception. The Causal Perception Classifier performed exceptionally well, particularly on the Yelp dataset, where it achieved a Pearson correlation of 0.783 with human judgments.

Pearson correlation between algorithmic estimates and human ground truth across datasets.

Several critical insights emerged:

  • Contextual Sensitivity: A word's impact is not static. Models successfully identified that word changes have a "ceiling effect"; if a sentence is already saturated with gendered cues, an additional word change has a diminishing impact.
  • Domain Generalization: Models trained on Yelp data generalized effectively to Twitter, suggesting a consistent underlying logic to how lexical changes shift social perception.
  • Subjectivity Barriers: Performance was lower on the Airbnb dataset. The authors suggest that "desirability" is inherently more subjective and harder to model than identity markers like gender.

Conclusion

By bridging the gap between observational text data and causal inference, this work provides a scalable method for understanding the nuances of language. While currently limited to single-word synonyms from the Paraphrase Database (PPDB), future work could expand this to multi-word substitutions and more complex stylistic shifts, further refining our understanding of when and why words matter.

Summary updated

BibTeX
@inproceedings{wang2019when,
  author = {Zhao Wang and Aron Culotta},
  title = {{When Do Words Matter? Understanding the Impact of Lexical Choice on Audience Perception Using Individual Treatment Effect Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33017233},
}
Figure