Lexical Selection Bias in Dataset Construction

140 concepts128 papers1 proposal

When building datasets to study online discourse, public health trends, or corporate risks, researchers face a fundamental chicken-and-egg problem: to analyze a phenomenon, they must first find it. The standard solution is to query a database or social media API using a curated list of keywords or hashtags.

While intuitive, this approach introduces an invisible, systematic filter known as lexical selection bias. By defining the boundaries of a dataset through a static set of explicit terms, we inadvertently skew the data along demographic, behavioral, and semantic lines. The resulting models do not learn the underlying concept; instead, they learn the narrow vocabulary of the initial query.

The Mechanics of the Filter

Why does keyword filtering distort our data? The core issue is that explicit markers—like hashtags or formal technical terms—are not distributed randomly across populations or sentiments. They are highly behavioral.

Consider how people use hashtags. As Minglei Li et al. demonstrate in their work on Emotion Corpus Construction Based on Selection from Hashtags, relying on hashtags as natural labels skews data toward highly expressive, explicit content. Nuanced or implicit emotional expressions rarely feature overt tags, leaving them entirely unrepresented.

This skew becomes even more pronounced when tracking social attitudes. In a longitudinal study of social solidarity, Steffen Eger et al. discovered that sampling data exclusively through popular hashtags systematically over-represents pro-social sentiment. The reason is behavioral: hostile or anti-social discourse frequently occurs in posts without organized hashtags. Consequently, a hashtag-based dataset creates a false impression of community harmony by hiding exclusionary content outside the tagged ecosystem.

Furthermore, these explicit markers introduce demographic and topic biases. Christoph Leiter et al. found that hashtag-driven sampling skews datasets toward highly engaged or tech-savvy users, missing organic public discourse. Similarly, Chris Chinenye Emezue et al. note that hashtag- and handle-based collection strategies in multilingual contexts heavily over-represent news and celebrity content, failing to capture everyday conversational language.

[Entire Online Discourse]
       │
       ├──> [Posts with Explicit Hashtags] ──> Skewed toward pro-social, explicit, tech-savvy users
       │
       └──> [Posts without Hashtags]       ──> Contains subtle, nuanced, or hostile discourse (LOST)

The Evolving Lexicon Bottleneck

Even when avoiding hashtags in favor of standard keywords, datasets quickly decay due to the dynamic nature of language. Static keyword lists assume a frozen vocabulary, but real-world communities constantly adapt their terminology.

This bottleneck manifests in three distinct ways:

  1. Orthographic Variation and Censorship Circumvention: Online communities frequently alter spellings to maintain identity or bypass platform filters. Jacob Eisenstein et al. document this in #anorexia, #anarexia, #anarexyia: Characterizing Online Community Practices with Orthographic Variation, showing that newcomers drive creative misspellings to evade censorship. A static keyword search fails to capture these isolated linguistic evolutions.
  2. Colloquialisms vs. Technical Terms: In public health monitoring, relying on formal terminology isolates the dataset from the target population. Qiudan Li et al. observe that technical keyword lexicons exclude massive portions of relevant discourse where users rely on evolving slang or colloquialisms to describe symptoms.
  3. Domain Jargon: This issue extends beyond social media. In software engineering, Ruiyin Li et al. show that mining code reviews using standard architectural keywords misses critical design critiques because developers frequently use informal, project-specific jargon.

When we rely on fixed keywords, we draw an arbitrary boundary around a topic. As Eric Rice argues in the context of analyzing public attitudes toward homelessness, narrow keyword sets create massive coverage gaps by missing synonymous terms and subculture-specific language.

Downstream Consequences: Spurious Shortcuts

The danger of lexical selection bias is not just that datasets are incomplete; it is that the models trained on them learn the wrong features. When a dataset is filtered by keywords, the presence of those keywords becomes highly correlated with the target label. The machine learning model exploits this correlation as a shortcut.

In their seminal analysis of abusive language detection, Michael Wiegand et al. exposed this vulnerability. Datasets collected via narrow keyword searches caused models to rely on irrelevant topical cues (such as the word "football") rather than an actual understanding of abusive language. When evaluated outside the narrow collection domain, the models' performance collapsed.

Keyword-Filtered Training Data:
[Post: "I hate football"] ──> Labeled as Abusive (due to keyword collection focus)
Model learns: "football" = Abusive

Real-World Deployment:
[Post: "Let's play football"] ──> Model flags as Abusive (Spurious Shortcut Failure)

This recall bottleneck also impacts advanced architectures like Large Language Models (LLMs). A study on clinical text snippet labeling demonstrates that using keyword pre-filtering to narrow down label candidates prevents the model from evaluating instances where a medical condition is described through indirect or non-standard terminology. The model never gets the chance to evaluate the text because the keyword filter blocked it at the gate.

Finally, manual keyword selection causes severe dataset fragmentation. Sacha Lévy demonstrates that different research teams investigating the exact same phenomenon often produce completely non-overlapping data samples simply due to minor variations in their initial, intuitive keyword choices. This lack of standardization makes replicating and generalizing findings across the field incredibly difficult.

Breaking the Lexical Bottleneck

To build representative datasets, researchers are shifting from static, human-curated keyword filters to dynamic, semantic sampling methods. The goal is to allow the data collection boundary to expand organically based on contextual meaning rather than exact string matches.

Static Filtering:  [Seed Keywords] ──> Strict String Match ──> Fragmented Dataset

Dynamic Expansion: [Seed Keywords] ──> Semantic Embeddings ──> Contextual Discovery ──> Representative Dataset

Dynamic Keyword Discovery

Instead of guessing terms upfront, researchers use continuous semantic expansion. Maya Srikanth et al. address the limitations of human intuition by using word embedding models (like GloVe) to dynamically discover new, relevant keywords in real-time. This approach allows data collection pipelines to adapt to rapidly shifting language patterns and adversarial communities without manual intervention.

Textual Entailment and Zero-Shot Expansion

Rather than filtering sentences by explicit keywords, models can evaluate the underlying meaning of a sentence. Ziqian Zeng et al. leverage a Textual Entailment model as an enhanced annotator to self-label massive amounts of text. By focusing on whether a sentence logically entails an event, the system captures metaphorical or novel language that falls entirely outside traditional seed keyword lists.

Reactive Supervision

When explicit markers are used, they can be leveraged indirectly to capture implicit data. Boaz Shmueli introduced Reactive Supervision to bypass the bias of self-labeled sarcasm hashtags (like #sarcasm). Instead of collecting posts containing the tag, the pipeline listens for conversational replies where a second user points out the behavior (e.g., "I can't tell if you're being sarcastic"). This captures subtle, natural examples that lack overt self-labels.

By shifting the focus from what words are used to what meaning is expressed, these frameworks mitigate lexical selection bias, ensuring that models learn robust concepts rather than superficial vocabulary shortcuts.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 114 papers
  2. Version 1 · 2026-10-04 · 194 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_keyword_based_selection_bias_5a99c7ca,
  title = {keyword-based selection bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/keyword-based-selection-bias/rdr_5a99c7ca6da84600b4c0e941b3ead8cf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_keyword_based_selection_bias_5a99c7ca_v2,
  title = {keyword-based selection bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/keyword-based-selection-bias/rdr_5a99c7ca6da84600b4c0e941b3ead8cf/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake