Variationist: Exploring Multifaceted Variation and Bias in Written Language Data

Alan Ramponi, Camilla Casula, Stefano MeniniACL 2024

TL;DR

VARIATIONIST is a new, highly modular, and extensible Python tool designed to help researchers explore and visualize language variation and bias in textual data across multiple dimensions. It addresses the lack of a unified, customizable platform for deep data inspection, enabling users to uncover quality concerns and harmful biases before model training.

What Problem Does This Paper Address?

Understanding language data is crucial for fields ranging from Natural Language Processing (NLP) to linguistics and social sciences. NLP practitioners rely on this data to train large language models (LLMs), while linguists and social scientists use it to study human language and behavior. However, there's a significant gap: existing tools often lack the flexibility to seamlessly inspect and visualize language variation and bias across various dimensions (e.g., different text units, multiple variables, diverse metrics). Many tools are English-centric, offer limited customization, or focus on surface-level statistics and basic charts. This makes it challenging for researchers to uncover subtle quality issues or harmful biases hidden within the data, especially when dealing with complex, multi-dimensional relationships.

Methods

VARIATIONIST is built around a modular design, allowing users to customize their analysis pipeline. Its core components include:

  • Datasets: Supports various input formats like TSV, CSV, pandas DataFrames, and direct import from Hugging Face Datasets.
  • Texts: Can handle one or two text columns for comparative analysis.
  • Units: Flexible definition of language units, from characters to n-grams and co-occurrences, with options for preprocessing like lowercasing and stopword removal.
  • Tokenizers: Integrates default whitespace tokenizers, Hugging Face Tokenizers, and custom tokenizers to avoid language-specific assumptions.
  • Variables: Accommodates a wide range of variable types (nominal, ordinal, quantitative, coordinate) and semantics (temporal, spatial, general), allowing for rich meta-information integration.
  • Metrics: Implements various association metrics (e.g., PMI variants, relevance metrics) and lexical diversity measures (e.g., TTR, Maas' index), with the ability to define custom metrics.
  • Charts: Automatically generates interactive, multi-dimensional charts (up to five dimensions) using Altair, supporting various chart types like choropleth maps, scatter plots, and heatmaps.

The tool is designed for ease of use, modularity, and extensibility, enabling researchers to plug in their own custom functions and metrics.

Datasets

The paper demonstrates VARIATIONIST's capabilities using several datasets:

  • DIATOPIT: A corpus of geolocated social media posts in Italy, used for computational dialectology.
  • Measuring Hate Speech (MHS) corpus: Used for human label variation analysis, focusing on annotator demographics.
  • Human ChatGPT Comparison Corpus (HC3): Used for analyzing differences between human-written and AI-generated texts.

No new datasets were introduced.

Key Insights and Ideas

  • Unified Exploration: VARIATIONIST provides a single platform to explore language data across diverse disciplines, bridging the gap between NLP practitioners and linguists/social scientists.
  • Multi-dimensional Analysis: The ability to combine multiple variable types and semantics with various metrics allows for a much deeper understanding of language variation and bias than traditional tools.
  • Customization is Key: The modular design and support for custom tokenizers and metrics make VARIATIONIST highly adaptable to specific research needs and languages, moving beyond English-centric assumptions.
  • Unveiling Hidden Biases: By facilitating the exploration of associations between textual units and meta-information, the tool can help uncover subtle, potentially harmful biases in datasets.
  • Interactive Visualization: The automatically generated interactive charts allow for dynamic exploration and filtering, making complex data relationships more accessible.

Key Results

The paper presents case studies showcasing VARIATIONIST's effectiveness:

  • Computational Dialectology: Alice successfully identified the geographical distribution of the lexical item "ghe" in Italy, visualizing its use at different granularities using choropleth, geo scatter, and binned maps.
  • Human Label Variation: Bob discovered that annotators' sociodemographic characteristics influence their perception of hate speech, showing how "gay" is more indicative of hate speech for straight annotators and how reclaimed words like "n*ggas" are perceived differently across racial groups. Heatmaps were used to visualize these relationships.
  • Text Generation: Carol found that human-written texts are generally longer, have larger vocabulary diversity, and use more everyday language compared to ChatGPT-generated texts, which tend to be more formal. Summary statistics and bar charts comparing lexical variety (root_ttr) and informative bigrams (npw_pmi) were used.

Limitations and Future Work

  • Current Limitations: VARIATIONIST is currently limited to analyzing lexical-level features in written data.
  • Future Work: The authors plan to extend the tool's functionalities to include grammar, the speech modality, and additional visualization libraries. They also aim to integrate more advanced features for exploring specific linguistic phenomena and improving the user experience.

Summary updated

BibTeX
@inproceedings{ramponi2024variationist,
  author = {Alan Ramponi and Camilla Casula and Stefano Menini},
  title = {{Variationist: Exploring Multifaceted Variation and Bias in Written Language Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure