When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity

Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel ChochoyEMNLP 2025

TL;DR

This paper introduces Topological Data Analysis (TDA) using the Mapper algorithm to explore how language models represent ambiguity in text. They found that fine-tuning models creates structured, non-convex regions in embedding space that align with predictions, even when human annotators disagree, offering a new way to diagnose model behavior beyond traditional metrics.

What Problem Does This Paper Address?

Understanding how language models (LMs) handle ambiguity is crucial, especially in tasks where human annotators frequently disagree. Traditional evaluation metrics like accuracy often fail to capture the nuances of how LMs internally represent these ambiguous cases. While fine-tuning LMs can improve performance, it's unclear how it reshapes the internal representation space, particularly concerning subjective or ambiguous inputs. Existing geometric tools like PCA and UMAP can oversimplify high-dimensional structures, potentially flattening important topological relationships. This paper aims to bridge this gap by using a topological approach to visualize and understand how LMs encode and resolve ambiguity in their internal representations.

Methods

The core of this paper is the application of the Mapper algorithm, a technique from Topological Data Analysis (TDA). Mapper works by:

  1. Filtering: Projecting high-dimensional data (LM embeddings) into a lower-dimensional space using a "lens function." This function highlights specific geometric or semantic properties.
  2. Covering: Dividing the projected space into overlapping intervals (hypercubes).
  3. Clustering: Clustering the original data points that fall into each interval.

Each cluster becomes a node in a Mapper graph, and edges connect nodes that share data points, revealing the overall structure of the data. The authors used several 1D lens functions (e.g., centroid projection, PCA, eccentricity, L2 norm, random projections) to explore different aspects of the embedding space. They also employed HDBSCAN for clustering, which is robust to varying densities.

Datasets

The primary dataset used is MD-Offense (Leonardelli et al., 2021). This dataset consists of 10,753 English tweets annotated for offensive language detection by five U.S. annotators per instance. The data spans three domains: COVID-19, the 2020 U.S. presidential election, and racial justice. Instances are labeled with a binary ground-truth label (offensive/non-offensive) based on majority vote and also categorized by annotator agreement level (A++, A+, A0), where A0 indicates high disagreement and thus greater ambiguity.

Key Insights and Ideas

  • Fine-tuning creates topological regularity: Even for ambiguous cases, fine-tuning restructures the embedding space into more organized, non-convex regions.
  • Models impose structure on ambiguity: Instead of just reflecting noisy labels, LMs seem to impose their own structured certainty on ambiguous inputs by grouping them into coherent "prediction regions."
  • Mapper reveals hidden tensions: While fine-tuning increases prediction purity within components, the alignment with ground-truth labels can drop in ambiguous data, highlighting a tension between structural confidence and label uncertainty.
  • Geometric tools can be misleading: PCA and UMAP can flatten complex, multimodal structures, while Mapper preserves connectivity and reveals fragmentation or overlaps that are crucial for understanding ambiguity.
  • Lens choice matters: Different lens functions highlight different aspects of the data, and the choice of lens can influence the resulting topological structure, underscoring the need for careful selection.

Key Results

  • High Prediction Purity: After fine-tuning, over 98% of connected components in the Mapper graphs were "prediction-pure" (i.e., dominated by a single predicted label), even in the most ambiguous A0 subset.
  • Strong Majority Match: Despite the high prediction purity, the predicted majority label still matched the ground-truth majority label for a significant proportion of components (77.7% in A0, and over 94% in A+ and A++).
  • Modular, Non-Convex Regions: Fine-tuning reshaped the embedding space into modular, non-convex regions that align with model predictions, rather than simple, linearly separable clusters.
  • Robustness: The observed topological patterns were consistent across different 1D and 2D lenses, and also across various encoder-only models (BERT-Base, DeBERTa-Base, DistilBERT, E5-Base, MiniLM), suggesting the findings are not model or projection-specific.

Limitations and Future Work

  • Mapper Sensitivity: The Mapper algorithm is sensitive to its hyperparameters (lens function, covering resolution, clustering strategy), requiring careful calibration and robustness checks. Topological patterns should be interpreted relatively, not as absolute truths.
  • Task and Dataset Specificity: The study focused on offensive language detection on the MD-Offense dataset. It's an open question whether similar topological phenomena would occur in other tasks (e.g., NLI, sentiment analysis) or with different types of ambiguity.
  • Lens Expressiveness: The study primarily used simple, interpretable lenses. More complex, task-specific, or uncertainty-aware lenses could potentially reveal richer structures.
  • Metric Interpretation: The proposed topological metrics (Component Purity, Edge Agreement, Majority Match) capture interpretable geometric trends but are Mapper-specific and don't directly correlate with classification accuracy or generalization. Improvements could stem from genuine learning or overfitting.
  • Future Work: The authors suggest exploring hybrid pipelines combining topological summaries with statistical validation, investigating alternative graph structures, and conducting multi-model, multi-task replication. They also propose using Mapper for proactive modeling strategies, such as guiding subset selection or fine-tuning regimes for ambiguous regions.

Summary updated

BibTeX
@inproceedings{rair2025when,
  author = {Nisrine Rair and Alban Goupil and Valeriu Vrabie and Emmanuel Chochoy},
  title = {{When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure