Specifying Similarity Metrics for Individual Fairness

179 concepts97 papers1 proposal

Algorithmic fairness is often framed in terms of groups: ensuring that the acceptance rates for a loan or a college admission are equalized across demographic categories like race or gender. However, group fairness metrics can overlook harms at the individual level, occasionally leading to situations where highly qualified individuals are passed over to satisfy aggregate quotas.

To address this, Cynthia Dwork et al. introduced the principle of individual fairness in their seminal paper, Fairness through awareness. The core intuition is simple and philosophically compelling: similar individuals should be treated similarly.

Mathematically, this is operationalized as a Lipschitz condition on a classification model $f$:

$$||f(x) - f(y)|| \leq d(x, y)$$

where $x$ and $y$ are two individuals, $||f(x) - f(y)||$ represents the difference in their algorithmic outcomes, and $d(x, y)$ is a task-specific metric that quantifies how similar $x$ and $y$ are for the given decision context. If the distance between two people is small, the distance between their outcomes must be correspondingly small.

While elegant, this formulation shifts the entire burden of fairness onto a single, external component: the similarity metric $d(x, y)$. If the metric itself is biased, incomplete, or poorly specified, the resulting fairness guarantees become meaningless.

The Metric Dilemma

In practice, defining $d(x, y)$ is the primary bottleneck of individual fairness. As Richard S. Zemel et al. point out in Learning Fair Representations, standard mathematical distance functions—such as Euclidean or Minkowski distances—fail to capture the nuanced, qualitative context required for ethical decision-making. For instance, in a credit scoring application, a simple Euclidean distance across all features might treat an extra year of work experience as equivalent to a minor drop in credit score, which may not align with a fair assessment of creditworthiness.

Furthermore, selecting these metrics is inherently subjective and value-laden. As demonstrated by Alexander Vargo et al. in Individually Fair Gradient Boosting, if an auditor or practitioner uses an incorrectly specified metric, the machine learning model will simply inherit and reinforce those underlying errors. Because human stakeholders rarely agree on exactly which features matter and by how much, constructing a universally accepted ground-truth metric from scratch is often impossible.

Learning Metrics from Human Feedback

Rather than forcing domain experts to manually write down complex mathematical formulas for $d(x, y)$, a prominent branch of research focuses on learning the similarity metric from data or human supervision.

One approach is to query human oracles for pairwise comparisons. In An Empirical Study on Learning Fairness Metrics for COMPAS Data with Human Supervision, Hanchen Wang et al. showed that even when human intuition is inconsistent or noisy, aggregating feedback on whether individual $A$ is similar to individual $B$ can yield robust, learned fairness metrics that outperform standard mathematical baselines.

Taking this a step further, Pragya Sur demonstrated in Abstracting Fairness: Oracles, Metrics, and Interpretability that it is possible to reverse-engineer the underlying similarity metric simply by observing which classifiers a "weak oracle" accepts or rejects as fair. By treating the collection of approved models as a proxy, the system mathematically reconstructs the coarse distance function guiding the oracle's judgments.

Fairness as Invariance and Robustness

Another way to circumvent the difficulty of defining a global metric is to reframe individual fairness as a problem of invariance or adversarial robustness. Instead of measuring the distance between every possible pair of individuals, these methods define a "sensitive subspace" or a neighborhood of impermissible variations.

For example, in SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness, Mikhail Yurochkin et al. treat individual fairness as invariance to perturbations along sensitive directions (such as changing a zip code or a name that correlates with race). The intuition here borrows directly from adversarial machine learning: a model is individually fair if its predictions do not change when an individual's features are perturbed within a defined sensitive set.

This connection is formalized by Matt Fredrikson et al. in Individual Fairness Revisited: Transferring Techniques from Adversarial Robustness. The authors flip the standard paradigm: instead of training a model to fit a predefined metric, they calculate the "minimal metric" that a model currently satisfies. They then apply randomized smoothing—a technique originally designed to defend neural networks against adversarial attacks—to guarantee that the model treats local neighborhoods of similar individuals consistently, without destroying predictive accuracy.

Standard Individual Fairness:
Given Metric d(x,y)  ──>  Enforce Smoothness  ──>  Fair Model

Invariance / Robustness Approach:
Define Sensitive Subspace  ──>  Apply Randomized Smoothing  ──>  Certified Local Fairness

Auditing and Enforcing Fairness

Once a similarity metric or subspace is established, how do we verify and enforce it across an entire dataset?

For auditing existing models, Mikhail Yurochkin et al. introduced the Fair Transport Hypothesis (FaiTH) in Auditing ML Models for Individual Bias and Unfairness. By leveraging optimal transport theory, FaiTH evaluates how much an entire distribution of outcomes shifts when mapped across a similarity metric, providing statistical confidence intervals to certify whether a model is individually biased.

If a model is found to be unfair, we can enforce individual fairness via post-processing without retraining the model from scratch. In Post-processing for Individual Fairness, Felix Petersen et al. represent a dataset as a graph where edges represent the similarity distances between individuals. By applying Graph Laplacian regularization, they "smooth" the model's predictions across the graph, ensuring that connected (similar) nodes receive similar outcomes.

Ultimately, the field has evolved from treating similarity metrics as fixed, static inputs to viewing them as dynamic, learnable structures. Whether through human feedback, subspace invariance, or graph-based smoothing, the goal remains the same: bridging the gap between abstract ethical definitions of equality and the mathematical constraints of machine learning models.

Key Foundational Work

Fairness Through Awareness (2011) — contributed 4 concept elements to this direction.
Fairness through awareness (2012) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 114 papers
  2. Version 1 · 2026-10-04 · 104 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_similarity_metric_specification_c704e16c,
  title = {similarity metric specification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-metric-specification/rdr_c704e16c59984dc2964e362992f6b945},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_similarity_metric_specification_c704e16c_v2,
  title = {similarity metric specification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-metric-specification/rdr_c704e16c59984dc2964e362992f6b945/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake