Metric differential privacy is a way to privatize text by randomly swapping each word for a nearby word in an embedding space, a geometric map where similar words sit close together. The guarantee is relative: two words are hard to tell apart in proportion to how close they are on that map. This makes the privacy parameter ε a peculiar number, because doubling every distance on the map has exactly the same effect as doubling ε. A reported ε = 5 on 50-dimensional GloVe vectors and a reported ε = 5 on 300-dimensional vectors, a hyperbolic space, or a custom synonym list are therefore not the same amount of privacy. Papers in this area nonetheless compare mechanisms "at the same ε" and draw rankings from those comparisons.
We propose a PRISMA-style systematic literature review of word- and sentence-level metric privacy for text. PRISMA is the medical protocol for searching, screening, and extracting studies with a documented audit trail. The review is paired with a quantitative recalibration. For every included paper we extract the metric, the embedding, and the reported ε, then convert each reported setting into units that do not depend on the metric: how often a word survives unchanged, and how many alternative words it could plausibly have come from. We then re-run a core set of mechanisms twice, once matched on ε and once matched on these units, and test whether the published privacy-utility rankings survive. The review method supplies the evidence base and the audit. The recalibration is the scientific claim.
Core Claim
A mechanism $M$ satisfies metric differential privacy with parameter $\epsilon$ under a distance $d$ if, for all inputs $x, x'$ and outputs $y$,
$$\Pr[M(x)=y] \le e^{\epsilon\, d(x,x')}\,\Pr[M(x')=y].$$
Only the product $\epsilon\, d$ enters this bound, so ε has no meaning apart from the scale and shape of $d$. Our claim is that the literature's cross-mechanism comparisons are largely comparisons of metric scale rather than of mechanism quality. We expect that re-expressing results in operational units will reorder mechanisms. Those units are retention probability $N_w$, the chance that word $w$ is left unchanged, and hiding-set size $S_w$, the number of words that could plausibly have produced a given output. Both were proposed for practitioner calibration in Calibrating Mechanisms for Privacy Preserving Text Analysis.
That paper supplies our method. It gives us the units but was not used to audit or re-rank the field, which is what we do here. The claim is falsifiable: if rankings at matched $N_w$ agree with rankings at matched ε, the field's current practice is vindicated, and we would report that.
Research Program
The review. It follows PRISMA's identification, screening, eligibility, and inclusion phases, with the explicit exclusion rationale at every step that Bermejo et al. and Schlegel identify as the core of review integrity. We pre-register a PICO-style question, borrowing the question format that clinical reviews such as Li et al. fix in advance to avoid selection bias. In our version the population is text datasets, the intervention is a metric-privacy mechanism, the comparison is a non-private baseline or rival mechanism, and the outcomes are utility plus a privacy measure.
The extraction form records several items for each paper. The first is the embedding and its dimension. The second is the metric and whether it is a true metric; this matters because CusText deliberately uses non-metric similarity sets. The form also records the ε grid and how it was chosen, whether $N_w$, $S_w$, or any empirical attack is reported, the datasets used, and whether code and data links resolve.
Here the review borrows its audit design from Sav, Demetriou, and Liem. Their PRISMA audit of the 100 most-cited recommender-system papers found that only 61% linked all datasets and that 0% of papers using human labels reported annotator training. We apply the same meta-scientific lens to a field where the reporting failure is specific and technical: the metric scale behind ε goes unstated.
Screening. To reduce the manual screening burden, we use an instruction-tuned screener as a first pass. Bio-SIEVE reports 0.82 accuracy for single-task Guanaco-7B on Cochrane abstract screening, against 0.60 for ChatGPT, which it found excessively lenient. Every screener decision is checked by two human reviewers, and we report agreement. We treat automation as a labor aid, not as evidence.
The recalibration. For every extracted (mechanism, embedding, ε) triple that can be reconstructed, we compute the implied median $N_w$ and $S_w$ by Monte Carlo sampling on the stated vocabulary. This places the whole literature on a common axis.
We then re-implement the seven mechanisms benchmarked in A Comparative Analysis of Word-Level Metric Differential Privacy: SynTF, calibrated multivariate perturbations, Mahalanobis, SanText, truncated Gumbel, Vickrey, and the truncated exponential mechanism. To these we add CusText and DYNTEXT. DYNTEXT is a mechanism that widens candidate pools for rare words, so it changes $S_w$ unevenly across the vocabulary; it is the most interesting stress case for a single-number calibration. We also include the hyperbolic mechanism of Feyisetan et al., which reports up to a 20-fold privacy improvement over flat space. Our recalibration directly tests whether that gain persists once hiding sets are matched, or whether it partly reflects a change in distance scale.
Baselines
The naive reference point is the field's current practice: ranking mechanisms by utility at matched ε ∈ {1, 5, 10} on GloVe 50/100/300. This is the protocol of the Meisenbacher et al. benchmark. That benchmark is the strongest existing comparison, because it unifies seven mechanisms under one pipeline and a Privacy-Utility Composite score, and it already found that metric noise can sometimes improve accuracy.
The file does not give its per-cell accuracies: no reported result found on Lacuna for those cell values on IMDb or AG News. We will therefore regenerate them with the same data, embeddings, classifier, and seeds before any comparison. Our recalibrated ranking uses exactly the same runs and an identical tuning budget. Only the rule for choosing each mechanism's ε changes: it is set per mechanism so that median $N_w$ hits targets of 0.5, 0.2, and 0.05.
"Beating" the baseline here means showing that its conclusions are unstable. The pre-registered criterion is a Kendall rank correlation below 0.6 between the matched-ε and matched-$N_w$ rankings in at least one target regime, with bootstrap confidence intervals over seeds.
A second privacy axis uses the BERT-based token inference attack of DYNTEXT, which reports attack success falling to 0.00008 on IMDb against baselines as high as 97%. We ask whether matched-$N_w$ mechanisms also have matched attack success. If they do not, a single operational unit is itself insufficient.
The ablation isolates metric scale. We rescale each embedding space by a constant and confirm that matched-ε rankings shift while matched-$N_w$ rankings do not. This directly demonstrates the scale artifact.
Why not just…?
Why not just extend the Meisenbacher benchmark instead of doing a review? This is the most serious alternative. The recalibration experiment is, in effect, that extension, and it could stand alone. What the review adds is the population of claims. A benchmark shows whether seven mechanisms reorder under a new rule. The review shows how many published conclusions across the field depend on unmatched metric scales, and how often papers even report enough to check. If extraction finds that most papers already report $N_w$ or $S_w$, the review's contribution shrinks to confirmation, and we would say so.
Why not just tell authors to report plausible deniability, as the calibration paper already proposed? Proposing a unit is different from showing that ignoring it changes conclusions. Without the re-ranking evidence, the recommendation has not persuaded the field; the extraction will quantify how rarely it is followed. Our retroactive conversion also lets existing results be reused rather than discarded.
Why not just use empirical attack success as the common axis and drop ε altogether? Attacks depend on the attacker's model and training data, so they trade one non-comparable number for another. We report attack success alongside $N_w$ and treat any disagreement between the two as a finding, not a nuisance.
Proposed Experiments
The primary experiment is the matched-ε versus matched-$N_w$ re-ranking on IMDb and AG News, with nine mechanisms, three embedding dimensions, three targets, and five seeds.
The secondary experiment repeats the calibration on PubMedQA contexts (PubMedQA). Biomedical vocabularies are sparse and full of rare terms, so the same ε should imply much smaller hiding sets there. This is the density effect DYNTEXT exploits. It tests whether ε settings imported from consumer-text papers under-protect scientific text. We measure answer accuracy against the reported 68.1% BioBERT and 78.0% human reference points, alongside $N_w$.
The review's deliverables are three: the PRISMA flow diagram, the extracted dataset with every reconstructed $N_w$, and the fraction of published cross-mechanism claims whose direction flips under recalibration.
The decisive checks are two. The first is whether the ranking correlation falls below the pre-registered threshold. The second is whether the rescaling ablation moves matched-ε rankings while leaving matched-$N_w$ rankings fixed. If both hold, the field's ε comparisons are shown to be partly artifacts of metric scale. If neither holds, current practice is empirically defensible, and we would report that result.
References
- Calibrating Mechanisms for Privacy Preserving Text Analysis
- AR/VR Teaching-Learning Experiences in Higher Education Institutions (HEI): A Systematic Literature Review
- Review of: "Explaining the Unexplainable: A Systematic Review of Explainable AI in Finance"
- Clinical Efficacy of Including Capecitabine in Neoadjuvant Chemotherapy for Breast Cancer: A Systematic Review and Meta-Analysis of Randomized Controlled Trials
- A Customized Text Sanitization Mechanism with Differential Privacy
- Annotation Practices in Societally Impactful Machine Learning Applications: What are Popular Recommender Systems Models Actually Trained On?
- Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
- A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking the Privacy-Utility Trade-off
- DYNTEXT: Semantic-Aware Dynamic Text Sanitization for Privacy-Preserving LLM Inference
- Hyperbolic Embeddings for Preserving Privacy and Utility in Text
- PubMedQA: A Dataset for Biomedical Research Question Answering