Introduction
As multilingual language models (MLMs) like mBERT and XLM-R become foundational to global AI infrastructure, the demand for "trustworthy" AI has intensified. Trustworthiness is typically defined by three pillars: Differential Privacy (DP), which prevents the leakage of training data; Linguistic Fairness, which ensures equitable performance across languages; and Transparency, often achieved through instance-based interpretability (identifying which training examples influenced a specific prediction).
While these goals are often pursued in isolation, this research investigates their intersection. The authors present a rigorous analysis of whether these objectives align or conflict, providing mathematical proofs and empirical evidence for the "impossibility" and "possibility" of satisfying these requirements simultaneously.
Theoretical Foundations
The Privacy-Interpretability Conflict (Impossibility)
The paper's first major contribution is Theorem 1, which establishes a fundamental trade-off between DP and sparse training data influence. In instance-based interpretability, a model is considered transparent if a prediction can be traced back to a small, highly influential subset of training data. However, the authors prove that a private model (with a low privacy budget $\epsilon_p$) must necessarily have high influence uniformity. To protect individual data points, the model must "smear" influence across the entire training set. Consequently, a more private model is inherently harder to interpret using instance-based methods.
The Privacy-Fairness Synergy (Possibility)
Conversely, Theorem 2 offers a more optimistic outlook regarding linguistic fairness. The authors define fairness through the lens of multilingual compression—the degree to which a model maps different languages into a shared representational space. They prove that perfect compression actually facilitates DP. When a model treats different languages identically, an attacker cannot distinguish which language a specific piece of information originated from, effectively providing a form of $k$-anonymity across the language dimension.
Methodology
The study validates these theorems through extensive experiments using XLM-R and mBERT. The models were fine-tuned using DP-AdamW, a private variant of the standard optimizer, across a range of privacy budgets ($\epsilon \in \{1, 3, 8, 15, 30, \infty\}$).
Metrics and Tasks
- Tasks: Evaluation was performed on Part-of-Speech (POS) tagging (using PUD treebanks) and Natural Language Inference (NLI) (using the XNLI dataset).
- Compression Metrics: To quantify how well languages were merged in the latent space, the authors employed Sentence Retrieval Precision, Centered Kernel Alignment (CKA), IsoScore, and Representational Similarity Analysis (RSA).
- Influence Uniformity (InfU): A new diagnostic metric, InfU, was introduced based on the TracInCP method. It measures the distribution of training data influence; high uniformity indicates that no single point is overly influential, aligning with DP requirements but hindering interpretability.
Empirical Results
The Performance-Privacy Trade-off
As expected, increasing privacy constraints led to significant performance degradation. For the XNLI task, accuracy plummeted from approximately 90% in non-private settings to 36% at a strict privacy budget of $\epsilon=1$. This underscores the high utility cost of current DP mechanisms in complex multilingual tasks.
The "U-Shaped" Compression Curve
The relationship between privacy and multilingual compression revealed a non-linear, U-shaped trend. Both highly private models ($\epsilon=1$) and non-private models ($\epsilon=\infty$) exhibited high compression, but for different reasons:
- Non-private models achieve compression by learning deep semantic commonalities across languages.
- Highly private models achieve compression because the injected DP noise forces the model to ignore language-specific nuances, effectively collapsing representations into a simpler, shared state.
The "valley" of lowest compression occurred at intermediate privacy levels ($\epsilon$ between 8 and 30), where the model had enough capacity to learn language-specific features but was still hampered by noise.
Validating Influence Uniformity
Experimental data confirmed the theoretical link between privacy and interpretability. For the POS tagging task, the researchers found a strong correlation ($R=0.77$) between privacy levels and Influence Uniformity. As the privacy budget decreased, the influence of training instances became increasingly uniform, validating the claim that DP inherently suppresses the "influential signals" required for instance-based transparency.
Conclusion and Future Work
This research highlights a critical tension in the development of trustworthy MLMs. While linguistic fairness and differential privacy can be mutually reinforcing through representational compression, there is a hard mathematical limit on achieving both privacy and instance-based interpretability.
Future work must address the optimization gap. While the paper proves these objectives can be compatible in theory, current training regimes do not yet provide a seamless way to balance them. The development of multi-objective optimization frameworks that can navigate these trade-offs—particularly for generative tasks beyond classification—remains a vital frontier for the field.
Summary updated