Abstract
As Large Language Model (LLM) applications expand into retrieval-augmented generation (RAG) and long-form reasoning, the secure handling of sensitive information remains a significant challenge. We introduce InvisibleInk, a scalable framework for long-form text generation that provides rigorous differential privacy (DP) guarantees. InvisibleInk optimizes the exponential mechanism over LLM logits through two key innovations: DClip, which isolates and clips only the sensitive delta between private and public logits, and Top-k+ sampling, which improves text quality by restricting sampling to a high-probability token subset. Our evaluations demonstrate an 8x reduction in computational overhead compared to state-of-the-art baselines, achieving high-utility private text generation at less than 10x the cost of non-private methods.
Introduction
The integration of LLMs into specialized domains like medicine and law often requires processing sensitive documents. While paradigms like RAG enhance model utility, they also risk leaking private data through generated outputs. Differential Privacy (DP) provides a rigorous framework to mitigate this risk, but traditional DP text generation methods have historically been impractical due to extreme computational costs (often 100x overhead) and poor output quality (coherence issues).
InvisibleInk addresses these "deal-breakers" by treating token sampling as an iterative application of the exponential mechanism. By focusing privacy-preserving noise only where it is needed most and employing smarter truncation strategies, InvisibleInk enables high-quality, private long-form generation at a fraction of the previous cost.
The InvisibleInk Framework
Targeted Clipping (DClip)
A core insight of InvisibleInk is that the majority of an LLM's output is governed by general linguistic patterns—grammar, syntax, and common knowledge—which are not private. Standard DP methods apply noise to the entire logit distribution. In contrast, DClip isolates the difference between the "private" logits (conditioned on sensitive data) and "public" logits (conditioned only on general context). By clipping only this residual difference, the framework can use a much tighter clip norm, significantly reducing the amount of noise required to satisfy DP guarantees.
Top-k+ Sampling
Standard DP generation typically requires sampling from the entire vocabulary to avoid infinite privacy loss (which occurs if a token's probability is zero in one distribution but non-zero in another). However, sampling from the "long tail" of a 32k+ token vocabulary often leads to incoherent "gibberish."
InvisibleInk introduces Top-k+ sampling, which constructs a tight superset of the most likely private tokens using only public information. This allows the model to utilize truncated decoding—a standard technique in non-private LLMs—to maintain high text quality and coherence without compromising the privacy budget.
Efficient Privacy Accounting
Unlike previous state-of-the-art methods that rely on computationally expensive grid searches to calibrate privacy parameters, InvisibleInk utilizes a non-adaptive accounting procedure based on zero-Concentrated Differential Privacy (zCDP). This allows for off-the-shelf deployment and predictable privacy budgets.
Experimental Evaluation
The framework was evaluated across three sensitive domains: medical discharge summaries (MIMIC-IV-Note), legal case notes (TAB), and user reviews (Yelp).
Efficiency and Utility
InvisibleInk demonstrates a consistent 8–16x reduction in computation cost compared to prior baselines. While previous methods required up to 100x the overhead of non-private generation to produce coherent text, InvisibleInk achieves high utility with less than 10x overhead. On the MIMIC dataset, InvisibleInk achieved MAUVE scores (a metric for human-like text) of 71-74% using only 8-16 inference calls per token. Baselines failed to reach this quality even with 128 calls.
Downstream Performance
The utility of the generated text was further validated through downstream tasks. Synthetic data generated by InvisibleInk was used to train classifiers, significantly outperforming existing methods. For instance, on a 50-class classification task, InvisibleInk achieved 33% accuracy, compared to just 7% for the AdaPMixED baseline.
Discussion and Limitations
While InvisibleInk represents a significant step forward, several considerations remain for practical deployment:
- Model Access: The framework requires "white-box" access to model logits, meaning it is currently best suited for open-weights models (like Llama 3) or internal deployments by API providers.
- Trust Model: It assumes a trusted central aggregator with access to the raw sensitive references to perform clipping.
- Scope of Privacy: The current implementation focuses on example-level DP (protecting individual documents). Future work is needed to extend these guarantees to user-level privacy across multiple documents.
Conclusion
InvisibleInk bridges the gap between theoretical privacy and practical utility in LLM text generation. By recognizing that "language is mostly public" and optimizing the sampling process for coherence, it enables the safe application of LLMs to sensitive datasets without the prohibitive costs previously associated with differential privacy.
Summary updated