TL;DR
InvisibleInk is a highly scalable framework for generating long-form text with rigorous differential privacy (DP) guarantees at a fraction of the cost of prior methods. By isolating and clipping only the sensitive information in model logits and introducing a novel truncated decoding algorithm called Top-$k$+ sampling, it reduces computational overhead by $8\times$ or more compared to state-of-the-art baselines. This brings the cost of private text generation down to less than $4$-$8\times$ that of non-private generation, making DP practical for real-world large language model (LLM) deployments like Retrieval-Augmented Generation (RAG).
What Problem Does This Paper Address?
As LLMs are increasingly trusted with long-form text generation and complex reasoning tasks (such as RAG or "deep research" agents), they routinely process highly sensitive data provided at inference time. This includes confidential corporate documents, medical records, or private user queries. Unfortunately, detailed long-form generation carries a high risk of inadvertently leaking these sensitive details.
While Differential Privacy (DP) can provably mitigate this risk, existing white-box DP text generation methods suffer from severe practical limitations:
- Massive Compute Overhead: Prior state-of-the-art methods require up to $100\times$ or more computational overhead compared to non-private generation to avoid producing gibberish.
- Text Degeneration: To maintain privacy, these methods typically sample from the full vocabulary distribution at high temperatures, which destroys the fluency, coherence, and structure of the generated text.
Methods
InvisibleInk models next-token sampling as an iterative application of the exponential mechanism over the LLM's logits. It introduces two key innovations to solve the compute and quality bottlenecks:
1. DClip (Targeted Clipping Function)
Standard DP methods clip the raw private logits directly to bound sensitivity. However, private logits contain a massive amount of non-private information (e.g., grammar, syntax, and general world knowledge).
DClip isolates only the sensitive information by computing the difference between the private logits ($\phi_i$) and the public logits ($\phi_{pub}$) generated without the sensitive context: $$\text{DClip}_C(\phi_i, \phi_{pub}) := \phi_{pub} + \text{clip}_C(\phi_i - \phi_{pub})$$ Because the public model already provides a strong linguistic prior, the delta ($\phi_i - \phi_{pub}$) is tightly concentrated around zero. This allows the framework to use a much smaller clipping norm ($C \approx 1$ vs. $C \approx 10$ in baseline methods) without distorting the distribution, drastically lowering privacy sensitivity and compute cost.
2. Truncated Top-$k$+ Sampling
Standard top-$k$ sampling restricts token selection to the highest-probability tokens to prevent text degeneration, but doing so directly on private logits violates DP.
InvisibleInk introduces **Top-$k$+$ sampling**, which constructs a tight *superset* ($V_k^+$) of the top-$k$ private tokens using *only* the public logits. By mathematically shifting the public top-$k$ threshold downward by $2C/B$ (where $B$ is the batch size), the authors guarantee that any token that *would* have made it into the private top-$k$ vocabulary is safely included in this expanded public vocabulary. Because $V_k^+$ is constructed purely from public information, sampling from it incurs zero privacy cost while preserving the high utility of truncated decoding.
Datasets
The authors evaluated InvisibleInk across three distinct, privacy-sensitive domains featuring long-form text:
- MIMIC-IV-Note: A large, de-identified collection of clinical ICU discharge summaries containing highly sensitive medical diagnoses and treatment plans. This dataset is legally unavailable for standard LLM pretraining, making it an excellent out-of-distribution test bed.
- Yelp Reviews: A standard commercial benchmark dataset consisting of user-generated business reviews, personal opinions, and location references.
- TAB (Text Anonymization Benchmark): A low-resource legal dataset containing court case transcripts from the European Court of Human Rights (ECHR), annotated with personal identifiers.
Key Insights and Ideas
- Don't Pay for What the Public Model Already Knows: The core philosophical shift in DClip is recognizing that grammar, syntax, and basic facts aren't secrets. By penalizing only the deviation from the public model, the privacy budget is spent exclusively on protecting the actual sensitive context.
- Graceful Fallback: If the compute or privacy budget approaches zero, InvisibleInk naturally and gracefully degrades to the performance of the base public model. In contrast, previous methods collapse into uniform random sampling, outputting pure gibberish.
Key Results
- $8\times$ Compute Reduction: Across medical, legal, and commercial domains, InvisibleInk consistently achieved an $8\text{--}16\times$ reduction in computational cost (measured via next-token inference calls and wall-clock time) compared to prior state-of-the-art white-box baselines for the same privacy-utility trade-offs.
- Superior Text Quality: Using the MAUVE metric to evaluate distributional similarity to real human text, InvisibleInk significantly outperformed baselines. For instance, on the MIMIC dataset with a strict compute budget ($B=7$), baselines completely failed to generate text (marking them as infeasible), while InvisibleInk achieved high-quality generations matching non-private baselines.
- Downstream Utility: When training a RoBERTa classifier on synthetic data generated by the various private methods, data generated by InvisibleInk yielded substantially higher classification accuracy ($32.98\%$) compared to Amin et al. ($29.44\%$) and AdaPMixED ($7.44\%$).
Limitations and Future Work
- White-Box Requirement: InvisibleInk assumes a trust model where a central aggregator has white-box access to the LLM's weights and next-token logits. It cannot be used out-of-the-box with closed, API-only models (like GPT-4), though it completely outperforms existing API-based DP alternatives like AugPE when white-box access is available.
- Example-Level vs. User-Level Privacy: The current framework provides example-level DP (protecting a single document or reference text). Extending this to user-level DP (protecting all documents contributed by a single user) remains an open direction.
- Token Budget Scaling: The privacy guarantee scales with the maximum token length $T$. Finding ways to improve this dependence using regularization or advanced composition techniques is left for future work.
Summary updated