Deepfake detectors are trained on videos of real people's faces, which are biometric data. Anyone who wants to train a detector on footage collected from users, such as uploads to a platform, has to worry that the trained model, or the gradients shared while training it, leaks those faces. Zhu et al. reconstruct training images almost pixel-for-pixel from shared gradients (Deep Leakage from Gradients). The standard defense is differential privacy (DP), a formal guarantee that the model barely changes when any one person's data is removed. Its usual implementation, DP-SGD, clips each example's gradient and adds Gaussian noise. That noise grows with the number of trained parameters, and fine-grained forensic cues are exactly what it erases first. Detectors already fail badly on new datasets even without noise: the average detector trained on FaceForensics++ reaches only 56.9% AUC on Celeb-DF (Celeb-DF). Private training looks like it should make this worse.
We propose to train the detector by changing only the singular values of a pretrained vision encoder. Each weight matrix is factored as $W = U\Sigma V^\top$. The directions $U, V$ stay frozen, and only the diagonal scales $\Sigma$ are trained under DP-SGD. This one restriction serves two purposes. It shrinks the noised parameter vector to a few tens of thousands of numbers while still changing every layer at full rank, which addresses the privacy cost. It also stops the detector from inventing new feature directions, which is plausibly how detectors latch onto dataset-specific artifacts, which addresses the generalization cost. We add an auxiliary task of locating the manipulated region, and we show that under DP the per-task clipping thresholds act directly as task weights. Gradient balancing between tasks therefore costs no extra privacy.
Core Claim
Our central claim is about cross-dataset AUC at a fixed privacy budget. A detector trained privately on FaceForensics++ (FF++) by singular value fine-tuning should lose less cross-dataset AUC on Celeb-DF and DeeperForensics-1.0, for a given budget $\epsilon$, than detectors trained privately with low-rank adapters, with full fine-tuning, or with a linear probe on frozen features. We also expect its private cross-dataset AUC to come close to its own non-private cross-dataset AUC, because the main source of degradation in this setting is the shift between datasets, not DP noise.
A second claim concerns the auxiliary localization task. Localization helps under privacy only if the binary-classification gradient and the mask gradient are clipped separately. With a single joint clip, the dense mask loss dominates the gradient norm, and the clip then quietly discards most of the classification signal.
Each strand of prior work supplies one piece of this, and none tests the combination. Singular value fine-tuning supplies the mechanism. It adapts few-shot segmentation with 0.25% of backbone parameters (SVF). It avoids forgetting in class-incremental learning (SVF for FSCIL). In Robust SAM, tuning 512 singular values filters adversarial noise while keeping clean VOC at 80.9 mIoU, against 63.4 for a LoRA variant (Robust SAM).
The privacy side is motivated by Differentially Private Adapters. There, full-model DP-SGD collapsed to about 53% accuracy, while private residual adapters trained with PATE reached 91.82%, against 96.49% non-private. That work uses bottleneck adapters, teacher ensembles and speech data. We instead use DP-SGD on singular values only, on vision forensics, with cross-dataset evaluation.
On the forensics side, LAE shows that supervising where the forgery is improves transfer to unseen manipulations. It trains a full autoencoder with no privacy mechanism.
RotoGrad frames conflict between tasks as a mismatch in gradient magnitude plus a mismatch in direction. We borrow that diagnosis but not its learned rotations. Rotations would add trained, noised parameters and would complicate privacy accounting.
Within this file, we found no work that applies DP to singular value adaptation, and none that trains deepfake detectors with formal DP.
Research Program
Backbone and trained parameters. The detector is a ViT-Base encoder pretrained on generic images, the same backbone family used in IMSE. We precompute the SVD of every attention and MLP weight matrix. We train only a multiplicative correction to each singular value, plus a small classification head and a light mask decoder. That amounts to roughly the IMSE-scale count of about 36.8K spectral parameters. The privacy unit is the identity, not the frame: all frames of one person are grouped together, and the per-example clip is applied to that person's summed gradient. This prevents the many frames of each identity from silently inflating the effective privacy guarantee.
Private multi-task update. For identity $i$ in a sampled batch $B$, let $g^{b}_i$ be the gradient of the real/fake loss and $g^{m}_i$ the gradient of the localization loss, both taken with respect to the trainable spectral and head parameters. The update direction is
$$\tilde g = \frac{1}{|B|}\Big(\sum_{i\in B}\big[\mathrm{clip}(g^{b}_i, C_b) + \mathrm{clip}(g^{m}_i, C_m)\big] + \mathcal{N}\big(0, \sigma^2 (C_b+C_m)^2 I\big)\Big).$$
Here $\mathrm{clip}(g, C)$ rescales $g$ to have norm at most $C$, and $\sigma$ is the noise multiplier. Each identity's contribution has norm at most $C_b + C_m$, so standard accounting applies unchanged. The ratio $C_b/C_m$ fixes the maximum influence each task can have on the update. This is RotoGrad's magnitude balancing, obtained as a by-product of the privacy mechanism.
Measurements and hyperparameters. During training we log the cosine similarity between the summed per-task gradients in $\Sigma$-space. This tells us whether directional conflict remains after magnitudes are equalized. Because selecting hyperparameters under DP itself consumes privacy budget (Fine-Tuning with DP Necessitates an Additional Hyperparameter Search), we fix learning rate, $C_b$, $C_m$ and $\sigma$ on a small public proxy split. We then report results both with and without the cost of that search counted.
Spectral analysis. The SVF paper found that singular values tied to foreground objects grow during fine-tuning, while those tied to background shrink. We ask the forensic analogue. In-domain-only artifact detection may rely on the low-energy tail of the singular spectrum, while transferable cues may rely on high-energy directions. If so, a simple restriction to the top-$k$ singular values per layer would trade a little in-domain accuracy for cross-dataset AUC, and it would also further reduce the noise dimension.
Baselines
Non-private references. The do-nothing reference is the pretrained encoder with a non-private linear head trained on FF++, then evaluated on Celeb-DF and DeeperForensics-1.0. It shows how much transfer comes free from pretraining. Alongside it we run non-private versions of every method below, so that each private result can be read as a loss relative to its own ceiling.
Strongest existing generalization methods (non-private context). These are reported numbers, not reruns under our conditions. The Celeb-DF authors report 56.9% average baseline AUC on Celeb-DF and an 87.4% average AUC across datasets for the best detector, DSP-FWA (Celeb-DF). EfficientNet-v2-L with a reinforcement-learned test-time augmentation policy reaches 0.669 AUC from FF++ to Celeb-DF and 0.952 AUC on DeeperForensics-1.0 (On Improving Cross-dataset Generalization). That 0.669 is the cross-dataset target we cite. For a private method, matching it at $\epsilon \le 8$ with a far smaller backbone would be a meaningful result. We found no reported result on Lacuna for differentially private deepfake detection on any of these benchmarks, so every private number in this study is new.
Strongest private competitors. These are DP-SGD on LoRA adapters at ranks chosen to match our trainable-parameter count and at higher ranks, and DP-SGD on a linear probe over frozen features. All are run on the same backbone, with the same identity-level privacy unit, accountant, $\epsilon$ grid (1, 4, 8), epochs and public-proxy tuning budget. We also include full-parameter DP-SGD, expected to be weak given the roughly 53% collapse reported in Differentially Private Adapters. Beating these baselines means a higher Celeb-DF AUC at equal $\epsilon$ and equal trainable-parameter count, averaged over three seeds with non-overlapping intervals.
Ablations. Four ablations isolate the key ingredients. The first is singular value tuning without the localization task. The second uses a joint single clip instead of per-task clips. The third varies the clip ratio $C_b/C_m$. The fourth restricts training to the top-$k$ singular values.
Why not just…?
Why not just use DP-LoRA, the standard private parameter-efficient recipe? This is the real competitor, and the file contains no head-to-head result under DP. The indirect evidence favors singular values. Robust SAM's LoRA variant lost clean accuracy (63.4 against 80.9 mIoU on VOC). Calibrating and Rotating links LoRA's lower singular-value entropy to weaker results on the GLUE language benchmark (88.57 against 89.23 for SORA). Neither result involves privacy or forensics, so the parameter-matched DP-LoRA comparison above is what settles the question. If DP-LoRA matches our method at equal parameters, the spectral restriction contributes nothing specific, and we would report the study as a characterization of private forensics rather than a new method.
Why not just train a private linear probe on frozen features? A probe has the fewest parameters and the least noise, and pretrained features may already carry enough forensic signal. It cannot reshape internal features, however, and FF++ shows that compression removes surface cues (FaceForensics++). We therefore expect the probe to fail most on the low-quality c40 split and on Celeb-DF. If it does not, singular value tuning is unnecessary, and that would itself be a useful finding.
Why not just train non-privately on public forgery data and skip DP? This works whenever no private data is needed. The do-nothing reference measures exactly that option. Our method only matters if private training on in-distribution footage adds AUC beyond public-only training. We test this by treating the Celeb-DF real/fake training split as the "private" pool for an FF++-pretrained detector and comparing against FF++-only training.
Why not just use one joint clip for the multi-task loss? It is simpler, and it is one of our ablations. Our prediction is that it underperforms because the mask loss dominates the gradient norm. Logging the fraction of each sample's clipped norm that comes from each task will show whether that mechanism is actually what happens.
Proposed Experiments
We train on FF++ (c23 and c40) with identity-level DP and evaluate AUC on Celeb-DF and DeeperForensics-1.0, plus cross-manipulation transfer within FF++. We plot privacy-utility curves of AUC against $\epsilon$ for every method. We also run gradient inversion in the style of Deep Leakage from Gradients, at batch sizes up to 8, on gradients of the singular values alone and on gradients of full weights. This is only an empirical sanity check alongside the formal guarantee.
Two outcomes would decide the proposal. The first is whether private singular value tuning keeps its Celeb-DF AUC within a few points of its own non-private value at $\epsilon = 8$ while beating parameter-matched DP-LoRA. The second is whether per-task clipping makes localization help rather than hurt under DP. If either fails, the spectral analysis still tells us where forensic signal sits in pretrained encoders and how DP noise interacts with it.
References
- Deep Leakage from Gradients
- Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
- Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning
- Singular Value Fine-Tuning for Few-Shot Class-Incremental Learning
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models
- Differentially Private Adapters for Parameter Efficient Acoustic Modeling
- Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation
- Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search
- On Improving Cross-dataset Generalization of Deepfake Detectors
- Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
- FaceForensics++: Learning to Detect Manipulated Facial Images