Photo enhancement has no single right answer. The MIT-Adobe FiveK dataset makes this concrete: each of its 5,000 raw photos was retouched by five professional photographers, and they disagree. Unpaired enhancement methods learn to improve photos from two unaligned piles, one of "before" images and one of "after" images, without matched pairs. These methods still pool every "after" image into one target, so the model learns an averaged taste that no individual holds. Personalized enhancers avoid the averaging, but they need matched before/after pairs, either real or synthesized.
We propose a single enhancer that handles many users. A user supplies a handful of photos they like, unpaired and already edited. A small second network, a hypernetwork (a network whose output is the weights of another network), reads those photos in one forward pass. It emits low-rank weight corrections for the enhancer and for the adversarial critic that judges whether outputs match that user's taste. Because the hypernetwork is shared across roughly a thousand users, each user's critic borrows statistical strength from all the others. Without that sharing, a critic trained on twenty photos would simply memorize them. The central test is whether the system reproduces how people actually disagree, not only whether it matches each person on average. The model should produce divergent edits on the images where FiveK's experts diverged, and similar edits where they agreed.
Core Claim
In unpaired enhancement with many users, each holding only a small set of preferred photos, the bottleneck is not the generator. It is the critic, known as the discriminator. A monolithic discriminator pulls every user toward a shared average. A separate discriminator per user overfits a set of 10 to 50 images. Our claim has three parts:
- A set-conditioned hypernetwork that generates low-rank adapters for both generator and discriminator gives better per-user fidelity than either extreme.
- It generalizes to unseen users from a few exemplars, with no gradient steps.
- It preserves the item-level disagreement structure among human retouchers.
We state in advance that part of this claim could turn out to be about conditioning the discriminator rather than about generating weights. The experiments are designed to separate the two.
Relation to Prior Work
The closest work is Hypernetworks for Perspectivist Adaptation. It generates per-annotator LoRA matrices for a frozen RoBERTa classifier from annotator and layer IDs, cutting trainable parameters from about 125M to about 5.6M on subjective text tasks. This paper supplies our mechanism and motivates the "one input, many valid labels" framing. We differ from it in three ways:
- The primary network is an image generator trained adversarially without pairs, not a supervised classifier.
- The hypernetwork also shapes the discriminator.
- Conditioning comes from a set of exemplar images rather than a discrete ID. This directly addresses the cold-start limitation that paper acknowledges.
Two unpaired enhancers show the problem we target:
- UEGAN is a one-path GAN with fidelity, quality and identity losses. It explicitly notes that experts' styles "may not meet the needs or preferences of general users," yet it trains a single static discriminator on a pooled target.
- EnlightenGAN is a one-path low-light enhancer with global and local discriminators. It likewise assumes one "normal light" target.
We keep their one-path design, which avoids the cycle-reconstruction failure under one-to-many mappings discussed in Generative Adversarial Networks: Recent Developments. What we change is what the target distribution is.
Two personalized enhancers solve personalization but not the unpaired setting:
- PieNet averages embeddings of preferred images into a single user vector and injects it into decoder features. It trains on real pairs.
- Masked Style Modeling predicts an image-specific style token from the user's exemplars. It gets around missing pairs by training a degrading model to fake "before" images, which yields Flickr-100K with 1,000 users.
We reuse Flickr-100K's user collections but only their edited images. The synthesized "before" halves are discarded, so the problem stays genuinely unpaired.
From the disagreement literature, The Treasure Beneath Multiple Annotations supplies an evaluation idea. It supervises a variance head with empirical inter-annotator variance on edges and shows that disagreement is structured, learnable signal. We use FiveK's five experts the same way, as a held-out measurement of per-image disagreement. We do not supervise variance directly.
DisCo and Architectural Sweet Spots for Modeling Human Label Variation both find that jointly modeling items and annotators, and relating perspectives to one another, beats siloed modeling. This motivates a shared hypernetwork over independent per-user models.
Research Program
Generator. The primary generator is a UEGAN-style one-path U-Net enhancer that predicts a residual added to the input.
Discriminator. The discriminator is a global-local pair in the style of EnlightenGAN.
Hypernetwork. A frozen image encoder embeds each of a user's $k$ exemplar edits. A permutation-invariant attention pooling layer turns this set into a user code $u$. The hypernetwork maps $u$ and a learned layer embedding to rank-$r$ matrices $A_\ell, B_\ell$ for each adapted layer $\ell$, so the layer's weight becomes $W_\ell + A_\ell B_\ell$. This follows the layer-ID design of the perspectivist hypernetwork and of A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune. The hypernetwork uses separate output heads for the generator and the discriminator. Hypernetwork training is known to be unstable, so we adopt the parametrization advice of Magnitude Invariant Parametrizations Improve Hypernetwork Learning.
Training episodes. Each episode proceeds as follows:
- Sample a user and split their edited photos into a conditioning set and a disjoint "real" set for the discriminator. This split stops the critic from rewarding copies of the exemplars.
- Draw a batch of unedited source photos from a shared unpaired pool.
- Train with UEGAN's fidelity loss, a relativistic hinge adversarial loss under the generated user discriminator, and the identity loss.
Mismatched-user term. We add one contrastive adversarial term. User $j$'s discriminator should reject outputs generated for user $i \neq j$. This forces conditioning to carry real style information instead of collapsing to the mean.
Deployment. At test time a new user supplies $k$ exemplars. One hypernetwork pass produces their adapters, with no fine-tuning.
Baselines
Naive reference points. The first is the identity map, meaning the unedited input. The second is UEGAN trained on the pooled edited photos of all users. The pooled UEGAN is the "do nothing about subjectivity" model. The file reports that UEGAN reaches PSNR 22.88 and SSIM 0.8882 on FiveK, from Towards Unsupervised Deep Image Enhancement With Generative Adversarial Network. We retrain it on our splits rather than reuse that figure, because protocols differ. We will beat it if per-expert PSNR on held-out experts improves significantly across images.
Strongest existing method: Masked Style Modeling. Masked Style Modeling is the strongest method for few-exemplar personalization. It reports PSNR 23.18 on FiveK against PieNet's 20.54 in the same setup, and a 79.5% blind-study preference over PieNet, all from Personalized Image Enhancement Featuring Masked Style Modeling. It is the direct competitor on our question because it also conditions on a handful of a user's photos and trains on the same 1,000 Flickr users. We run it with its pseudo-pairs, which is its intended regime, on the same user splits, the same exemplar count $k$, and matched parameter count and GPU-hours. Hyperparameter search budgets are equal. Beating it without pairs, or matching it while generalizing better to unseen FiveK experts, would be the headline result. Merely matching it would still show that the degrading model is unnecessary.
PieNet adapted to unpaired training. We also include PieNet, with its averaged preference vector feeding a PUB-style decoder, retrained under our unpaired losses. This isolates the cost of a single averaged style vector.
Oracle per-user LoRA. We fine-tune a separate LoRA for each FiveK expert on that expert's full unpaired half, about 2,500 images each. This is an upper bound for data-rich users, not a competitor.
Ablations. Three ablations isolate the key ingredients, each at matched parameters:
- Hypernetwork adapters on the generator only, with a monolithic discriminator. This tests whether the discriminator is the bottleneck.
- The same user code injected through FiLM, meaning per-channel scale and shift, into both networks instead of generated LoRA. This tests whether weight generation beyond scale and shift matters.
- Removal of the mismatched-user adversarial term.
Why not just…?
Why not just condition a single generator with FiLM or AdaIN on a style vector? This is a real threat, since scale-and-shift conditioning is itself a restricted hypernetwork. The evidence that shared filters fail under conflicting styles comes mainly from text, in the perspectivist hypernetwork paper, and from PieNet's reported drop in multi-user tests. Neither shows this for unpaired image generation. The matched FiLM ablation settles it. If FiLM on both networks matches generated LoRA, we will report that the contribution is user-conditioned discrimination, not weight synthesis.
Why not just train one small model or adapter per user? With five FiveK experts this is feasible, and we include it as the oracle. It fails exactly where most users sit, with 10 to 50 photos and a discriminator that memorizes them. It also cannot serve an unseen user without training. The experiment varies $k$. We expect independent adapters to lose at small $k$ and to close the gap as $k$ grows. Where the curves cross is itself a finding.
Why not just synthesize pairs with a degrading model and train supervised? This is exactly Masked Style Modeling, and it is our strongest baseline, not a strawman. Its possible weakness is that biases of the degrading model propagate to the enhancer, and EnlightenGAN argues synthetic degradations miss real capture statistics. We test this by evaluating on real FiveK raw inputs from users never seen during training.
Why not just average the five experts into a consensus target? An average of a moody edit and a vibrant edit satisfies neither user. This mirrors the edge-detection finding that collapsing annotators discards useful structure (The Treasure Beneath Multiple Annotations). The disagreement check below measures what averaging loses.
Proposed Experiments
Data. Training uses the edited photos of Flickr-100K's 1,000 users as per-user unpaired target sets. Sources are unedited photos drawn from a pool disjoint from all evaluation images. If the released data lack per-user grouping, we will regroup it following the MSM collection protocol.
Held-out users. Evaluation holds out all five FiveK experts, which lets us run leave-one-expert-out. Each expert's exemplars come from one half of FiveK. PSNR and SSIM are measured on raw images from the other half, against that expert's retouch, at $k \in \{5, 10, 20, 50\}$. We also report per-user results on 100 held-out Flickr users, using NIQE on the output images.
Discrimination check. A user-identification probe asks whether an off-the-shelf style classifier can tell which user an output was generated for.
Disagreement check. For each test raw image, we compute the spread of the five experts' actual retouches, as per-image variance in a color space. We then correlate it with the spread of our five conditioned outputs. We expect a clearly positive rank correlation. The pooled UEGAN, which yields zero spread, and PieNet-style averaging are the reference points. A model can score well on per-expert PSNR while failing this test by applying a fixed per-user tone shift regardless of content. So this is the check that tells us whether the model has learned the structure of human disagreement rather than a lookup table of five offsets.
Human study. A blind human study with each FiveK-style preference collected from new participants' own exemplars follows MSM's pairwise protocol against MSM and the pooled UEGAN.
Decisive outcomes. The proposal succeeds only if three things hold together:
- The full model beats the generator-only ablation on held-out experts.
- It matches or beats MSM without pseudo-pairs.
- It shows a positive disagreement correlation.
If it fails on the first point, the discriminator hypothesis is wrong.
References
- Hypernetworks for Perspectivist Adaptation
- Towards Unsupervised Deep Image Enhancement with Generative Adversarial Network
- EnlightenGAN: Deep Light Enhancement without Paired Supervision
- Generative Adversarial Networks: Recent Developments
- PieNet: Personalized Image Enhancement Network
- Personalized Image Enhancement Featuring Masked Style Modeling
- The Treasure Beneath Multiple Annotations: An Uncertainty-Aware Edge Detector
- Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo
- Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!
- A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune
- Magnitude Invariant Parametrizations Improve Hypernetwork Learning