Some photographs stick in memory and others slip away. Different people tend to agree on which is which, so a model can learn to predict it. Tools that edit images to be more memorable already exist: style transfer, edits to face images, and steering inside image generators. All of them have the same blind spot. They push an image toward whatever an automatic memorability predictor rewards and never check whether the edit has also changed what the image communicates emotionally. That matters for two reasons. A photo editor that makes a wedding picture "more memorable" by making it look ominous has failed its user. And the question of whether memorability can be edited independently of emotion is itself unanswered.
We propose an editor for ordinary photographs, not just faces. It applies local, interpretable adjustments (cropping, sharpening or contrast on a particular region, saturation on a particular region) chosen to raise predicted memorability. It must also keep the image's emoji-reaction fingerprint fixed: the probability distribution over which emoji people would attach to the image when posting it. That fingerprint comes from emoji-based distant supervision, which means learning from labels people supply incidentally (here, the emoji they chose for their own Instagram posts) instead of paying annotators. The fingerprint serves as an affect-preservation constraint and as a measuring instrument. With it we can ask a sharp question in human memory experiments: how much of an editor's memorability gain survives once the emotional reaction is held constant?
Core Claim
The claim has two parts. The first is engineering: a constrained, local photo-editing policy can produce human-verified memorability gains on general scenes while leaving the elicited emoji reaction statistically unchanged. The second is scientific: by comparing the same editor with and without the emoji constraint in human memory tests, we can measure how far scorer-driven memorability enhancement works by shifting affect.
There is a hint that affect matters. Changing the Image Memorability: From Basic Photo Editing to GANs found that boosting saturation helps dull images but hurts already-memorable ones, which is exactly the pattern an affect-mediated effect could produce.
What is new is not "emoji plus memorability." What is new is using a distantly supervised affect model as an invariance constraint on memorability edits, and as the instrument that separates the two effects.
Prior Work and Where This Sits
Three lines of work motivate the editing side, and all three share the gap we target.
Changing the Image Memorability trains an attribute-editing GAN on 200,000 CelebA faces pseudo-labelled by the AMNet predictor (AMNet). It reports a 0.09 average memorability shift, with the change in the intended direction for 92% of faces in a human memory game. It is limited to faces and to three discrete memorability levels.
How to Make an Image More Memorable? A Deep Style Transfer Approach picks, for each image, one of 500 abstract art styles by predicting the memorability gain from applying it. Its edits are global restylings that can wipe out detail, which Understanding and Predicting Image Memorability at a Large Scale warns against: cartoonization hurts memorability.
Discovering Multiple and Diverse Directions for Cognitive Image Properties finds directions in a BigGAN generator's input space that raise MemNet-predicted memorability, with a manipulation slope of 0.901 versus 0.522 for GANalyze. It works on synthetic images and is scored by the same kind of network it optimizes. None of these three papers constrains or measures how the emotional reaction changes.
Three papers supply method and evidence for the editing space we use. Sharpening is the only classical filter that reliably helps, by +0.023 (Changing the Image Memorability). Tighter cropping and centring of the main subject raise memorability (The Computational Memorability of Iconic Images). An image's memorability is dominated by its most memorable object (What Makes an Object Memorable?). Together these argue for edits that act on specific objects and regions, not filters applied to the whole frame.
For the predictor, Don't Judge Before You CLIP shows that a frozen CLIP ViT-L/14 image encoder adapted with LoRA reaches SRCC 0.741 on LaMem and 85.28% on 8-class EmoSet. LoRA is low-rank adaptation, a cheap way to fine-tune a large model by training a few small added matrices. This recipe supplies our memorability scorers and our emoji model.
On the affect side, DeepMoji shows that predicting 64 emoji over 1.2 billion tweets yields fine-grained emotional representations that transfer to new tasks. Multimodal Emoji Prediction shows that Instagram images alone carry emoji signal. That second paper also constrains us: its image-only model reaches just 16.1% F1 on top-20 emoji. A raw emoji classifier is therefore too weak to use as a constraint by itself. We use the full predicted distribution, built on a stronger CLIP backbone, and validate it against human emoji choices before relying on it.
Finally, Is Image Memorability Prediction Solved? shows that predictors now essentially match the human consistency ceiling of 0.68. It also shows that memorability depends on which images were seen just before. Our human evaluation must therefore hold the surrounding image sequence fixed across conditions.
Research Program
Emoji-reaction model. We train a CLIP ViT-L/14 with LoRA, following PerceptCLIP, on the image side only of the Instagram multimodal emoji corpus (299,809 posts, top-20 emoji). The model outputs a 20-way emoji distribution $p(e\mid x)$ for an image $x$.
Two risks need early checks. First, the original posts may have disappeared from Instagram over time, as DeepMoji notes for social data. If the corpus cannot be fully reconstructed, we will recollect posts that each carry a single emoji, using the same filter. Second, the model may not track human judgment. Before using it as a constraint, we check that its distribution agrees with the emoji humans choose for held-out LaMem images, and that it is sensitive to edits we know change affect, such as a strong saturation increase or cropping out a person.
Memorability scorers. We train an ensemble of three CLIP+LoRA scorers on disjoint LaMem training subsets. The editor optimizes their mean, $\bar M$. A fourth scorer of a different architecture, MemBoost-style ResNet features with gradient boosting, is trained on separate data and never seen during optimization. It serves only as a held-out judge, to detect edits that fool the training scorers without fooling people.
Editor. A segmentation model proposes regions for the main subject and the background. A small policy network reads CLIP features and outputs continuous parameters for a set of differentiable operations: crop box, per-region unsharp-mask strength, per-region local contrast, per-region saturation, and subject/background tone. It is trained on unlabelled scene images with
$$ \max_\theta \; \mathbb{E}_x\Big[\bar M(E_\theta(x)) \;-\; \lambda\,\mathrm{KL}\big(p(e\mid x)\,\|\,p(e\mid E_\theta(x))\big) \;-\; \mu\,\mathrm{LPIPS}(x, E_\theta(x))\Big], $$
where $E_\theta(x)$ is the edited image, $\lambda$ weights the emoji-reaction invariance term, and LPIPS is a standard learned measure of how different two images look, weighted by $\mu$.
Sweeping $\lambda$ from 0 upward traces a curve of predicted memorability gain against preserved affect. That curve is the main analytic object of the project. We deliberately do not let a generative model repaint the image: a bounded, inspectable set of edits keeps the result genuinely "post-processing" and rules out hallucinated content as the source of any memorability gain.
Baselines
The do-nothing reference is the unedited image placed in the same position of the same memory-game sequence. Every gain is measured against it.
Global sharpening is the strongest classical filter. Changing the Image Memorability reports a +0.023 average gain, measured by the AMNet predictor on 2,000 LaMem images. We apply it with sharpening strength tuned on our scorers using the same search budget as the policy, then evaluate it with humans. Global saturation, tuned the same way, tests the affect-mediation hypothesis directly. If saturation's gains turn out to come with large shifts in emoji reaction, that supports affect mediation.
The strongest existing scene-level method is S-cube style selection (How to Make an Image More Memorable?), which we re-implement on our scorers and its 500-style setup. No reported human-measured result was found on Lacuna for S-cube; the paper reports only scorer-level comparisons.
The best human-verified generative result in the file is AttGAN on faces: a 0.09 average shift and 92% correct direction (Changing the Image Memorability). We report results separately on the face images in LaMem so the comparison is like-for-like, and we treat roughly 0.09 there as the bar to meet.
Two ablations isolate the key ingredient. The first is the same editor with $\lambda=0$, which is unconstrained scorer chasing. The second replaces the emoji term with a CLIP-embedding similarity constraint, and alternatively with an EmoSet 8-class emotion constraint. All variants share the training data, the scorers, and the compute and hyperparameter budget.
The proposal wins under three conditions. The constrained editor must retain a human memorability gain that is significantly above zero and above sharpening. It must show a significantly smaller human-measured affect shift than $\lambda=0$. And on that trade-off it must beat both the CLIP-similarity constraint and the EmoSet constraint.
Why not just…?
Why not just sharpen? Sharpening is cheap and genuinely helps, but its reported gain is small (+0.023, predictor-measured) and it acts on the whole frame. The evidence on object anchoring (What Makes an Object Memorable?) predicts that edits targeted at regions should do better. Sharpening is a baseline in our human study. If it matches the editor, the honest conclusion is that learned enhancement is not worth it for scenes, and we will report that.
Why not just preserve the CLIP embedding instead of an emoji distribution? This is the most serious threat. CLIP similarity may already preserve affect, and it needs no social-media data. It is run as an ablation under an identical budget. The emoji constraint is only justified if, at matched memorability gain, it shows a measurably smaller shift in human emoji choice and EmoSet emotion. The reason to expect it might is that CLIP similarity treats every semantic dimension equally, whereas the emoji distribution weights exactly the reaction dimensions. If the two constraints tie, we drop the emoji model and report that distant affect supervision added nothing here.
Why not just use an 8-class emotion classifier? PerceptCLIP's EmoSet model is accurate (85.28%), but eight coarse classes may miss distinctions such as amused versus affectionate that 20 emoji capture, which is the granularity argument DeepMoji makes for text. This alternative is also an ablation. If it matches emoji on human affect preservation, it is the simpler choice and we will recommend it.
Why not just steer a generator, as in BigGAN direction discovery? That approach edits synthetic images, not a user's photo. Its slope of 0.901 is measured by the same type of network that was optimized. Our cross-scorer and human evaluation is designed to expose exactly that gap, and the bounded edit space keeps the image's content fixed.
Proposed Experiments and Decisive Checks
The test set is 600 held-out LaMem images, covering faces, interiors and landscapes, plus SUN-Mem images to check generalization. Each image appears in exactly one condition per participant. Conditions are counterbalanced across participants, and the filler images in each sequence are identical across conditions so that sequence effects cancel. Because Understanding and Predicting Image Memorability at a Large Scale notes that absolute scores need many observers per image to stabilize, our primary measure is the paired difference in hit rate between an image's original and edited versions, pooled across images. We will size the participant pool so that a 0.03 difference is detectable.
Alongside the memory game, separate raters perform an emoji-choice task and a two-alternative "which one was edited?" realism task on original–edited pairs. Machine metrics come from the held-out MemBoost-style judge and a held-out emoji model.
The decisive check is the human comparison between $\lambda=0$ and the chosen $\lambda$.
If most of the unconstrained editor's human memorability gain disappears under emoji invariance, scorer-driven enhancement works largely by changing affect. That would be an important caution for every editor cited above.
If the gain survives, memorability is editable independently of elicited emotion, and the constrained editor is the practical tool.
A third outcome is a failure mode we commit to reporting. If gains measured by the training scorers do not appear under the held-out judge or in humans, the editor has learned to fool its scorers rather than to make images more memorable.
References
- Changing the Image Memorability: From Basic Photo Editing to GANs
- AMNet: Memorability Estimation With Attention
- How to Make an Image More Memorable? A Deep Style Transfer Approach
- Understanding and Predicting Image Memorability at a Large Scale
- Discovering Multiple and Diverse Directions for Cognitive Image Properties
- The Computational Memorability of Iconic Images
- What Makes an Object Memorable?
- Don’t Judge Before You CLIP: A Unified Approach for Perceptual Tasks
- Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm
- Multimodal Emoji Prediction
- Is Image Memorability Prediction Solved?