TL;DR
HALO (Human-Aligned Layered transfOrmations) is an end-to-end framework for image retargeting that changes an image’s aspect ratio without squishing subjects or cutting out critical content. By decomposing images into salient and background layers and training with a structure-aware perceptual loss adapted from DreamSim, HALO produces artifact-free retargeted images in under a second and outperforms prior approaches in human preference by over 13%.
What Problem Does This Paper Address?
Display platforms have wildly different aspect ratios—from widescreen desktop monitors to vertical mobile reels. Traditional solutions either stretch the image (distorting subject geometry) or crop it (discarding potentially crucial context).
Previous automated retargeting approaches (including seam carving, patch-matching optimization, and single-image generative models) struggle with major trade-offs:
- Geometric distortion & repetition: Methods often warp foreground subjects or duplicate patterns (e.g., doubling car wheels).
- Out-of-Boundary (OOB) artifacts: Applying a single non-uniform warp across an entire image often stretches the background beyond canvas limits or leaves blank gaps.
- Speed: Per-image optimization methods (like Patch Nearest Neighbors or single-image diffusion models) can take anywhere from tens of seconds to hours per picture.
Methods
HALO formulates image retargeting as a fast, feed-forward warping task guided by human perceptual priorities:
- Layered Decomposition: Human vision is acutely sensitive to distortions in salient subjects, but much more tolerant of background adjustments. HALO splits the input into a salient layer (via the off-the-shelf segmenter MDSAM) and a non-salient layer, with foreground holes inpainted via LaMA.
- Multi-Flow Network (MFN): The network processes both the original image and a naively resized reference using a shared CNN encoder and $L=3$ cross-attention blocks. It outputs separate affine transformation parameters for the salient and background layers.
- Perceptual Structure Similarity Loss (PSSL): Training retargeting models without paired ground truth usually relies on perceptual metrics like LPIPS, which are surprisingly blind to aspect-ratio distortion. The authors turn to DreamSim, a metric tuned for mid-level layout and structure. Because DreamSim naively normalizes inputs to $224 \times 224$ (masking distortion), the authors introduce layout augmentation: creating pseudo-ground-truth targets by applying random affine perturbations (uniform scaling and translation) at the target aspect ratio.
- Non-Saliency Regularization ($\mathcal{L}_{NSReg}$): A pixel-wise penalty on background warping prevents extreme shifts that cause out-of-boundary artifacts.
Input Image ───┬─► Saliency Split & Inpaint ──► Salient / Background Layers ──┐
│ │
└─► Resized Image + Cross-Attn ─► Predict Layer Warp Fields ───┴─► Composited Output
Datasets
- Training: UHRSD (Ultra-High Resolution Saliency Dataset), containing 4,932 training images and 988 test images across varied scenes (landscapes, street views, wildlife).
- Evaluation:
- RetargetMe Benchmark: 80 standard evaluation images tested across multiple scale factors ($0.5\times$, $0.75\times$, and $1.25\times$).
- In-the-wild: 400 uncurated images from Unsplash to verify zero-shot generalization.
Key Insights and Ideas
- Single-flow warping is inherently constrained: Trying to retarget an image with a single continuous warp field creates an irreconcilable conflict between preserving foreground proportions and filling the frame. Independent affine flows for foreground and background sidestep out-of-boundary stretching.
- Standard perceptual losses hide aspect ratio stretching: A naively squished image resized to a square will appear "normal" to standard deep feature extractors. Generating an undistorted, scaled/translated reference image ($I_{aug}$) provides the anchor DreamSim needs to penalize squished geometry.
- Emergent affordance awareness: The decoupled representation allows foreground subjects to move relative to background elements while preserving contextual plausibility (e.g., maintaining a fish swimming behind sea anemone tentacles rather than shearing through them).
Key Results
- User Study: In a 16-participant study across all 80 RetargetMe images (1,280 pairwise votes), HALO captured 43.44% user preference, beating the runner-up (MagicFixup at 30.23%) by +13.21% and outclassing RL and generative baselines (Self-Play-RL at 20.86%; GPDM at 5.47%).
- Quantitative Quality: HALO achieved state-of-the-art across combined metrics on RetargetMe:
- CLIP Similarity (Content): 90.2% (vs. 88.7% for Self-Play-RL; 84.8% for MagicFixup)
- DreamSim Similarity (Structure): 78.0% (vs. 76.2% for Self-Play-RL)
- Overall Average: 73.2%
- Inference Speed: Runs in 0.59 seconds on an NVIDIA A100 for a $1024 \times 813$ image, significantly faster than patch-based or diffusion methods (which take 11 seconds to several hours).
Limitations and Future Work
- Saliency Map Dependency: If the saliency detector fails to segment an object's shadow or reflections, the subject may look disconnected or "float" in the final composition.
- Scenes Without Clear Salience: In texture-heavy images or uniform landscapes lacking distinct focal objects, saliency masks become ambiguous. The authors note the framework defaults to cropping (via an all-ones mask) under these conditions.
- Future Directions: Integrating joint shadow/reflection association modules and exploring fully end-to-end joint saliency-retargeting architectures.
Summary updated