Introduction
Standard image generation models often produce blurry results because they rely on pixel-wise loss functions that fail to capture human perception. This paper introduces DeePSiM (Deep Perceptual Similarity Metrics), a hybrid loss function that combines deep feature similarity, adversarial training, and pixel-space distance to generate sharp, semantically accurate images even from highly compressed representations.
The Problem with Pixel-wise Loss
In computer vision, "similarity" is a complex concept. A simple shift of an image by one pixel can result in a large Squared Euclidean (L2) distance, despite the images being perceptually identical. Conversely, a blurry average of two images might yield a low L2 error while appearing unnatural to a human observer.
When training generative models like autoencoders, standard pixel-wise losses force the model to "hedge its bets." If the model is uncertain about the exact location of a feature, it averages all possible locations, resulting in characteristic blurriness. The authors aim to bridge this gap by creating a loss function that prioritizes "perceptual similarity"—the textures, shapes, and structures that define human visual recognition.
The DeePSiM Approach
The authors propose a multi-part loss function, $L$, which is a weighted sum of three distinct terms:
- Feature Loss ($L_{feat}$): Instead of comparing pixels, the model compares the activations of a pre-trained "comparator" network (typically AlexNet). By measuring distance in high-level feature space, the model becomes invariant to small spatial shifts while remaining sensitive to important structures like edges and textures.
- Adversarial Loss ($L_{adv}$): Utilizing a discriminator network, the generator tries to produce images that are indistinguishable from real natural images. This acts as a "natural image prior," forcing the model to produce realistic textures.
- Pixel Loss ($L_{img}$): A standard L2 distance in pixel space is maintained to provide training stability.
The generator uses "up-convolutional" layers to project low-dimensional features back into full-resolution images.
Key Insights
- The "Mean" is the Enemy of Sharpness: Euclidean distance in pixel space leads to "mean" results. If a model is uncertain, it averages possibilities, creating blur.
- Feature Spaces are Contractive: A single feature vector can represent many different images. The adversarial loss ensures the generator chooses a natural-looking version from the set of possibilities.
- Supervision is Optional: Interestingly, the "comparator" network doesn't strictly need labeled data. Even using a network with random weights or one trained on unsupervised video data (VideoNet) yielded better results than pixel-loss alone.
- Deep Layers Preserve Information: A striking finding is that recognizable images can be reconstructed from the very last layer of AlexNet (FC8), suggesting that high-level abstractions preserve significant information about color, position, and texture.
Results and Conclusion
DeePSiM was evaluated on benchmarks like STL-10 and ImageNet. Qualitatively, the approach produced images that were significantly sharper and more recognizable than those produced by standard MSE/L2 losses.
Quantitatively, while DeePSiM resulted in higher Euclidean error, it led to better classification accuracy (up to 52.3% on STL-10 vs 34.6% for SE loss), proving the learned features are more semantically meaningful. The authors also successfully "inverted" AlexNet, showing that high-level layers contain enough information to reconstruct the original scene's layout.
While hyperparameter sensitivity remains a challenge, DeePSiM represents a significant step toward generative models that align more closely with human perception. Future work may explore sampling multiple different images that satisfy the same feature representation.
Summary updated