Abstract: Pedestrian image generation in the desired pose can be used in a wide range of applications e.g., person re-identification and tracking which are among the fundamental challenges in self-driving cars. This is a hard task because it should be invariant to a set of nuisances such as body poses, illuminations, or changes in camera viewpoint. In this work, we want to study the task of synthesizing a latent canonical view of a pedestrian that will potentially be robust to the mentioned factors of nuisances. Our goal is to generate the unique frontalized view of a pedestrian observed in the wild. The generated image should visually be the same regardless of the body pose. We propose a new generative framework that goes beyond the 1 to 1 supervision commonly used. We propose to jointly reason on multiple inputs and outputs thanks to a carefully chosen loss function acting as a regularizer. Our experiments show the benefits of our framework on challenging low-resolution datasets.
Introduction: The Challenge of Person Re-identification
For autonomous vehicles, tracking pedestrians across different camera views is a critical safety requirement. This task, known as person re-identification (re-ID), is complicated by the fact that pedestrians are non-rigid objects. Their appearance changes significantly based on body pose, camera angle, and lighting conditions. These variations, often termed "nuisances," can confuse standard tracking algorithms.
The research presented at the 19th Swiss Transport Research Conference proposes a solution: synthesizing a "canonical" frontal view of a pedestrian. By transforming any arbitrary sighting of a person into a standardized frontal pose, the system creates a stable representation that simplifies the tracking and identification process.
Methodology: A Generative Framework for Frontalization
The authors developed a framework based on Generative Adversarial Networks (GANs) designed to go beyond simple 1-to-1 image translation. The architecture integrates several components to ensure both visual realism and identity consistency.
System Architecture
The process begins with pose detection using the OpenPose model to extract the skeletal structure of the pedestrian. This pose information, along with the source image features, is fed into an encoder-decoder Generator. The Generator is tasked with synthesizing a new image of the person in a predefined "front pose."
Joint Reasoning and Loss Functions
A key innovation of this work is "joint reasoning." Instead of processing images in isolation, the model looks at a batch of images of the same individual. To guide the training, the researchers employed a combination of loss functions:
- Adversarial Loss: Encourages the generator to produce images that are indistinguishable from real pedestrian photos.
- L1 and Style Loss: Ensures the synthesized image retains the specific colors, textures, and identity of the original person.
- Consistency Regularizers: These penalize the model if it generates different-looking frontal views for the same person when starting from different input poses. This forces the model to find a stable, pose-invariant representation.
Experimental Setup and Results
The study utilizes the Market-1501 dataset, a standard benchmark in the re-ID community consisting of 32,668 images of 1,501 different people. Images are captured from six different cameras, resulting in low-resolution shots with significant pose diversity and frequent occlusions.
Key Findings
- Stability over Realism: While many GANs focus on making images look "pretty," this work prioritizes stability. The goal is for the model to produce the exact same output image regardless of whether the input was a side view, a back view, or a 45-degree angle.
- Pose Invariance: The model successfully synthesized frontal views even when the input images featured heavy occlusions or extreme angles.
- Visual Consistency: Qualitative comparisons demonstrate that when fed different angles of the same pedestrian, the proposed method's outputs are nearly identical, whereas previous state-of-the-art methods showed significant flickering or identity shifts.
Conclusion and Future Work
This work demonstrates the potential of generative models to create stable, canonical representations of pedestrians for autonomous driving applications. While the current results show high consistency, the authors note that image resolution and background "bleeding" remain areas for improvement. Future work will focus on refining image fidelity through body masking and testing the framework's generalizability across more diverse environmental datasets.
Summary updated