Abstract
Pedestrians are arguably one of the most safety-critical road users to consider for autonomous vehicles in urban areas. In this paper, we address the problem of jointly detecting pedestrians and recognizing 32 pedestrian attributes from a single image. These encompass visual appearance and behavior, and also include the forecasting of road crossing, which is a main safety concern. For this, we introduce a Multi-Task Learning (MTL) model relying on a composite field framework, which achieves both goals in an efficient way. Each field spatially locates pedestrian instances and aggregates attribute predictions over them. This formulation naturally leverages spatial context, making it well suited to low resolution scenarios such as autonomous driving. By increasing the number of attributes jointly learned, we highlight an issue related to the scales of gradients, which arises in MTL with numerous tasks. We solve it by normalizing the gradients coming from different objective functions when they join at the fork in the network architecture during the backward pass, referred to as fork-normalization. Experimental validation is performed on JAAD, a dataset providing numerous attributes for pedestrian analysis from autonomous vehicles, and shows competitive detection and attribute recognition results, as well as a more stable MTL training.
Introduction
For an autonomous vehicle (AV), knowing a pedestrian's location is only the first step toward safe navigation. To drive safely, the AI must go beyond simple detection and begin to predict behavior: Is that person looking at their phone? Are they about to step into the street? Most existing systems use a two-stage approach: first, a detector finds a bounding box, then a separate classifier analyzes that box. This is computationally expensive and often ignores the surrounding spatial context.
Furthermore, while Multi-Task Learning (MTL)—training one model to perform many tasks—is efficient, it becomes notoriously unstable as the number of tasks increases. The gradients from dozens of different "heads" can collide at the shared "backbone" of the network, leading to "exploding gradients" that destabilize the training process.
Methodology: Composite Fields and Fork-Normalization
The proposed end-to-end model utilizes a composite field framework. Instead of traditional bounding box regression, the model produces spatial maps (fields) that represent different aspects of the scene:
- Presence Field ($S$): Estimates the probability that a pixel belongs to a pedestrian.
- Vector Field ($V$): Points from each pixel toward the center of the pedestrian instance.
- Attribute Fields ($F$): Scalar maps for 32 attributes (e.g., age, gender, motion direction, intention).
To group these pixels into individuals, the system uses the OPTICS clustering algorithm. Once clustered, the model aggregates the attribute predictions across all pixels belonging to that person.
The most significant technical contribution is Fork-Normalization. When the network back-propagates errors during training, the gradients from all 32 tasks meet at a "fork" before entering the shared backbone. Standard summing of these gradients causes the backbone's learning signal to scale uncontrollably with the number of tasks. The authors introduced a weighting parameter ($\kappa$) to normalize these signals. Specifically, their "fork-power normalization" ($\kappa = 1/\sqrt{T}$, where $T$ is the number of tasks) proved most effective at keeping the training stable and balanced.
Experimental Setup and Results
The model was trained and evaluated on the JAAD (Joint Attention in Autonomous Driving) dataset, which includes over 40,000 images from 177 videos. JAAD is unique because it provides high-level behavioral and intentional annotations specifically tailored for AV scenarios, such as "road crossing intention" and "time-to-crossing."
Key Results
- Detection Performance: The model outperformed the popular RetinaNet detector by more than 25% in Average Precision (AP) on the JAAD dataset.
- Stable Scaling: Using fork-normalization, the model's performance actually improved or stayed stable as more attributes were added. In contrast, standard methods saw performance drops of up to 7% when scaling from 2 to 32 tasks.
- Efficiency: The single MTL model uses only 1.6GB of memory to predict 32 attributes. To achieve the same results using 32 separate single-task networks would require 38.4GB, making the MTL approach 24 times more memory-efficient for embedded AV hardware.
Discussion and Future Work
The research demonstrates that the "More Tasks Penalty" often seen in MTL is not an inherent limit of the data, but rather a failure of gradient scaling. By using a field-based approach, the model leverages spatial context, which is particularly useful for low-resolution images where a pedestrian might only be a few pixels wide.
However, some limitations remain. The current model analyzes single images ("snapshots"). While it can predict "intention to cross," incorporating temporal information from video frames would likely improve behavioral prediction accuracy. Additionally, the model currently treats all 32 attributes as independent tasks. Future work could explore "hierarchical sharing" to exploit correlations between attributes like "walking" and "moving direction." Finally, performance in cases of extreme occlusion (more than 75% hidden) remains a challenge for future iterations.
Summary updated