Estimation of Pedestrian Pose Orientation Using Soft Target Training Based on Teacher-Student Framework

DuYeong Heo, Jae-Yeal Nam, ByoungChul Ko

TL;DR

This paper introduces a novel teacher-student framework for estimating pedestrian pose orientation, crucial for applications like autonomous driving. By leveraging a "soft target" training approach, it achieves high accuracy with a more compact and efficient student model compared to traditional deep learning methods.

What Problem Does This Paper Address?

Estimating a pedestrian's pose orientation (i.e., which way they are facing or walking) is vital for intelligent systems like autonomous vehicles and robots. Knowing this helps predict their intentions and avoid collisions. However, accurately estimating this orientation from images, especially in dynamic environments with varying lighting and occlusions, is challenging. Traditional deep learning models for this task require vast amounts of labeled data and significant computational resources, making them impractical for real-time applications or embedded systems. Existing methods often struggle with ambiguity and require complex pre-processing or specialized hardware.

Methods

The core of this paper is a teacher-student framework for model compression and efficient training.

  • Teacher Model: A powerful, deep neural network (ResNet-101) is trained on labeled data (dataset A) to learn pedestrian orientation. To enhance its output, a Random Forest (RF) is also trained using feature maps from the teacher network.
  • Soft Target Generation: Instead of using the hard labels directly for the student, the teacher model's outputs are "softened." This involves combining the softened softmax output of the teacher network and the teacher RF. This soft target provides richer information than hard labels, capturing finer nuances learned by the teacher.
  • Student Model: A shallower, more efficient network (based on DarkNet reference) is trained using these soft targets. This student network is designed to mimic the teacher's performance but with significantly fewer parameters and operations.
  • Wavelet Transform Pre-processing: To improve robustness to lighting changes and preserve spatial/gradient information, a wavelet transform is applied to the input images.
  • Soft Weighting for Final Prediction: For the final orientation prediction, a soft weighting method is employed. It considers the probabilities of the predicted class and its neighbors, adjusting the final angle to be closer to the actual pedestrian's direction, rather than just picking the single highest probability class.

Datasets

  • TUD Dataset: Used for training and evaluating the model in real driving scenarios. It contains images with bounding boxes and orientation annotations.
  • KITTI Dataset: A challenging real-world benchmark used for further validation. It includes various difficulties (easy, moderate, hard) and continuous orientation labels.

Key Insights and Ideas

  • Soft Targets for Efficiency: Using "soft targets" from a teacher model to train a student model is a powerful way to transfer knowledge and achieve high performance with a compressed model, avoiding the need for massive labeled datasets for the student.
  • Teacher Ensemble for Richer Supervision: Combining a deep neural network with a Random Forest as the "teacher" provides a more robust and informative soft target for the student.
  • Wavelets for Robustness: Wavelet transforms are effective pre-processing steps for pedestrian pose estimation, enhancing performance under varying illumination conditions by preserving crucial spatial and gradient information.
  • Soft Weighting for Nuanced Prediction: Moving beyond simple class-wise prediction to a soft weighting scheme that considers neighboring classes leads to more accurate and nuanced orientation estimation.

Key Results

  • Improved Accuracy with Efficiency: The proposed student model achieves competitive pedestrian pose orientation (POE) accuracy, comparable to or better than many state-of-the-art CNN-based methods, while being significantly more computationally efficient.
  • TUD Dataset Performance: On the TUD dataset, the proposed S-Model achieved an Average Precision (AP) of 78.3% and Average Recall (AR) of 79.3%, with a low Average False Positive Rate (AFPR) of 2.9%. This is a good performance considering the model's reduced size.
  • KITTI Dataset Performance: The proposed student model achieved an Average Orientation Similarity (AOS) of 81.1% (Easy), 68.61% (Moderate), and 63.54% (Hard) on the KITTI dataset, outperforming several other advanced methods.
  • Model Compression: The student model is approximately five times smaller in terms of operations and 19.6 times smaller in terms of parameters compared to the teacher model, while maintaining a high level of performance. It also outperforms other compression models like MobileNet V2 and SqueezeNet V1.1 in terms of accuracy.

Limitations and Future Work

  • Limitations: The paper notes that the proposed student model's performance is slightly lower than the teacher model. It also acknowledges that the model might require further improvement in handling severely deformed pedestrian shapes or when pedestrians are occluded by other objects.
  • Future Work: The authors plan to:
  • Develop a faster student framework to improve prediction speed for real-time applications.
  • Enhance the algorithm to reduce false predictions in cases of occlusion or severe deformation.
  • Improve the teacher-student model to recognize pedestrian POE robustly under various environmental changes (day/night, illumination) using a single model, potentially through image-to-image translation pre-processing.

Summary updated

BibTeX
@inproceedings{heo2019estimation,
  author = {DuYeong Heo and Jae-Yeal Nam and ByoungChul Ko},
  title = {{Estimation of Pedestrian Pose Orientation Using Soft Target Training Based on Teacher-Student Framework}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=q34RcPjmRcB},
}

Loading related content...

Figure