TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration

Kunyu Peng, Alina Roitberg, Kailun Yang, Jiaming Zhang, Rainer Stiefelhagen

Introduction

Advanced Driving Assistant Systems (ADAS) are critical for modern vehicle safety, requiring the ability to accurately recognize driver activities—such as texting, eating, or paying attention—to manage vehicle control handovers or issue timely warnings. However, developing robust models for Driver Activity Recognition (DAR) faces two significant hurdles. First, real-world driving data is characterized by a long-tail distribution; common actions like "sitting still" dominate the dataset, while safety-critical but rare actions like "reaching for an object" are underrepresented. Second, models often face domain shifts when deployed, where a system trained on dashboard-mounted RGB cameras must function using infrared sensors mounted on rearview mirrors.

Previous approaches utilizing Convolutional Neural Networks (CNNs) or Graph Neural Networks (GNNs) have struggled to generalize across these varying sensor modalities and often fail to accurately classify infrequent behaviors. This paper introduces TransDARC, a framework designed to overcome these challenges through a transformer-based architecture and a novel latent space calibration technique.

Methodology

TransDARC departs from traditional 3D CNN pipelines by leveraging the temporal modeling strengths of transformers and a unique data augmentation strategy performed within the model's internal feature space.

Video Transformer Backbone

At its core, TransDARC utilizes a Video Swin Transformer to extract spatiotemporal features from video clips. Unlike CNNs, which have a limited receptive field, transformers use self-attention mechanisms to aggregate long-term contextual information. This capability is essential for understanding complex human behaviors that unfold over several seconds.

Latent Space Feature Calibration (LSFC)

The primary innovation of TransDARC is its approach to data imbalance. Rather than relying on standard image-level augmentations (like cropping or rotation), the system performs Latent Space Feature Calibration. The model calculates the statistical distributions (mean and covariance) of "common" class clusters within the latent space. For a "rare" class sample, the model identifies the nearest common class clusters and uses their statistical properties to synthetically generate new feature embeddings for the rare class. This process enriches the decision boundaries for underrepresented actions without requiring additional raw data.

The TransDARC architecture features a two-stage training process: Stage 1 focuses on the Transformer backbone, while Stage 2 introduces latent space calibration and an attentional classification head.

Attention-Based Classification & Hard Sample Mining

Instead of a standard fully connected layer, TransDARC employs an attention-based classification head. During the second stage of training, the model actively identifies "hard samples"—predictions with high loss—and forces the network to undergo additional training epochs specifically on these difficult examples to improve robustness.

Experimental Results

The framework was evaluated on the Drive&Act dataset, a massive benchmark containing over 9.6 million frames of distracted driving behaviors across six different camera views and three sensor modalities (RGB, NIR, and Depth).

State-of-the-Art Performance

TransDARC achieved a massive improvement over previous benchmarks. On fine-grained activity recognition, it reached 93.58% validation accuracy and 89.65% test accuracy. This represents an absolute improvement of approximately 21-24% over the previous state-of-the-art (CTA-NET).

Cross-Modal and Rare Class Robustness

The model demonstrated exceptional generalization. When trained on one infrared view and tested on a completely different, unseen infrared view, TransDARC outperformed standard I3D CNN baselines by 13.48%. Furthermore, the combination of the transformer backbone and the LSFC module improved accuracy on rare classes by over 30% compared to older CNN baselines, while simultaneously boosting the accuracy of common classes by 4-5%.

Qualitative comparison showing Ground Truth (GT), Baseline (BA), and TransDARC (PR) predictions. TransDARC correctly identifies complex actions like "Opening Laptop" and "Fetching an Object" where baselines fail.

Conclusion

TransDARC sets a new standard for driver activity recognition by demonstrating that transformers, when paired with feature-level calibration, can effectively mitigate the issues of data imbalance and domain shift. By generating synthetic embeddings in the latent space, the model learns more robust representations that translate well across different camera angles and sensor types. Future work aims to adapt this framework for broader applications, such as recognizing activities of daily living for household robotics.

Summary updated

Abstract

Traditional video-based human activity recognition has experienced remarkable progress linked to the rise of deep learning, but this effect was slower as it comes to the downstream task of driver behavior understanding. Understanding the situation inside the vehicle cabin is essential for Advanced Driving Assistant System (ADAS) as it enables identifying distraction, predicting driver's intent and leads to more convenient human-vehicle interaction. At the same time, driver observation systems face substantial obstacles as they need to capture different granularities of driver states, while the complexity of such secondary activities grows with the rising automation and increased driver freedom. Furthermore, a model is rarely deployed under conditions identical to the ones in the training set, as sensor placements and types vary from vehicle to vehicle, constituting a substantial obstacle for real-life deployment of data-driven models. In this work, we present a novel vision-based framework for recognizing secondary driver behaviours based on visual transformers and an additional augmented feature distribution calibration module. This module operates in the latent feature-space enriching and diversifying the training set at feature-level in order to improve generalization to novel data appearances, (e.g., sensor changes) and general feature quality. Our framework consistently leads to better recognition rates, surpassing previous state-of-the-art results of the public Drive&Act benchmark on all granularity levels. Our code is publicly available at this https URL.
BibTeX
@inproceedings{peng2022transdarc,
  author = {Kunyu Peng and Alina Roitberg and Kailun Yang and Jiaming Zhang and Rainer Stiefelhagen},
  title = {{TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981445},
}
Figure