ShapeAug++: More Realistic Shape Augmentation for Event Data

Katharina Bendig, René Schuster, Didier Stricker

Introduction

Dynamic Vision Sensors (DVS), commonly known as event cameras, represent a paradigm shift in visual sensing. Unlike traditional cameras that capture full frames at fixed intervals, event cameras asynchronously record pixel-level brightness changes. This unique mechanism offers significant advantages for high-speed applications like autonomous driving, including microsecond-level latency and an exceptionally high dynamic range. However, the development of deep learning models for event data is often hampered by the relative scarcity of large-scale datasets compared to the RGB domain, leading to significant risks of overfitting.

The Limitation of Current Augmentation

To combat overfitting, researchers employ data augmentation. While techniques like flipping or cropping are standard, simulating realistic occlusion—where foreground objects like trees or other vehicles partially block the target—is particularly challenging for event data. Because event cameras are motion-dependent, an occluding object creates a specific temporal "signature."

The predecessor to this work, ShapeAug, introduced simulated occlusions using simple geometric shapes (squares) moving in linear paths. However, real-world occlusions are rarely so uniform. Simple augmentations risk "teaching" the model to ignore specific shapes rather than developing a generalized robustness to visual interference.

ShapeAug++: Simulating Real-World Complexity

ShapeAug++ addresses these limitations by introducing three layers of geometric and kinematic complexity to simulated occluders:

  1. Irregular Polygons: Instead of primitive shapes, ShapeAug++ generates random polygons. By selecting 6 to 10 random points and calculating their convex hull, the method creates a diverse array of irregular shapes that more closely approximate the silhouettes of real-world objects.
  2. Non-linear Kinematics (Bézier Curves): Real-world motion is rarely constant or linear. ShapeAug++ utilizes quadratic Bézier curves to define the trajectory of occluding shapes. This approach naturally introduces acceleration and deceleration. Since event density is directly proportional to velocity, this creates a more realistic and varied distribution of events across the temporal window.
  3. Self-Rotation: To further increase the challenge, shapes rotate around their own axes as they traverse the field of view, creating a dynamic event signature that forces the model to distinguish between the target's motion and the occluder's complex movement.

The augmentation process involves generating frames of these moving shapes, computing the difference between consecutive frames to simulate DVS events, adding artificial noise, and masking the background data where the occluder is present.

Experimental Evaluation

The authors evaluated ShapeAug++ across four diverse event-based classification benchmarks:

  • DVS-CIFAR10 & N-Caltech101: Datasets derived from static images recorded via event cameras.
  • N-CARS: A real-world dataset for vehicle classification.
  • DVS-Gesture: A dataset capturing 11 distinct human hand gestures.

Performance and Robustness

ShapeAug++ consistently outperformed the original ShapeAug and other state-of-the-art methods like EventDrop (random event deletion) and EventMix (sample blending). Notably, the method achieved a 3.7% improvement in top-1 accuracy on the N-Caltech101 dataset.

Beyond raw accuracy, the method significantly enhanced model robustness. When tested against noisy validation data in the DVS-Gesture benchmark, models trained with ShapeAug++ maintained higher performance levels than those trained with simpler geometric or stochastic augmentations. This suggests that the complexity of the augmentation is a critical factor in forcing the neural network to learn more resilient features.

Key Insights

  • Complexity as a Regularizer: The transition from squares to polygons prevents the network from "shortcut learning" (e.g., simply learning to ignore squares) and encourages the extraction of more fundamental object features.
  • The Importance of Motion Dynamics: By simulating non-uniform event density through Bézier curves, the model becomes better at handling the varying data rates inherent to event-based sensing.
  • Dataset-Specific Scaling: The research identified that the optimal maximum size ($s_{max}$) for occluding shapes varies by dataset, indicating that augmentation must be scaled relative to the typical size of the target objects to be effective.

Conclusion and Future Directions

ShapeAug++ demonstrates that more realistic, complex simulations of occlusion can significantly improve the performance and reliability of event-based vision models. While the current implementation uses random placement, future iterations could leverage saliency maps to target critical regions of the input data. Furthermore, while this study focused on classification, the principles of ShapeAug++ are highly applicable to more complex tasks such as object detection and tracking in dynamic autonomous driving environments.

Summary updated

BibTeX
@inproceedings{bendig2024shapeaug,
  author = {Katharina Bendig and René Schuster and Didier Stricker},
  title = {{ShapeAug++: More Realistic Shape Augmentation for Event Data}},
  booktitle = {ICPRAM},
  year = {2024},
  doi = {10.1007/978-3-032-06007-5\_1},
}
Figure