Utilizing dataset affinity prediction in object detection to assess training data

Stefan Becker, Jens Bayer, Ronny Hug, Wolfgang Hübner, Michael Arens

Abstract

Modern object detection models often rely on massive, heterogeneous data pools to achieve robustness. However, the contribution of individual datasets within these pools is often opaque. We present a method to quantify this contribution by integrating a "dataset affinity head" into the YOLOv7-X architecture. This allows the model to predict the source dataset most relevant to a specific detection during inference. Our results demonstrate that this affinity score can guide training data pruning—reducing data volume while maintaining performance—and provide insights into spectral generalization and domain bias.

Introduction

In computer vision, the prevailing strategy for improving model performance is "data pooling"—combining multiple datasets (e.g., MS COCO, VisDrone, KITTI) into a single training set. While this increases diversity, it introduces challenges. Large domain gaps between pooled datasets and the target application can degrade performance, while highly correlated datasets introduce redundancy. Traditionally, measuring the "information value" of a specific dataset within a pool is difficult. We address this by proposing an ante-hoc explanation method that enables the model to identify the training sources it relies on for its predictions.

Proposed Method: The Dataset Affinity Head

The core of our approach is the modification of the standard YOLOv7-X detector to include a dataset affinity prediction task.

Figure 1: Architecture of the modified YOLOv7-X with an additional dataset affinity head alongside standard objectness, classification, and localization heads.

As shown in Figure 1, we extend the multi-scale feature extraction pipeline. While standard heads predict objectness, class labels, and bounding boxes, the new affinity head performs multinomial logistic regression to predict the source dataset. We treat the dataset names in the training pool as distinct classes. To train this head, we introduce an integrated loss function that includes an affinity loss term ($L_{aff}$), ensuring the model learns to associate specific visual features with their originating datasets. To facilitate pooling, we unified labels into a "vehicle" super-category and employed image patching to standardize resolutions across disparate sources.

Experimental Setup

The model was trained on a heterogeneous pool including MS COCO (general), DETRAC (surveillance), UAVDT and VisDrone (drones), and FLIR (visible and infrared). Evaluation was conducted on unseen datasets such as MODISSA (proprietary VIS/IR) and MSOD (Multi-Spectral Object Detection), as well as several out-of-domain datasets like DroneVehicle and KAIST MPD.

Results and Discussion

The affinity head provides a built-in sanity check for model logic. For instance, when processing infrared (IR) imagery, the model correctly assigns high affinity scores to the FLIR-IR training set.

Figure 4: Qualitative detection results across diverse domains, including high-altitude drone perspectives, ground-level visible light, and infrared imagery.

Smart Pruning and Efficiency

By analyzing affinity scores on a validation set, we identified "supporting" datasets. We found that the training pool could be significantly pruned by removing datasets with low affinity scores. In the MODISSA VIS test, pruning the pool to only the two most relevant datasets resulted in a [email protected] of 0.913, nearly identical to the full pool's 0.921, despite a massive reduction in training data.

Spectral Generalization

Tests on the MSOD dataset revealed a clear shift in affinity across the electromagnetic spectrum. For visible light (VIS), the model relied on MS COCO and FLIR-VIS. As the wavelength increased toward Far-IR (FIR), affinity scores shifted almost entirely to the FLIR-IR training data, demonstrating the model's ability to recognize spectral-specific features.

Identifying Dataset Bias

Affinity scores also reveal potential biases. If a model identifies a vehicle in a drone shot but attributes it to a ground-level dataset like MS COCO, it indicates a reliance on features that may not generalize well to high-altitude perspectives.

Conclusion

Integrating dataset affinity prediction into object detectors offers a powerful tool for understanding and optimizing training data. This method enables efficient data pruning, provides insights into domain generalization, and serves as a diagnostic tool for identifying dataset bias. Future work will explore scaling this approach to more complex object categories and moving toward sample-level affinity predictions.

Summary updated

BibTeX
@inproceedings{becker2024utilizing,
  author = {Stefan Becker and Jens Bayer and Ronny Hug and Wolfgang Hübner and Michael Arens},
  title = {{Utilizing dataset affinity prediction in object detection to assess training data}},
  booktitle = {Robovis},
  year = {2024},
  doi = {10.1007/978-3-031-59057-3\_17},
}
Figure