Utilizing Dataset Affinity Prediction in Object Detection to Assess Training Data

Stefan Becker, Jens Bayer, Ronny Hug, Wolfgang Hübner, Michael Arens

TL;DR

Researchers from Fraunhofer IOSB have developed a way for object detectors to "point their finger" at the specific training datasets that helped them identify an object. By adding a lightweight dataset affinity head to a standard YOLOv7 pipeline, the model provides real-time feedback on which data sources are most relevant to a target domain, allowing for massive training data pruning without sacrificing accuracy.

What Problem Does This Paper Address?

In computer vision, the mantra is often "more data is better." To build robust models, researchers frequently use data pooling—combining multiple datasets (like MS COCO, VisDrone, and KITTI) into one giant training bucket.

However, data pooling isn't a free lunch. If a new dataset is too different from your target domain (a large "domain gap"), it can actually degrade performance. Conversely, if datasets are highly correlated, they add redundancy and waste computational resources. The challenge is that it’s traditionally very hard to measure the "information value" of an individual dataset once it's been mixed into a pool. We usually don't know which specific training samples are doing the heavy lifting during inference.

Methods

The authors propose a novel ante-hoc explanation method. Unlike "post-hoc" methods (like Grad-CAM) that try to explain a model's behavior after it's trained, this approach builds the explanation directly into the architecture.

  1. The Affinity Head: They extended the YOLOv7-X detector by adding an additional inference head. While the standard heads predict "what" (class) and "where" (bounding box), this new head predicts "where did I learn this from?" (dataset affinity).
  2. Multinomial Logistic Regression: The affinity task is treated as a classification problem where the "classes" are the names of the source datasets in the training pool.
  3. Integrated Loss: They modified the total loss function to include an affinity loss term ($L_{aff}$), ensuring the model learns to associate specific visual features with their originating datasets during training.
  4. Data Alignment: To make pooling work, they unified various vehicle labels (car, van, truck, etc.) into a single "vehicle" super-category and used image patching to handle different resolutions across datasets.

Datasets

The study utilized a diverse "heterogeneous pool" for training and several distinct datasets for evaluation:

  • Training Pool: MS COCO (general), DETRAC (low-altitude surveillance), UAVDT & VisDrone (mid/high-altitude drones), and FLIR (ground-level, split into VIS and IR spectral ranges).
  • Evaluation (Unseen) Datasets:
    • MODISSA: A proprietary dataset from a Fraunhofer measuring vehicle, featuring both visible (VIS) and infrared (IR) cameras.
  • MSOD (Multi-Spectral Object Detection): A public dataset covering VIS, Near-IR (NIR), Mid-IR (MIR), and Far-IR (FIR).
  • Others: DroneVehicle, KAIST MPD, DDAD, REDS, and AMVD were used for qualitative "out-of-domain" testing.

Key Insights and Ideas

  • The Model "Knows" its Roots: The affinity score acts as a built-in sanity check. For example, when the model sees a vehicle in an infrared test image, the affinity head correctly assigns high scores to the FLIR-IR training set.
  • Smart Pruning: By analyzing which datasets had the highest affinity scores on a small validation set, the authors could identify the "supporting" datasets. They found they could prune the training pool significantly—removing entire datasets that provided low affinity—and maintain nearly identical mAP (mean Average Precision).
  • Identifying Dataset Bias: The affinity head reveals which datasets are "dominating" the model's logic. If a model identifies a car in a drone shot but attributes it to MS COCO (a ground-level dataset), it suggests the model is relying on features that might not generalize well to high-altitude perspectives.

Key Results

  • Efficiency: The researchers demonstrated that object detectors can be trained on a significantly sparser set of training samples without losing detection accuracy.
  • Spectral Generalization: In the MSOD (Multi-Spectral) tests, the model showed a clear "shift" in affinity. For visible light (VIS), it relied on MS COCO and FLIR-VIS. As the wavelength increased toward Far-IR (FIR), the affinity scores shifted almost entirely to the FLIR-IR training data.
  • Performance Stability: In the MODISSA VIS test, the full pool achieved a [email protected] of 0.921. After pruning the training set to only the two most "relevant" datasets identified by the affinity head, the mAP stayed at a high 0.913, despite using much less data.

Limitations and Future Work

  • Category Scope: The current study focused exclusively on the "vehicle" super-category. It remains to be seen how well this affinity prediction scales to hundreds of diverse object classes with overlapping features.
  • Granularity: The affinity is predicted at the dataset level. Future work could potentially move toward sample-level affinity, identifying specific images or clusters within a dataset that are most beneficial.
  • Automated Data Selection: While the paper shows that pruning is possible based on these scores, the next step would be an automated pipeline that dynamically selects the best training mixture for a specific target domain without human intervention.

Summary updated

BibTeX
@inproceedings{becker2024utilizing,
  author = {Stefan Becker and Jens Bayer and Ronny Hug and Wolfgang Hübner and Michael Arens},
  title = {{Utilizing Dataset Affinity Prediction in Object Detection to Assess Training Data}},
  booktitle = {Robovis},
  year = {2024},
  doi = {10.1007/978-3-031-59057-3\_17},
}
Figure