Ground-truth or DAER: Selective Re-query of Secondary Information

Stephan J. Lemmer, Jason J. CorsoICCV 2021 · DOI · arXiv

Abstract

In many computer vision applications, models rely on secondary "seeds"—such as user-provided clicks or coarse labels—to guide inference. While standard practice assumes these seeds are perfect, real-world data is often noisy. This paper introduces DAER (Dual-loss Additional Error Regression), a framework designed to identify and reject seeds that would lead to significant performance degradation. By decoupling seed correctness from the magnitude of resulting error, DAER enables systems to selectively request better information, reducing human effort by over 23% while maintaining high task accuracy.

Introduction

Seeded inference is a common paradigm where a primary input (e.g., an image) is augmented with a secondary "seed" (e.g., a keypoint or category) to constrain the search space or provide context. Traditionally, these seeds are treated as "gold-standard" ground truth. However, seeds sourced from crowdsourcing or automated generators are frequently imperfect.

A critical observation of this work is that error in the input space (the seed) does not map linearly to error in the output space (the task prediction). As shown in Figure 1, a keypoint that is physically close to the ground truth might cause a large geodesic error in viewpoint estimation, while a more distant keypoint might still yield a correct result.

Figure 1: Illustration of how input-space distance from a gold-standard keypoint does not directly correlate with output-space geodesic error.

The DAER Framework

To address this, the authors propose DAER. Rather than simply predicting if a seed is "wrong," DAER estimates the Expected Additional Error—the performance drop incurred by using a noisy seed instead of a perfect one.

The core innovation is a dual-loss training objective:

  1. Correctness Loss: A classification head that estimates the probability that a seed is correct (e.g., matching a ground-truth category or location).
  2. Regression Loss: A regression head that predicts the magnitude of the task error specifically when the seed is incorrect.

By multiplying these two outputs at inference time, the model calculates the expected error. If this value exceeds a threshold, the system rejects the seed and requests a re-query.

Experimental Tasks and Insights

The framework was evaluated on two distinct tasks: Keypoint-Conditioned Viewpoint Estimation (KCVE), utilizing 6,042 new crowdsourced keypoints on PASCAL3D+, and Hierarchical Scene Classification (HSC) using the SUN397 dataset.

Figure 5: Examples from the KCVE and HSC tasks demonstrating that the impact of seed noise is highly dependent on the specific image content.

The researchers found that models exhibit varying sensitivity to noise based on the primary input. For instance, in viewpoint estimation, images showing objects at "transition points" between viewpoints are highly sensitive to small keypoint shifts, whereas "boring" angles are robust to significant noise. This highlights why predicting error magnitude is more valuable than just predicting seed distance.

Results

DAER demonstrates significant improvements in efficiency and ranking. In the HSC task, it reduced the need for manual re-queries by over 23% compared to competitive baselines for the same target performance. In viewpoint estimation, DAER achieved an Area under the Mean Additional Error curve (AMAE) of 0.2864, significantly outperforming baselines like Softmax Response (0.9306) and even "Known Distance" (0.3964), which uses oracle knowledge of seed displacement.

Conclusion

DAER provides a robust, task-agnostic mechanism for managing noisy secondary information in computer vision. By identifying which specific seeds are likely to cause catastrophic failures, it moves beyond general noise robustness toward intelligent, selective human-in-the-loop systems. Future work will explore more complex multi-modal seeds and the potential biases inherent in automated rejection models.

Summary updated

BibTeX
@inproceedings{lemmer2021ground,
  author = {Stephan J. Lemmer and Jason J. Corso},
  title = {{Ground-truth or DAER: Selective Re-query of Secondary Information}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00074},
}
Figure