Abstract
With the advancement of human-machine hybrid intelligence technology, the importance of images in interaction has become increasingly high, and accurate evaluation of image semantic quality has become crucial. However, traditional evaluation models may be limited in this environment and new methods are needed to improve evaluation accuracy. On the ground of this, an evaluation model for gradient based uncertainty calculation method has been proposed. The study conducts semantic distortion perception analysis at two levels: the overall level (average recognition accuracy) and the sample level (confidence level of a single sample). Experiments have shown that compared to humans, machines have a higher tolerance for distortion, but are weaker in terms of generalization and stability. The proposed method performed well on the complex CIFAR100 dataset. In addition, the accuracy of the framework proposed by the research institute reached 68.03%, which is significantly better than 59.83% for humans and 40.16% for machines, indicating its ability to effectively combine the advantages of different decision-makers.
Introduction
Traditional Image Quality Assessment (IQA) tools, such as Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), are effective at measuring mathematical differences between pixels but often fail to capture "semantic distortion." An image may be mathematically noisy yet perfectly recognizable to a human, or conversely, clear but semantically ambiguous to an AI.
In human-machine hybrid intelligence systems, it is critical to identify when a machine is likely to fail due to image distortion. Current deep learning models often suffer from overconfidence on distorted or out-of-distribution (OOD) data. This research addresses the challenge of accurately estimating uncertainty to determine when a machine should process a task and when it should "reject" the image, passing it to a human expert.
Methodology: Gradient-Based Uncertainty
The proposed model shifts the focus from pixel-level noise to the model's internal sensitivity. By analyzing the Jacobian matrix—the partial derivatives of the output with respect to the input—the system can estimate "cognitive uncertainty."
To ensure the uncertainty signal remains clear, the researchers utilized Guided Backpropagation. This method truncates negative gradients in ReLU layers to prevent them from canceling out positive gradients, which would otherwise hide the true scale of uncertainty. The study found a direct correlation between the sparsity of features in these layers and the gradient norm, providing a mathematical "fingerprint" for model struggle.
The Human-Machine Hybrid Framework
The system operates on a "rejection learning" basis. A Deep Neural Network (DNN) first attempts to recognize the image; if the calculated uncertainty exceeds a specific threshold, the system redirects the image to a human decision-maker.
Because human and machine confidence are measured on different scales, the authors developed a Confidence Alignment calibration method. This maps both inputs onto a unified scale, allowing for a fair and efficient "hand-off" between the two entities. This synergy leverages the machine's high-speed processing for clear cases and the human's superior semantic understanding for difficult ones.
Experimental Results and Discussion
The study utilized standard benchmarks like CIFAR-10 and CIFAR-100, alongside a custom Subjective Monitoring Dataset focusing on faces, pedestrians, and license plates subjected to JPEG compression and motion blur.
Key Findings
- Superior OOD Detection: On the CIFAR-100 dataset, the proposed uncertainty method achieved a False Positive Rate (FPR) of 5% at a 95% True Positive Rate, significantly outperforming established methods like ODIN (44%) and Mahalanobis (13%).
- Hybrid Synergy: The hybrid framework reached an accuracy of 68.03%. This represents a substantial improvement over the machine's solo performance (40.16%) and even surpasses the human baseline (59.83%).
- Machine Resilience vs. Brittleness: Experiments revealed that machines often have a higher tolerance for raw distortion than humans. However, they are far less capable of generalizing; once distortion crosses a specific threshold, machine performance drops significantly compared to the more stable human baseline.
Conclusion
By implementing a gradient-based uncertainty calculation and a collaborative decision-making framework, this research provides a new pathway for image quality evaluation. The ability to quantify when a machine is "confused" allows for a hybrid intelligence model that maximizes the strengths of both AI and human experts, improving application performance in fields ranging from security monitoring to autonomous driving.
Summary updated