Research on Image Semantic Quality Evaluation Model for Human Machine Hybrid Intelligence: A Gradient Based Uncertainty Calculation Method

Ziyan Yue, Senyang Lu, Hong LuInformatica 2024 · DOI

Abstract

With the advancement of human-machine hybrid intelligence technology, the importance of images in interaction has become increasingly high, and accurate evaluation of image semantic quality has become crucial. However, traditional evaluation models may be limited in this environment and new methods are needed to improve evaluation accuracy. On the ground of this, an evaluation model for gradient based uncertainty calculation method has been proposed. The study conducts semantic distortion perception analysis at two levels: the overall level (average recognition accuracy) and the sample level (confidence level of a single sample). Experiments have shown that compared to humans, machines have a higher tolerance for distortion, but are weaker in terms of generalization and stability. The proposed method performed well on the complex CIFAR100 dataset. In addition, the accuracy of the framework proposed by the research institute reached 68.03%, which is significantly better than 59.83% for humans and 40.16% for machines, indicating its ability to effectively combine the advantages of different decision-makers.

Introduction

Traditional Image Quality Assessment (IQA) tools, such as Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), are effective at measuring mathematical differences between pixels but often fail to capture "semantic distortion." An image may be mathematically noisy yet perfectly recognizable to a human, or conversely, clear but semantically ambiguous to an AI.

In human-machine hybrid intelligence systems, it is critical to identify when a machine is likely to fail due to image distortion. Current deep learning models often suffer from overconfidence on distorted or out-of-distribution (OOD) data. This research addresses the challenge of accurately estimating uncertainty to determine when a machine should process a task and when it should "reject" the image, passing it to a human expert.

Methodology: Gradient-Based Uncertainty

The proposed model shifts the focus from pixel-level noise to the model's internal sensitivity. By analyzing the Jacobian matrix—the partial derivatives of the output with respect to the input—the system can estimate "cognitive uncertainty."

Taxonomy of uncertainty components in image prediction

To ensure the uncertainty signal remains clear, the researchers utilized Guided Backpropagation. This method truncates negative gradients in ReLU layers to prevent them from canceling out positive gradients, which would otherwise hide the true scale of uncertainty. The study found a direct correlation between the sparsity of features in these layers and the gradient norm, providing a mathematical "fingerprint" for model struggle.

The Human-Machine Hybrid Framework

The system operates on a "rejection learning" basis. A Deep Neural Network (DNN) first attempts to recognize the image; if the calculated uncertainty exceeds a specific threshold, the system redirects the image to a human decision-maker.

Schematic of the human-machine hybrid decision framework

Because human and machine confidence are measured on different scales, the authors developed a Confidence Alignment calibration method. This maps both inputs onto a unified scale, allowing for a fair and efficient "hand-off" between the two entities. This synergy leverages the machine's high-speed processing for clear cases and the human's superior semantic understanding for difficult ones.

Experimental Results and Discussion

The study utilized standard benchmarks like CIFAR-10 and CIFAR-100, alongside a custom Subjective Monitoring Dataset focusing on faces, pedestrians, and license plates subjected to JPEG compression and motion blur.

Key Findings

  • Superior OOD Detection: On the CIFAR-100 dataset, the proposed uncertainty method achieved a False Positive Rate (FPR) of 5% at a 95% True Positive Rate, significantly outperforming established methods like ODIN (44%) and Mahalanobis (13%).
  • Hybrid Synergy: The hybrid framework reached an accuracy of 68.03%. This represents a substantial improvement over the machine's solo performance (40.16%) and even surpasses the human baseline (59.83%).
  • Machine Resilience vs. Brittleness: Experiments revealed that machines often have a higher tolerance for raw distortion than humans. However, they are far less capable of generalizing; once distortion crosses a specific threshold, machine performance drops significantly compared to the more stable human baseline.

Conclusion

By implementing a gradient-based uncertainty calculation and a collaborative decision-making framework, this research provides a new pathway for image quality evaluation. The ability to quantify when a machine is "confused" allows for a hybrid intelligence model that maximizes the strengths of both AI and human experts, improving application performance in fields ranging from security monitoring to autonomous driving.

Summary updated

Abstract

With the advancement of human-machine hybrid intelligence technology, the importance of images in interaction has become increasingly high, and accurate evaluation of image semantic quality has become crucial. However, traditional evaluation models may be limited in this environment and new methods are needed to improve evaluation accuracy. On the ground of this, an evaluation model for gradient based uncertainty calculation method has been proposed. The study conducts semantic distortion perception analysis at two levels. Firstly, at the overall level, the recognition ability is analyzed by analyzing the average recognition accuracy of the dataset. Secondly, at the sample level, recognition ability analysis is conducted on the ground of the confidence level of a single sample. Experiments have shown that compared to humans, machines have a higher tolerance for distortion, but are weaker in terms of generalization and stability. The proposed method performed well on the complex CIFAR100 dataset, achieving the lowest FPR of 95%, the highest TPR of 528%, and the lowest error detection rate of 3.65%. In addition, the accuracy of the framework proposed by the research institute reached 68.03%, which is significantly better than 59.83% for humans and 40.16% for machines, indicating its ability to effectively combine the advantages of different decision-makers. This study is expected to provide new ideas for image quality evaluation, improving the application performance and user experience of images in multiple fields.
BibTeX
@inproceedings{yue2024research,
  author = {Ziyan Yue and Senyang Lu and Hong Lu},
  title = {{Research on Image Semantic Quality Evaluation Model for Human Machine Hybrid Intelligence: A Gradient Based Uncertainty Calculation Method}},
  booktitle = {Informatica},
  year = {2024},
  doi = {10.31449/inf.v48i14.5939},
}
Figure