Class-Conditional Conformal Triage for ICAO Face-Image Compliance under Class Imbalance

Mustafa Vedat Yildirim, Inna Skarga-BandurovaarXiv 2026 · arXiv

This study explores how to add a layer of "principled uncertainty" to automated face-image compliance systems. By applying Conformal Prediction, the authors move beyond simple pass/fail verdicts to a three-way triage system: ACCEPT, REJECT, or RETAKE.

TL;DR

The researchers developed a triage layer for ICAO face-compliance analyzers that quantifies uncertainty using Mondrian (class-conditional) conformal prediction. They demonstrate that while standard "marginal" conformal methods fail under class imbalance—significantly under-covering rare image violations—the Mondrian approach maintains reliable coverage guarantees for both compliant and non-compliant images.

What Problem Does This Paper Address?

Passports and identity documents must meet strict ICAO/ISO standards (e.g., neutral expression, no sunglasses, sharp focus). Automated systems typically provide a binary pass/fail for these criteria. However, a binary "fail" doesn't distinguish between a clear violation and a "borderline" case where the algorithm is simply unsure.

This creates two costly errors:

  1. False Rejection: Forcing a user to retake a perfectly good photo.
  2. False Acceptance: Accepting a photo that will later be rejected by a human agent or document printer.

Furthermore, violations are rare in the real world. In a dataset where 95% of images are compliant, a model can achieve 95% "coverage" while failing to catch a single actual violation. This class imbalance makes traditional uncertainty quantification unreliable.

Methods

The authors propose a modular, "analyser-agnostic" pipeline that sits on top of an existing tool (in this case, BioGaze).

Figure 1. Conformal layer on a rule-based ICAO analyser: BioGaze features, a classifier for each criterion, split conformal prediction sets, and
  1. Feature Extraction: They use 14 binary decisions and 7 auxiliary metrics from BioGaze as inputs.
  2. Meta-Classification: They train lightweight, class-balanced logistic regressors for each of the 14 criteria to estimate the probability of a violation.
  3. Conformal Calibration: They convert these probabilities into "prediction sets" (sets of plausible labels). They compare:
  4. Marginal Conformal Prediction: One threshold for all images.
  5. Mondrian (Label-Conditional) Conformal Prediction: Separate thresholds for "compliant" and "violation" classes to ensure the error rate is controlled for each.
  6. Triage Rule: If the prediction set is {compliant}, the image is ACCEPTED. If {violation}, it is REJECTED. If it contains both (or neither), it is sent for a RETAKE.

Datasets

The study primarily uses the TONO benchmark, a synthetic dataset designed for ICAO compliance. It contains 275 identities with specific violations. To prevent "identity leakage," the authors used a subject-disjoint split (60% train, 20% calibration, 20% test), ensuring the model never sees the same person in training and testing.

Key Insights and Ideas

  • The Failure of Marginal Coverage: Marginal conformal prediction is "blind" to class imbalance. In their tests, it achieved the overall 90% coverage target but crashed to as low as 37% coverage for specific violation types. It essentially "borrows" accuracy from the easy majority class to hide failures on the difficult minority class.
  • Mondrian as the Equalizer: By calibrating thresholds separately for each class, Mondrian CP successfully brought violation-class coverage back to the ~90% target across all 14 criteria.
  • The "Accuracy Artefact": Interestingly, the authors found that most of the reduction in False Positive Rates (FPR) came from the logistic regression layer itself, not the conformal wrapping. This suggests that even simple "meta-classifiers" can significantly improve upon raw rule-based systems.
  • The Joint Coverage Curse: When you have 14 different criteria, the probability of at least one being uncertain is high. Using a strict Bonferroni correction to guarantee that the entire image is correctly assessed resulted in nearly 100% of images being sent for retakes—a mathematically sound but practically useless result.

Key Results

  • Violation Coverage: Marginal methods failed significantly (mean coverage of 0.81, with some criteria at 0.37). Mondrian methods stayed on target (means of 0.89 to 0.92).
Figure 3. Coverage on the violation class for each criterion, marginal
  • Error Reduction: The logistic layer reduced the criterion-level FPR from 0.145 (raw BioGaze) to 0.048.
  • Efficiency Trade-offs: While Mondrian provides a formal guarantee, traditional "tuned" abstention rules (like Chow’s rule) often achieved lower FPRs for the same retake rate. The choice between them is a trade-off between empirical performance and statistical auditability.
Risk vs. Abstention: Tuned rules (lines) often outperform conformal points (dots) on raw FPR, but lack the distribution-free guarantee.

Limitations and Future Work

  • Synthetic Data: TONO is synthetic and each image contains exactly one violation. Real-world images often have multiple overlapping issues (e.g., poor lighting and a non-neutral expression).
  • The "Empty Accept" Problem: Because TONO only contains violations, the authors couldn't fully validate the "ACCEPT" decision path or image-level specificity.
  • Scaling Joint Coverage: Future work is needed to find a "tighter" simultaneous guarantee than Bonferroni, which is too conservative for multi-criterion triage.
  • Real-World Validation: The authors identify the DFIC dataset as the next logical step for testing these methods on real human subjects with complex, multi-attribute violations.

Summary updated

Abstract

Automated ICAO/ISO face-image compliance analysers provide criterion-level pass/fail decisions, yet these verdicts do not quantify uncertainty when deciding whether to accept, reject, or retake a photograph. We investigate conformal prediction as a triage layer for BioGaze, evaluating 14 compliance criteria on the synthetic TONO benchmark. Without retraining the analyser, we fit per-criterion logistic classifiers and compare marginal and label-conditional (Mondrian) conformal prediction with conventional abstention rules using subject-disjoint splits. Marginal conformal prediction achieved approximately 0.90 overall coverage but substantially under-covered violations, with a ten-partition mean of 0.37 for one criterion. The corresponding Mondrian means ranged from 0.89 to 0.92 across criteria. However, most of the reduction in criterion-level false positives came from the logistic layer (0.145 to 0.048), while tuned abstention rules achieved lower false-positive rates than Mondrian at comparable retake rates. Bonferroni correction for joint coverage across 14 criteria required retaking nearly all images. These findings show that class-conditional conformal prediction can achieve near-target empirical coverage for minority violation classes and reveal trade-offs between statistical coverage, classification performance, and retake requirements in multi-criterion face-image compliance assessment.
BibTeX
@misc{yildirim2026class,
  author = {Mustafa Vedat Yildirim and Inna Skarga-Bandurova},
  title = {{Class-Conditional Conformal Triage for ICAO Face-Image Compliance under Class Imbalance}},
  year = {2026},
  eprint = {2611.25342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25342},
}
Figure