Optimal transport is a way to match two collections of points, for example unlabeled image features and a handful of class prototypes, by finding the cheapest way to move one onto the other. It is increasingly used to generate pseudo-labels, which are machine-guessed labels, when real labels are scarce. The standard version must move every point, so a few outliers can distort the whole matching. The usual fix is unbalanced transport, which lets the solver ignore mass it would be too expensive to move. That fix brings its own problem. Whether a point is kept or thrown away depends on a penalty weight that nobody knows how to set. It also fails outright on "second-type" outliers: contaminating points that sit closer to the target than genuine samples do, as shown in On making optimal transport robust to all outliers.
We propose putting a human in the loop at exactly this weak joint. The human is not asked "what class is this?" but a cheaper binary question: "does this point belong in the transport at all?" The points worth asking about are those whose keep-or-discard decision would most change the downstream pseudo-labels. We score them by differentiating a downstream utility with respect to each point's mass through the unbalanced transport solution. Each answer does two jobs. It pins that point's mass to zero or to full weight, and it adds one observation for re-estimating the global discard penalty, so the answers gradually calibrate the solver's hyperparameter. The claim is that under label scarcity with a contaminated unlabeled pool, a budget of keep/discard answers chosen this way does more for pseudo-label accuracy per annotator-second than the same budget spent on class labels chosen by uncertainty or by balanced-transport influence.
What exists and what is missing
The closest work is AvAtar. It selects which pairwise matches a human should supply by differentiating a global alignment utility through an entropy-regularized, mass-preserving transport plan. It uses the adjoint method, a way of getting that gradient by solving one extra linear system, and solves the system with conjugate gradient in linear time. Its authors state that the unbalanced case is left open. Because every point must be transported, outliers in AvAtar's pool shape both the plan and the influence scores. Our proposal borrows AvAtar's adjoint machinery but changes the object being queried. AvAtar asks for supervision on the coupling, meaning which point matches which. We ask for supervision on the marginals, meaning how much mass each point carries. That kind of question only exists once mass can be created or destroyed.
Three lines of offline robust transport motivate the design and constrain it. P$^2$OT produces pseudo-labels for imbalanced clustering by partial transport with a "trash bin" cluster, raising the transported mass along a fixed sigmoid schedule. Its authors flag that schedule as arbitrary. Our human-calibrated penalty is meant to replace it with one driven by evidence. Outlier-Robust DRO via Unbalanced OT and Outlier-Robust Optimal Transport show that relaxing marginals yields provable robustness when the contamination level or penalty is known. Nietert et al.'s $\mathsf{W}_p^\varepsilon$, which is minimax optimal under Huber contamination, needs the outlier fraction $\varepsilon$ as input. We supply the missing estimate from human answers instead of from an oracle.
SROT is the most direct constraint. It shows that distance-based discarding keeps close-in outliers, and it patches this with a separate adversarially trained source/target classifier. We treat SROT's failure case as the regime where human mass queries should matter most, and we use SROT as a baseline. Automatic Outlier Rectification via Optimal Transport pulls in the other direction: it argues outliers should be moved back toward the data rather than deleted. We take this as a warning that "discard" is not always the right answer. For that reason the human's options are keep, discard, or keep-with-label.
On the annotation side, binary questioning for human-in-the-loop annotation motivates yes/no queries as low-effort. ActLocalizer shows that correcting about 5% of items and propagating the corrections can pay off. It reports +1.18% mAP on THUMOS14 and a 78% cut in annotation time, but it relies on visual inspection and a path heuristic to choose what to correct, where we use a transport-derived score.
Core mechanism
Let $x_1,\dots,x_n$ be unlabeled features with masses $a_i$ (initially $1/n$). Let $c_1,\dots,c_K$ be class prototypes built from the few labeled examples, with masses $b_k$. Pseudo-labels come from the plan $\pi$ that solves
$$\min_{\pi \ge 0}\ \langle \pi, C\rangle + \epsilon H(\pi) + \lambda\,\mathrm{KL}(\pi \mathbf{1} \,\|\, a) + \mathrm{KL}_{\text{hard}}(\pi^\top \mathbf{1} = b),$$
In this objective, $C_{ik}$ is the feature-to-prototype cost and $H$ is an entropy term that makes the solve fast and differentiable. The penalty $\lambda$ sets how cheaply the solver may drop mass from unlabeled points; the prototype side is held fixed, as in semi-unbalanced transport. After the solve, each point keeps a fraction $\pi\mathbf{1}_i / a_i$ of its mass, which we call its retained mass.
A human answer "discard" sets $a_i = 0$. "Keep" or "keep with label $y$" removes point $i$ from the relaxation, so its mass is enforced exactly, and in the labeled case also constrains its row of $\pi$. The query score for point $i$ is the gradient of a utility $U$ with respect to its mass, $|\partial U / \partial a_i|$, weighted by the model's uncertainty over the binary keep/discard answer. The utility is agreement of the pseudo-labels with a held-in set of human-verified points, smoothed by the plan. The gradient is computed through the fixed point of the unbalanced Sinkhorn iterations with an adjoint system, extending AvAtar's conjugate-gradient solve to the relaxed marginals.
After each batch of answers, $\lambda$ is re-fit by maximum likelihood. The fit treats retained mass as a predicted probability of "keep" and matches it against the human verdicts. The human answers therefore double as a calibration set for the one hyperparameter that controls robustness.
Research Program
The work has three parts. The first is derivation and engineering. We derive the adjoint system for the semi-unbalanced plan and check that it keeps AvAtar's linear-time structure under sparsity. We then build the loop around a fixed pretrained or self-supervised backbone: seed prototypes from $L$ labels, solve, score, query, update masses and $\lambda$, retrain the classifier on pseudo-labels, and repeat.
The second part is analysis. We ask when the mass gradient and a far cheaper proxy disagree. The proxy is the closed-form marginal slack $1 - \pi\mathbf{1}_i/a_i$. We expect agreement on far-away outliers, where slack is already informative, and disagreement on second-type outliers, where slack is near zero because the solver happily transports them. This prediction is the crux, and it is falsifiable.
The third part is annotator cost. With simulated oracles we measure accuracy per query. In a small human study we measure seconds per keep/discard answer versus per class label on the same images, so that budgets can be compared in time rather than in counts.
Baselines
The naive reference point is random querying of class labels at the same budget, fed into the same unbalanced pseudo-labeler with $\lambda$ fixed by a common default. It tells us whether any active choice matters.
The strongest open-set semi-supervised competitors are OpenMatch and SCOMatch. They are the methods built to handle the same threat, unknown-class distractors in the unlabeled pool, without transport. OpenMatch reports a 10.4% error rate on CIFAR-10 with 300 labels and a 6-known/4-unknown class split, against 20.3% for the prior best, MTC. SCOMatch reports 54.2% accuracy on TinyImageNet at a 0.3 mismatch ratio. We rerun both on our splits, with the same backbone and training schedule. Each receives its extra labels either randomly or by entropy, with the extra labels matched to our budget in annotator-seconds from the timing study. Our method must beat both at equal time budget, and must match OpenMatch's reported 10.4% in the standard 300-label configuration before any extra queries are added. Otherwise the pseudo-labeler itself is the weak link.
The strongest transport-based active competitor is AvAtar on balanced transport with label queries. On its own alignment suite it reports gains of up to 2.8% MRR and 2.6% Recall@1 over nine active-learning baselines. The file gives no absolute accuracy figures for it on CIFAR-10-C or ImageNet-C, so we will produce them ourselves.
Three offline robust baselines use no queries at all: P$^2$OT with its sigmoid schedule, SROT with hard and soft reweighting, and the $\mathsf{W}_p^\varepsilon$ estimator given the true $\varepsilon$ as an oracle. The last is an upper bound on what calibration alone could reach.
Two ablations isolate the key ingredients. The first keeps the unbalanced solver and adjoint scoring but asks label queries only. It isolates whether querying mass rather than labels is what helps. The second keeps mass queries but scores them by the closed-form slack instead of the adjoint gradient. It isolates whether the gradient earns its cost.
For initial seed selection, every method uses the same vote-k selection. Vote-k is a graph-based choice of representative examples, and it reports a 12.9% relative gain over random selection with 18 labels. Using it for every method keeps seeding from confounding the query comparison.
Why not just…?
Why not just filter outliers with an OOD detector first and then run balanced transport or AvAtar? This is the natural two-stage pipeline, and SROT is essentially its strongest form. Blanchet et al. report that decoupled filtering loses to a joint transport estimator even when the trimmed mean is told the true corruption level. SROT's own motivation is that distance- and density-based filters miss close-in outliers. The two-stage pipeline is a baseline here: an OpenMatch-style one-vs-all detector followed by AvAtar. If it matches us on the second-type-outlier splits, the human mass query adds nothing beyond a good detector, and we would report that.
Why not just tune $\lambda$ on a small labeled validation set? Under label scarcity a validation set is labels taken from the budget. We charge it against the same annotator-seconds and compare. This is a real threat when contamination is uniform. We expect our approach to win when contamination is concentrated near particular classes, because a global validation set samples those regions rarely while the mass gradient targets them.
Why not just query the points with the largest marginal slack? This is the most serious alternative, because it is free once the plan is solved. It is the second ablation above. Our prediction is that slack and gradient agree on far outliers and diverge on close-in ones, where slack is near zero. If the slack score matches the adjoint score across all contamination types, we drop the gradient. The contribution then narrows to mass-type queries plus penalty calibration, which is still untested in prior work.
Why not replace the human with an AI judge anchored by neighbors, as in SAGE? SAGE improves error-detection AUROC by 10 to 21 points on CIFAR-10. Its authors show the gain collapses when neighbors share the annotator's systematic error, which is exactly the situation that close-in outliers create. We run SAGE as a substitute oracle on the same queries. We expect it to track the human on far outliers and fail on the second-type splits, and the gap between those two cases measures how much human judgment is needed.
Proposed Experiments
The main testbed is open-set semi-supervised classification: CIFAR-10 in the 6/4 split with 300 labels, and TinyImageNet at mismatch ratios 0.3 and 0.6. The unlabeled pool is contaminated in two ways. Far contamination uses unrelated datasets, such as SVHN digits in the CIFAR pool. Close contamination uses CIFAR-100 classes chosen to lie nearest the known-class prototypes, which produces SROT's second-type outliers by construction. A secondary testbed is AvAtar's cross-modal retrieval on CIFAR-10-C and ImageNet-C, with injected unmatched items. This tests whether mass queries help alignment as well as classification.
Metrics are closed-set error, outlier-detection AUROC, and pseudo-label precision on kept points, each plotted against query count and against annotator-seconds. We also track the calibrated $\lambda$'s implied discard fraction against the true contamination rate.
The decisive checks are three. The first is whether mass queries beat label queries at equal annotator time on the close-contamination splits. This is the core claim; losing here falsifies it. The second is whether the adjoint score beats marginal slack specifically on close contamination. This decides whether the gradient machinery is kept. The third is whether the human-calibrated discard fraction converges toward the true $\varepsilon$ within a few hundred answers and approaches the oracle-$\varepsilon$ robust transport baseline. If it does, the loop removes the hyperparameter that every robust transport method in this literature leaves to guesswork.
References
- On making optimal transport robust to all outliers
- AvAtar: Learning to Align via Active Optimal Transport
- P$^2$OT: Progressive Partial Optimal Transport for Deep Imbalanced Clustering
- Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport
- Outlier-Robust Optimal Transport: Duality, Structure, and Statistical Analysis
- Automatic Outlier Rectification via Optimal Transport
- Optimal and Efficient Binary Questioning for Human-in-the-Loop Annotation
- Enhancing Single-Frame Supervision for Better Temporal Action Localization
- OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization
- SCOMatch: Alleviating Overtrusting in Open-Set Semi-supervised Learning
- Selective Annotation Makes Language Models Better Few-Shot Learners
- Detecting Errors in AI-Generated Annotations: When and Why Semantic Neighbors Help