Anchoring Pairwise Classifiers in the Target Dataset

Most classifiers learn a direct mapping from an input to a label. When the model moves to a new hospital, sensor, or camera, every input shifts slightly and that mapping breaks. A different kind of model learns only whether two inputs belong to the same class. Pairwise difference learning is the tabular version of this idea: it trains a binary "same class or not?" classifier on all pairs of training examples. To classify a new point, it compares that point against a set of labeled reference points, called anchors, and combines the answers (Uncertainty Quantification in Pairwise Difference Learning for Classification). Two things make this relevant to transfer across datasets. The number of training signals grows from $N$ to about $N^2$. And the anchors are an inference-time choice, so nothing forces them to come from the training dataset.

This proposal claims that a pairwise model makes a better cross-dataset classifier when two conditions hold. First, it must see only the difference between two inputs, not the inputs themselves. Second, it must be trained on pairs drawn from the same source dataset and then queried against a small set of labeled anchors from the target dataset. A dataset-wide offset, such as a sensor bias or a staining cast, adds equally to both members of a same-dataset pair, so it cancels out of their difference. What remains is how examples relate within one dataset, and that relation should carry across datasets. The idea is easy to state and cheap to test. The research file also shows it has not been tested: pairwise difference learning won on 75 of 99 OpenML datasets, but only on train/test splits taken from the same dataset.

Core Claim

Let $x$ be a feature vector and $D$ a dataset. Suppose the dataset shift is locally additive, so a target example is $x + b_D$ for some dataset-specific offset $b_D$. Then the pairwise difference is unchanged for any pair taken from one dataset:

$$|(x_i + b_D) - (x_j + b_D)| = |x_i - x_j|.$$

The published pairwise method feeds the classifier $[x_i, x_j, |x_i - x_j|]$ (Uncertainty Quantification in Pairwise Difference Learning for Classification). The two raw copies $x_i$ and $x_j$ do not cancel the offset. They let the pair classifier learn source-specific absolute values, which is the same shortcut that sinks ordinary classifiers.

Our variant makes three changes. It keeps only difference-type features: $|x_i - x_j|$ and $x_i - x_j$, symmetrized. It forms training pairs only within each source dataset. It classifies a target query $q$ against a small labeled target anchor set $A_T$:

$$p(y = c \mid q) \propto \prod_{a \in A_T,\, y_a = c} \gamma(q, a).$$

Here $\gamma$ is the learned same-class probability, and the product is the Bayesian anchor aggregation of the original method.

Real shifts are rarely purely additive, especially in learned embeddings. The claim is therefore empirical: removing absolute-value access and anchoring in the target should close a measurable part of the cross-dataset gap that neither ingredient closes alone.

There is a second, subordinate claim about uncertainty. The original method splits uncertainty into two parts by how the anchor votes disagree: noise inherent in the data, and uncertainty from the model's lack of knowledge. We test whether the knowledge-type component, computed with source anchors on unlabeled target data, predicts how far accuracy will fall on a new dataset. If it does, practitioners would get a label-free warning of the kind of performance drop seen in histopathology, where F1 fell from 0.788 on ICPR 2012 to 0.482 on ICPR 2014 (Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks).

Relation to Prior Work

Three groups of papers play different roles here.

The first group supplies the method. The pairwise difference learning papers provide the pair classifier, the symmetrization, and the anchor aggregation. We reuse all three and change only the feature map, the pairing rule, and where anchors come from. Pair and triplet expansion in small-data regimes supports the premise that relational training pays off when $N$ is small. Deep Ordinal Regression Based on Data Relationship for Small Datasets expands $n$ samples into $O(n^3)$ triplets. Dissimilarity Representation for Handwritten Signature Verification trains on signature pairs. Neither was evaluated across datasets.

The second group constrains the design. FIXED: Frustratingly Easy Domain Generalization with Mixup shows that mixing raw inputs blends domain style with class content and creates ambiguous points near class boundaries. It fixes this with feature-level mixing plus a margin loss. We take that as evidence against convex mixing and toward non-convex difference features, but FIXED remains our main competitor. Mix and Reason argues that full separation of content from style is unattainable and that the relations between classes are what should stay stable across domains. Our method makes that point at the level of individual instance pairs rather than class prototypes. Enhancing Compositional Generalization via Compositional Feature Alignment needs discrete domain labels and trains on single samples. We need only the dataset identity of each source example, which cross-dataset settings always provide.

The third group motivates the problem. Mix3D shows that combining pairs of scenes weakens reliance on context, yet it evaluates only within each dataset. On Improving Cross-dataset Generalization of Deepfake Detectors shows that static augmentation leaves standard detectors below 0.60 AUC on Celeb-DF when trained on FaceForensics++.

Research Program

The method sits on top of fixed features. For wearable sensors, these are standard windowed statistical features. For images, they are frozen embeddings from a pretrained encoder. Any tabular base learner can then serve as the pair classifier, with ExtraTrees as the default, matching the strongest configuration reported for pairwise difference learning.

We build four variants that cross two factors. The first factor is the feature map: the full published $[x_i, x_j, |x_i - x_j|]$ or difference-only. The second is the pairing rule: pairs pooled across source datasets, or pairs formed only within each source dataset. Each variant is queried with either source anchors or $k$ labeled target anchors per class, with $k \in \{1, 5, 20\}$.

The analysis has a mechanism part and a practical part. On the mechanism side, we inject controlled shifts into OpenML tabular datasets, splitting each into pseudo-datasets that differ by an additive offset, a per-feature scale, a nonlinear warp, and a change in class proportions. This locates where the cancellation argument holds and where it breaks. On the practical side, we run leave-one-dataset-out evaluation on real multi-source collections.

Inference cost grows with the number of anchors. Keeping target anchors to a few per class bounds this cost. We report wall-clock time alongside accuracy.

Baselines

The naive reference point is empirical risk minimization, meaning a standard classifier trained on the pooled source datasets. It uses the same base learner (ExtraTrees on the same features), is evaluated on the held-out dataset with no target labels, and receives the same hyperparameter search budget as our method.

The strongest existing domain generalization method in the file for these modalities is FIXED (FIXED: Frustratingly Easy Domain Generalization with Mixup). It reports beating its best baseline by 6.5% average accuracy on the HAR datasets. The file gives that gain but no absolute accuracy. It also gives no reported result found on Lacuna for FIXED under leave-one-dataset-out transfer between the HAR datasets. We will therefore re-run FIXED ourselves on identical features and splits with an equal tuning budget.

On PACS, the reference is MiRe at 84.8% average accuracy (Mix and Reason). Our frozen-embedding pipeline differs from MiRe's end-to-end training, so we report that number for context. The fair comparison runs FIXED-style feature mixing on the same frozen embeddings.

The published pairwise method with source anchors is the within-family baseline. Its reported score is 0.8113 average macro F1 on the 99 OpenML datasets, on same-dataset splits (Uncertainty Quantification in Pairwise Difference Learning for Classification). We expect that score to degrade under shift, and we will measure by how much.

Target-anchored methods get extra target labels, so every one of them is compared against baselines that receive the same $k$ labels per class. There are three such baselines. The first fine-tunes or refits the pooled classifier with the target labels added and upweighted. The second is nearest-neighbor or class-prototype classification using those labels. The third is per-dataset feature standardization followed by pooled training.

The ablations isolate the key ingredient. Full features versus difference-only features tests whether removing absolute-value access matters. Pooled versus within-dataset pairing tests whether the pairing rule matters. Source versus target anchors tests whether anchor location matters.

Beating the field means one of two things, measured by macro F1 averaged over held-out datasets. The difference-only, within-dataset, target-anchored variant must either exceed the best equal-label baseline, or match it while also winning in the class-proportion-shift condition.

Why not just…?

Why not just standardize each dataset's features separately? This is the most serious threat, because subtracting each dataset's mean also removes an additive offset. It fails when the target's class proportions differ from the sources'. In that case the target mean moves for a legitimate reason, and centering on it removes real signal. Differences within a dataset do not depend on class proportions. The synthetic experiment that crosses offset shift with class-proportion shift settles the question directly. If standardization matches our method even under class-proportion shift, we will report that the pairwise machinery adds nothing beyond normalization, and we will narrow the claim to the uncertainty signal.

Why not just use the same few target labels to fine-tune or run nearest neighbors? Any gain from target anchors could simply come from having target labels. That is why every target-anchored comparison gives the baselines exactly the same $k$ labels. The pairwise model should gain an advantage at very small $k$, because it was trained on about $N^2$ relations while a refit classifier sees only $k$ extra points. If that advantage disappears by $k = 5$, the method's niche is extreme scarcity, and we will say so.

Why not just use Mixup or FIXED? FIXED interpolates features to encourage invariance but still learns an absolute mapping from features to labels, which a residual dataset offset can move. The paper's own reported gains come from multi-source protocols, not from strict transfer to a new dataset. Our re-run under identical splits decides the comparison. We also test combining FIXED features with difference-only pairing, in case the two approaches are complementary.

Why not just rely on pretrained foundation models? Frozen CLIP and DINOv2 features still fail on unseen domain-class combinations (Enhancing Compositional Generalization via Compositional Feature Alignment). In our pipeline these encoders are the input features, not a competitor. The relevant question is whether pairing on top of them beats a linear probe on the same embeddings, and we include that probe.

Proposed Experiments

The first experiment is the controlled OpenML study. We split a subset of the 99 datasets into pseudo-datasets under the four shift types and measure macro F1 for all eight method-by-anchor combinations. The decisive check has three parts. Difference-only features should remove the drop from additive shift exactly, and partly remove the drop from scale and warp shifts. Full features should not remove it. Standardization should fail only when class proportions shift.

The second experiment uses the four wearable activity datasets (DSADS, PAMAP2, UCI-HAR, USC-HAD). We restrict to shared activity classes and hold out one dataset at a time. This is where sensor-placement offsets make the additive story most plausible. The check is whether the target-anchored difference model beats re-run FIXED and the equal-label baselines.

The third experiment uses PACS and Office-Home with frozen embeddings and a leave-one-domain-out protocol. This tests whether the effect survives in embedding spaces where shift is clearly not additive. A null result here would be informative, not embarrassing: it would mark the method's boundary.

For the uncertainty claim, we compute the knowledge-type uncertainty with source anchors on unlabeled target data. We then test, across all held-out datasets and synthetic shifts, whether its rank correlation with the realized accuracy drop is higher than that of max-softmax confidence from the pooled baseline.