Deception detectors trained on one setting, such as courtroom video, fall to near coin-flip accuracy when moved to another, such as laboratory lie games. A plausible reason is that the same behavior means different things in different settings. Courtroom deceivers increase eye contact while lab subjects look away, and in the game Diplomacy rising politeness signals an upcoming betrayal. Pooling every available source corpus therefore teaches the model contradictory rules.
This note proposes a narrower strategy that can be tested cheaply. Before training, choose a small, class-balanced subset of source examples whose behavior statistically resembles the unlabeled target setting, and train only on that subset. The selection problem has the "diminishing returns" structure that lets a simple greedy algorithm come provably close to the best possible subset. That guarantee makes it possible to separate two questions: whether the optimizer failed, or whether the idea of matching the target distribution failed.
Core Claim
The claim is that example-level source selection helps where training-time losses alone have not. The selection matches the target's distribution of behavior features, measured on unlabeled target data, and is constrained so every source domain and both classes stay represented. On the six-corpus multi-to-single audio-visual deception benchmark, this should beat the best continuous invariance method. It should also beat the cheap alternative of choosing which whole source domains to include.
The core mechanism reuses the prototype objective of Examples are not enough, learn to criticize! Criticism for Interpretability (MMD-critic). That paper selects prototypes minimizing maximum mean discrepancy, a kernel distance between two sample distributions, and shows the objective is submodular under diagonally dominant kernels. We change one ingredient. The reference distribution becomes unlabeled target-domain clips, while candidates are drawn from labeled source pools.
With $T$ the set of unlabeled target feature vectors, $S$ the selected source subset, and $k$ a positive-definite kernel, we maximize
$$J(S) = \frac{2}{|T|\,|S|}\sum_{t\in T}\sum_{s\in S} k(t,s) \;-\; \frac{1}{|S|^2}\sum_{s,s'\in S} k(s,s'),$$
which is the negative squared MMD between $S$ and $T$ up to a constant. The first term rewards source examples that look like the target. The second term penalizes redundancy among selected examples.
We impose a partition matroid on the choice. A partition matroid here means a set of per-cell quotas, where each cell is one (source domain, label) pair. Under these quotas, greedy selection retains the classical constant-factor guarantee for monotone submodular objectives under matroid constraints. The quotas also prevent a failure mode specific to this problem: matching the target by selecting mostly one class, or mostly one lookalike domain.
The guarantee concerns the surrogate objective, not accuracy. The empirical bet is that the surrogate tracks transfer accuracy. We state that bet as a measurable correlation below, so it can be falsified directly.
Where This Sits
The file motivates this direction from two sides. Benchmarking Cross-Domain Audio-Visual Deception Detection introduced multimodal inter-domain gradient matching (MM-IDGM), a training loss that aligns audio and visual gradient directions. MM-IDGM still peaked at 59.02% multi-to-single accuracy. The same paper found that domain-by-domain sequential training beat randomly mixed training. Both findings suggest that which data the model sees matters, which a loss on the full pool cannot express.
UNIDECOR found weak or contradictory lexical cue–label correlations across 14 text corpora. This is direct evidence of cue inversion, and it constrains us. Selection on input features alone cannot repair a relationship that truly flips sign, so our method can only help if the sources contain subpopulations whose cue–label relation resembles the target's.
Three selection methods supply technique but assume away our setting. PartitionSel uses partition-matroid greedy selection across data domains, but scores examples by alignment with a labeled target validation set. RETRIEVE and CRAIG select gradient-matching coresets, the former against a labeled validation set, and aim mainly at training speedups. S3VAADA greedily selects target examples for an annotator to label.
We differ from all four in the same way. We select source examples using only unlabeled target data, and our goal is transfer rather than speed. Diversifying Multiple Generative Agents shows that facility-location coverage beats K-Medoids at representing peripheral behavior. That supports using submodular coverage rather than clustering, although that work concerns prompting language models, not training detectors.
Research Program
The main testbed is the six-dataset audio-visual benchmark from Benchmarking Cross-Domain Audio-Visual Deception Detection. Its datasets are Real-Life Trial, Bag of Lies, MU3D, Box of Lies, DOLOS and MDPE. We follow its multi-to-single protocol, holding out each dataset in turn as the target.
Features are facial action units, gaze and audio descriptors. We choose these over raw pixels because the file identifies background, lighting and microphone artifacts as the dominant spurious signal, and a kernel computed on raw frames would match recording conditions rather than behavior. The kernel is computed on standardized behavior features. Its bandwidth is set by the median heuristic, and never by target labels.
Greedy selection fills the per-cell quotas. The total budget and the quota rule (proportional versus equal per domain) are tuned only by nested leave-one-source-out validation among the sources. The detector trained on the selected subset uses the same Attention-Mixer architecture as the benchmark, with MM-IDGM optionally added. This isolates selection from architecture.
We add a secondary diagnostic from MMD-critic. After selection, the witness function (the per-point difference in kernel density between target and selected source) flags target clips that the coreset covers poorly. We report accuracy separately on covered and flagged target clips. If flagged clips are where errors concentrate, the method also gives a practical abstention signal.
The text-only replication uses UNIDECOR's cross-corpus RoBERTa setting with leave-one-corpus-out targets. Its 13 source corpora make exhaustive search over domain subsets infeasible, which is exactly where greedy selection should matter most.
Baselines
The naive reference is the benchmark's pooled training on all source data. It has two variants: randomly mixed (domain-simultaneous) and domain-by-domain sequential. The latter is included because the benchmark reports it beating mixing. A random subset of the same size, obeying the same quotas, controls for the fact that training on less data sometimes helps small-sample models by itself.
The strongest existing method is Attention-Mixer with MM-IDGM from Benchmarking Cross-Domain Audio-Visual Deception Detection. That paper reports 59.02% average multi-to-single accuracy for it, with Attention-Mixer alone at 58.88%. For a single-pair reference, Flexible-modal Deception Detection with Audio-Visual Adapter reports 58.77% for training on court trials and testing on Bag of Lies. For UNIDECOR there is no reported result found on Lacuna for aggregate cross-corpus F1, so we will rerun its RoBERTa baseline ourselves.
We will rerun MM-IDGM under our splits, features and seeds rather than quote it. Every method gets the same unlabeled target clips wherever it can use them, the same encoder, and an equal hyperparameter search budget selected without target labels. Given datasets of roughly a hundred to a few hundred clips, we report means over at least ten seeds with bootstrap confidence intervals.
Beating the strongest baseline means two things. The first is an average multi-to-single gain whose confidence interval excludes zero; we set roughly three points over our rerun of MM-IDGM as the threshold of practical interest. The second is that the gain must hold on at least four of six held-out targets, not only on one easy pair.
Three ablations isolate the key ingredients. Unconstrained greedy selection (no quotas) tests whether the matroid matters. A version with domain quotas but no label quotas tests class-collapse. Kernel on raw visual embeddings versus behavior features tests the artifact hypothesis. An oracle that selects using target labels bounds how much selection could possibly help.
Why not just…?
Why not just pick the source examples nearest to each target clip? Nearest-neighbor retrieval is the simplest target-aware selector. KITE reports that it returns redundant near-duplicates and is beaten by joint relevance-and-diversity greedy selection, including a +4.55% SST-5 gain over BM25. That evidence comes from in-context exemplars, not detector training. We therefore run kNN selection under identical quotas, and treat a tie as evidence that the redundancy term is unnecessary here.
Why not just search over which whole source domains to include? This is the most serious threat. With five sources there are only 31 non-empty subsets, so exhaustive domain-level search with leave-one-source-out validation is cheap. If behavior truly differs by domain rather than within domains, it could match our method. We run it as a baseline on the video benchmark and run greedy domain-level selection on UNIDECOR. If domain-level search matches example-level selection on video, we will report that the useful unit is the corpus rather than the clip, and the proposal narrows to the 13-source text setting.
Why not just add a domain-alignment loss and train on everything? MM-IDGM is exactly such a loss and stalls at 59.02%. A loss cannot drop examples whose cue–label relation contradicts the target; it can only down-weight their gradients against a pooled average. The combined arm (selection plus MM-IDGM) against MM-IDGM alone settles whether discrete selection adds anything beyond the continuous loss.
Why not just use the sequential training schedule the benchmark already found helpful? Sequential training reorders data but keeps all of it. We run it as a baseline and also apply it to the selected subset, to test whether the two gains compound or overlap.
Decisive Checks
The first check tests the surrogate itself. We draw 200 random quota-satisfying subsets per target. For each, we compute achieved $J(S)$ and target accuracy, then report the rank correlation. If that correlation is near zero, the approximation guarantee is irrelevant and the idea fails regardless of headline numbers.
The second check is headroom. The label-using oracle must beat pooled training by a clear margin. Otherwise the ceiling on 59% reflects missing signal, not poor data choice, and no selector can help.
The third check is the head-to-head against both MM-IDGM and exhaustive domain-level search under the fair protocol above. The fourth is whether witness-flagged target clips carry disproportionate error.
We would count the proposal as succeeding if the correlation is positive, the oracle shows headroom, and target-matched selection closes a meaningful share of that headroom beyond both strong baselines. We would count it as failing, and publish that result, if domain-level search or kNN matches it.
References
- Examples are not enough, learn to criticize! Criticism for Interpretability
- Benchmarking Cross-Domain Audio-Visual Deception Detection
- UNIDECOR: A Unified Deception Corpus for Cross-Corpus Deception Detection
- Minibatch selection for Language Models via Partition Matroid Constrained Gradient Matching
- RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
- Coresets for Data-efficient Training of Machine Learning Models
- S3VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation
- Diversifying Multiple Generative Agents by Aligning with Human Populations
- Flexible-Modal Deception Detection with Audio-Visual Adapter
- KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning