Training Periodic Feature Embeddings From Pairs When Only a Few Dozen Labels Exist

Tabular deep learning has a clear lesson: the way each raw number is turned into a vector often matters more than how deep the network is on top. On Embeddings for Numerical Features in Tabular Deep Learning showed this with periodic embeddings. These map each scalar through learned sine and cosine functions and then a small linear layer, and they lift a plain multi-layer perceptron (MLP) from an average rank of 8.5 to 3.0 across 11 datasets, often matching gradient-boosted trees. That evidence comes from datasets with tens of thousands of rows. Clinical cohorts, rare-disease registries and pilot screening studies often have fewer than a hundred labelled patients. In that regime a per-feature bank of learned frequencies looks like an overfitting machine, and nobody in this literature has measured what it actually does.

We propose to keep the periodic representation, which is the part that carries the gain, and change two things. First, the encoder is trained with a Siamese objective (one network applied with identical weights to two inputs, trained so that same-label pairs end up close and different-label pairs far apart) over every pair in the tiny training set. Second, a single frequency bank is shared across all features instead of one bank per feature. A nearest-neighbour or linear rule then classifies in the learned space. The claim to test is specific. At $N \le 64$, the shared pairwise-trained embedding beats the same embedding trained with cross-entropy and beats a fixed, unlearned random-frequency embedding. The margin is largest when the negative class is a mixture of different conditions, which is the situation that defeats a single linear boundary.

Core Claim

Periodic embeddings fail at small $N$ for two separable reasons, and the proposal targets each with one ingredient. The first reason is parameter count. Per-feature frequencies and projections scale with the number of columns, a cost On Embeddings for Numerical Features in Tabular Deep Learning itself flags. Tying weights across features applies the Siamese weight-sharing principle along columns rather than samples. The second reason is the mismatch between the training objective and the decision rule. With a few dozen points, a cross-entropy head learns a boundary that the next draw of patients will not share. A pairwise objective shapes distances directly, and distances are what a nearest-neighbour classifier uses. Pairs carry no more label information than the $N$ labels they come from, so we do not claim "more data from $N^2$ pairs." We claim a better-matched inductive bias, and it is the kind of claim our ablations can refute.

Concretely, feature $j$ with value $x_j$, standardised by a per-feature scale $s_j$, is embedded as

$$e_j(x) = \mathrm{ReLU}\big(W\,[\sin(2\pi c\, x_j/s_j),\ \cos(2\pi c\, x_j/s_j)] + b_j\big),$$

where $c \in \mathbb{R}^k$ is one learned frequency vector shared by all features, $W$ is one shared $d \times 2k$ projection, and $b_j$ is a per-feature offset that lets columns stay distinguishable. The encoder $\phi$ concatenates the $e_j$ and passes them through one small layer. It is trained on all pairs $(i,i')$ of the training set with a contrastive loss

$$L = \sum_{i<i'} y_{ii'}\,\|\phi(x_i)-\phi(x_{i'})\|^2 + (1-y_{ii'})\,\max\big(0,\ m-\|\phi(x_i)-\phi(x_{i'})\|\big)^2,$$

where $y_{ii'}=1$ for same-label pairs and $m$ is a margin. With $N \le 128$ there are at most about 8,000 pairs, so we use every pair. This sidesteps the online hard-pair mining that Self-Alignment Pretraining for Biomedical Entity Representations found essential at the scale of millions of synonyms, and it also avoids the false-negative instability that mining causes on tiny sets. Because negatives are present, the encoder cannot collapse to a constant. That removes the need for the frozen-anchor trick of FewSOME. That trick, which pulls normal samples toward a fixed anchor, relies on ImageNet features to make the anchor meaningful, and we have no such backbone for tabular data.

Relation to Prior Work

Three papers motivate the gap. On Embeddings for Numerical Features in Tabular Deep Learning supplies the periodic encoding but tests it only at large $N$ with cross-entropy or squared-error losses. Unveiling COVID-19 from Chest X-ray with deep learning shows that small clinical cohorts punish capacity: ResNet-50 overfits where ResNet-18 does not, and models latch onto dataset artefacts. Contrastive Learning of Cough Descriptors for Automatic COVID-19 Preliminary Diagnosis shows that a triplet loss helps when the negative class is a mix of healthy people, asthma and bronchitis. It does this over 4,865 fixed hand-built acoustic features and explicitly asks how learned representations could replace them. Our proposal answers that question with learned periodic encodings and tests it far below their cohort size.

Two papers constrain the design. Unsupervised Representation Learning by Predicting Random Distances trains a weight-tied twin network on tabular data to predict distances under a fixed random projection. It needs thousands of rows to average out projection noise, so we use real label pairs rather than random targets. FewSOME reaches 84.7% AUC on MVTec AD with $N=30$ normal images, but only with pretrained visual features and only for one-class detection. Our setting is supervised and multi-cluster, with no pretrained backbone. The broader representation-centric results come from CTR prediction (CL4CTR) and speech (Aronowitz et al.). In both, a better embedding lets a factorization machine or a linear head beat complex models. They support our choice of a trivial head, but they assume plenty of data to train the embedding.

Research Program

We will build the shared-frequency Siamese encoder and a per-feature variant identical to MLP-PLR's embedding. Each will be trained with either the pairwise loss or cross-entropy, with frequencies either learned or frozen at random initial values, and read out by either $k$-nearest-neighbour or logistic regression. This factorial design is the core artefact. Each method's contribution can be attributed to one switch, which is the attribution discipline that CombinationTS argues forecasting research lacks.

We will also measure two diagnostics. The first is the per-feature parameter count of each variant. The second is how many distinct clusters the negative class forms in the learned space, estimated by fitting mixtures to the negative embeddings. This tests whether the pairwise objective preserves negative sub-structure that cross-entropy flattens.

Baselines

The naive reference is $k$-nearest-neighbour and logistic regression on standardised raw features. If neither the learned embedding nor the pairwise loss beats these, the representation is not the bottleneck at this scale.

The strongest existing tabular competitor is CatBoost, the model that On Embeddings for Numerical Features in Tabular Deep Learning treats as the bar MLP-PLR must reach. Gradient-boosted trees are also the practitioner's default on small tables. The strongest existing deep competitor is MLP-PLR itself, trained with cross-entropy, weight decay and early stopping. We will find no reported result on Lacuna for either CatBoost or MLP-PLR at $N<100$; the paper reports only average ranks at full scale.

On acoustic data, the reference numbers come from the cough task. The DiCOVA baseline MLP on 39-dimensional MFCCs reaches 69.85% test AUC on Track-1 (DiCOVA Challenge). The triplet network of Bhosale et al. reaches 72.87% validation AUC with roughly a thousand subjects. Both figures are full-cohort. We will rerun both methods on our subsampled training sets rather than compare against the published numbers.

Fairness rules apply to every method. All methods see the same training draws and are tuned by nested cross-validation inside each draw, with the same number of hyperparameter configurations (we fix 30). They also get the same CPU budget per draw. "Beating" a baseline means a higher mean test AUC across 50 draws at a given $N$, with a paired test over draws significant at the 5% level. Ties on average with lower variance across draws will be reported separately, since stability is itself valuable at small $N$.

Three ablations each isolate one ingredient. Swapping the pairwise loss for cross-entropy on the identical shared encoder isolates the objective. Swapping shared for per-feature frequencies isolates the parameter tying. Freezing the frequencies at random values isolates whether learning the encoding matters at all.

Why not just…?

Why not just use CatBoost? This is a real threat. Trees have very few effective parameters on small tables and need no learned encoding. The material contains no small-$N$ comparison either way, so we cannot argue past it. The factorial experiment settles it directly. If CatBoost matches or beats our method at every $N$ on the tabular datasets, we will report that and narrow the claim to the mixed-negative acoustic setting. There, a distance-based classifier has a structural reason to help, because one boundary cannot wrap several separate negative clusters.

Why not just use fixed random Fourier features with logistic regression? This is the most dangerous alternative. It provides the expressiveness of a periodic encoding with no learned parameters, which may be exactly what $N<100$ calls for. It is our frozen-frequency ablation. If it matches the learned encoder, the honest conclusion is that at small $N$ the periodic form helps and learning it does not. We would then propose that conclusion instead, which is still a useful correction to the full-scale results.

Why not just train MLP-PLR with cross-entropy and heavy regularisation? The pairwise objective could be pure ceremony. The loss-swap ablation answers this on an identical encoder with an identical tuning budget. We expect the pairwise loss to win most clearly where the negative class is heterogeneous, as Bhosale et al. observed for coughs. If the advantage appears equally on single-cluster negatives, or not at all, the mechanism we propose is wrong.

Why not just use supervised contrastive loss instead of a Siamese network? With weight tying and all-pairs training, the two are essentially the same objective, and we do not claim otherwise. The Siamese framing matters here only because it gives symmetry of the learned similarity, a property Oreo motivates, and because it suggests extending weight tying to the feature axis.

Proposed Experiments

The tabular benchmarks are the classification datasets of the GBDT-friendly suite in On Embeddings for Numerical Features in Tabular Deep Learning, specifically Adult and Covertype, together with the small, high-dimensional Lung dataset from the anomaly suite of Unsupervised Representation Learning by Predicting Random Distances, used as a supervised task. The acoustic benchmark is DiCOVA Track-1. We use its 39-dimensional MFCC statistics and, separately, a learned-embedding version of the multi-domain descriptor setting. Positives are COVID cases, and the negative class keeps its natural mix of conditions.

For each dataset we draw class-balanced training sets of $N \in \{16, 32, 64, 128\}$, with 50 random draws per size, and evaluate on a fixed large held-out test split. The main metric is AUC. On DiCOVA we also report sensitivity at 80% specificity, to match the published operating point.

Three results would be decisive. The first is the crossover size $N^*$ at which cross-entropy MLP-PLR catches up with the pairwise shared encoder. Our hypothesis predicts a finite crossover, with the pairwise encoder ahead below it. The second is the frozen-versus-learned frequency gap at $N \le 32$. If it is zero, the learned-encoding claim fails. The third is the correlation, across datasets, between the pairwise advantage and the number of negative sub-clusters. A positive correlation supports the mechanism; none refutes it.

To guard against the shortcut learning documented by Tartaglione et al., DiCOVA draws will be stratified by recording metadata where it is available, and we will report performance on test subjects from recording conditions unseen during training.