A robot that sorts household objects by touch has to say "I have never felt this material before," not force every touch into a known category. That is the open-set recognition problem: classify the known classes while rejecting genuinely new ones. Touch matters because appearance does not fix physical properties. A painted foam sphere and a steel sphere can look identical, and recovering stiffness or density from pixels alone is ill-posed (GIC). The difficulty is that touch signals from multi-sensor probes are noisy in uneven ways. A contact microphone, a force sensor, a thermal sensor and an inertial unit each have their own noise level. In clutter, a single touch also picks up vibration and contact from neighboring objects, so some touches are much noisier than others. Current open-set methods draw their "known region" with fixed distances learned on clean data. When the noise level changes at deployment, those boundaries become wrong: they either reject everything or accept the novel object.
We propose to borrow a tool from random matrix theory that makes uneven noise look uniform. Rows (sensor features) and columns (individual touches) are rescaled by noise levels estimated directly from the data, so the leftover noise follows a known eigenvalue distribution with a computable upper edge (Dyson equalizer; biwhitening). After this rescaling, a "known physical class" is the set of directions whose eigenvalues rise above that edge. A touch counts as novel when its leftover energy, outside those directions, exceeds what pure noise could produce. The rescaling is re-estimated on each deployment batch, so the rejection threshold adapts to new clutter without retraining. Counting how many new eigenvalues rise above the edge among rejected touches gives an estimate of how many distinct new materials are present.
Core Claim
Once multimodal touch data has been noise-equalized, the outlier eigenvalues give an open-set rule with three properties that learned-prototype methods lack. First, its false-rejection rate is set analytically rather than tuned on held-out unknowns. Second, it re-calibrates without labels when the noise profile shifts, for example when clutter is introduced only at test time. Third, it estimates the number of new physical categories without a density threshold. We expect equalized spectral rejection to roughly match the strongest learned method on clean, single-object tactile benchmarks. Under heteroscedastic, clutter-contaminated probing (noise whose level differs from feature to feature and touch to touch), it should hold its false-positive rate where learned thresholds drift.
The four research directions play unequal roles here. Variance-weighted spectral transformation is the mechanism. Open-set recognition is the task. Visual-physical ambiguity is the reason the task must be solved by touch rather than by a vision detector. Multi-object clutter is the source of the touch-to-touch noise differences that break existing methods, and it is the setting for the hardest test.
Positioning Against Prior Work
The closest work is Gaussian prototype learning (GPL). GPL trains a convolutional encoder so that each known tactile class forms a tight Gaussian cluster with a learned per-class rejection radius. It reports 70.76% open-set accuracy on LTM_108, against 59.1% for softmax thresholding and 62.5% for OpenMax, and a novelty AUC of 0.986 on both LTM_108 and PHAC-2. GPL motivates us, because it shows that bounded class regions are what make tactile open-set recognition work. It also constrains us: its AUC is near ceiling on clean benchmarks, so we do not claim to beat it there. Its radii are fixed after training and assume identically distributed noise across channels and touches. We replace the learned radius with a threshold derived from the noise-edge, which is re-estimated per batch. We also add clustering of the unknowns, which GPL explicitly leaves open.
CLAMP supplies the data regime. It is a crowdsourced dataset with five haptic modalities and 5,357 objects, fused with a two-layer MLP for closed-set material (0.87 accuracy) and compliance (0.88 accuracy) recognition. CLAMP shows that touch rescues tasks where vision fails: metallic retrieval from a cluttered bag succeeded in 6/13 trials versus 0/13 for vision only. But CLAMP is closed-set, and its fusion ignores per-modality noise. Vision-side open-world methods, namely UC-OWOD (pseudo-labeled unknown proposals with Student-t clustering) and the multi-object novelty detector of Salehi et al. (dense ViT feature distillation, 94.5% AUROC on multi-object COCO), handle clutter well. However, they define novelty by appearance, so by construction they cannot flag an object that looks familiar but is physically new. We use a vision method only to localize candidate objects to probe. For that we use MOST, which discovers multiple objects with DINO tokens and DBSCAN and reaches 74.8% CorLoc on VOC07. Vision is never the novelty judge.
The method comes from the heteroscedastic spectral literature. Biwhitening and the Dyson equalizer estimate row and column noise scalings so that the noise spectrum matches the Marchenko–Pastur law, the standard eigenvalue distribution of pure noise. Landa & Cheng show that the analogous doubly stochastic scaling protects affinity graphs from local noise. Several results act as negative constraints on simpler designs: inverse-variance weighting is suboptimal for low-rank recovery (Hong et al.; Optimally Weighted PCA); blind whitening amplifies noise in weak channels (Doi et al.); and exact detection thresholds in block-structured noise need a diagonal correction (Pak et al.; Mergny et al.). Those threshold results cover rank-one spikes with block-constant noise. Our use, with many classes and estimated noise profiles, lies outside their guarantees, so we treat whether the edge actually holds as an empirical question.
Research Program
Each touch is turned into a feature vector by concatenating per-modality features. We use two versions: handcrafted spectral and time-statistics features, and embeddings from frozen per-modality encoders of CLAMP's InceptionTime type, trained closed-set on known classes. Known-class touches form a matrix $X \in \mathbb{R}^{m \times n}$, with $m$ features as rows and $n$ touches as columns. We estimate positive scalings $D_r$ (one per feature) and $D_c$ (one per touch) with the Dyson equalizer, so that the noise in $\tilde X = D_r X D_c$ has roughly unit average variance per row and column. We then keep the $r$ singular directions $U_r$ whose squared singular values, divided by $n$, exceed the noise edge:
$$\lambda_+ = (1 + \sqrt{m/n})^2 .$$
Here $\lambda_+$ is the upper edge of the Marchenko–Pastur law for an $m \times n$ unit-variance noise matrix. We also keep per-class subspaces for classification.
For a test touch $y$, its column scaling $c_y$ is estimated in one of two ways: from its energy in feature bands that the known subspaces leave unused, or by re-running equalization on the test batch pooled with a reference set. The novelty score is the leftover energy
$$s(y) = \lVert (I - U_r U_r^\top) D_r\, c_y\, y \rVert^2 ,$$
which under pure unit-variance noise concentrates around $m - r$. This gives a threshold at a chosen false-rejection rate without any unknown samples. Touches that are rejected are pooled and equalized again. The number of their eigenvalues above $\lambda_+$ estimates the number $K$ of new physical categories, and the corresponding eigenvectors initialize the clustering.
Clutter enters through the touch scalings $D_c$. A touch whose probe also contacts neighbors has larger, partly structured noise, and the equalizer down-weights it without discarding it. As an ablation, we also predict a touch's noise level from vision: MOST's object masks give the number of objects inside the probe's contact neighborhood, which serves as a predictor of cross-talk variance. This tests whether scene decomposition can supply the noise profile that the theory says must otherwise be estimated.
The first analysis is diagnostic. We check whether the equalized matrix's bulk of non-signal eigenvalues actually follows the Marchenko–Pastur shape. Learned embeddings are not "low-rank signal plus independent noise," and if the bulk is badly misfit, the analytic threshold is void. In that case we fall back to the raw-feature version and report the failure.
Baselines
The naive reference is softmax confidence thresholding on the concatenated-feature classifier, CLAMP's fusion MLP retrained on the known classes. It shows how far a closed-set model gets for free. GPL reports 59.1% for softmax thresholding on LTM_108; that figure anchors the low end. OpenMax is the classical extreme-value baseline, at 62.5% on LTM_108 per GPL.
The strongest existing competitor is GPL, reported at 70.76% accuracy and 0.986 novelty AUC on LTM_108 and 0.986 AUC on PHAC-2. It is the only method in the file built for tactile open-set recognition, and it encodes exactly the bounded-region idea we compete with. We will reimplement it on the same encoders, splits and hyperparameter search budget as our method. We will also give it the same clutter-contamination augmentation during training, so any advantage we show cannot come from GPL simply never seeing noisy touches. On clean LTM_108 and PHAC-2, matching its reported AUC within noise counts as parity, not a win. On CLAMP open-set splits with shifted noise, beating it means a lower false-positive rate at 95% true-positive rate (FPR95) together with a calibration gap below a few points. The calibration gap is the difference between nominal and realized false-rejection rate, measured with no unknown-class tuning for our method, while GPL is allowed its validation tuning. No reported open-set result was found on Lacuna for CLAMP, so all CLAMP numbers are new.
Mahalanobis distance with shrinkage covariance on the concatenated features is the strong statistical baseline. It is the "just whiten" answer, and it isolates whether row-and-column equalization matters beyond full-covariance whitening. Two ablations isolate the key ingredient by keeping the subspace-residual score while changing only the rescaling: unweighted PCA residual, and per-modality inverse-variance weighting. If our method does not beat these, the equalization is not doing the work.
For counting and clustering novel categories, we compare against DBSCAN as in MOST, Student-t soft clustering as in UC-OWOD, and k-means given the true $K$ as an oracle upper reference. The vision-only reference on look-alike scenes is the Salehi et al. detector, run on the camera view.
Why not just…?
Why not just whiten with the empirical covariance, or divide each channel by its variance? This is the most serious simpler alternative. Doi et al. show that blind whitening inflates weak, noisy channels. Hong et al. prove that inverse-variance weighting is suboptimal in high dimensions. Neither result handles touch-to-touch variance, which is what clutter introduces, and full whitening treats each touch identically. Even so, these are theoretical arguments for a different setting. The shrinkage-Mahalanobis and inverse-variance baselines settle it directly. If either matches us under test-time clutter shift, we will report that the contribution reduces to calibration and the estimate of $K$.
Why not just train GPL with noise augmentation or per-modality dropout? This is also a real threat. Augmentation could teach the encoder to ignore cross-talk. Our counter-argument is narrow. Learned radii are fixed after training, so a noise level outside the augmentation range should miscalibrate them. Our threshold is re-estimated on each batch. The experiment trains GPL with the same contamination we use and tests at contamination levels both inside and beyond that range. If GPL holds its calibration beyond the range, our central claim fails.
Why not just ask a vision-language model, or use a visual open-world detector? On the look-alike cases that motivate this work, appearance does not carry the answer: inverse physics from optics is ill-posed (GIC). Vision-only retrieval in CLAMP succeeded in 0/13 trials. In PACS, the best model reached 70.1% on physical reasoning against 96.3% for humans. We still run the Salehi et al. detector on the look-alike scenes to show the failure empirically rather than assume it.
Why not just use DBSCAN to find the new materials? DBSCAN's radius has to be retuned as density changes, a brittleness already noted for MOST. Counting eigenvalues above the noise edge needs no radius. The comparison of $K$-estimation error across contamination levels decides this.
Proposed Experiments
The first experiment is synthetic. We build spiked matrices with several class subspaces and noise that varies by row and column, including block profiles of the kind studied by Mergny et al., and check that the realized false-rejection rate tracks the nominal rate while $K$ is recovered. We also check where this breaks as the class signal strength approaches the edge.
The second experiment uses LTM_108 and PHAC-2 under GPL's held-out-class protocol, to establish parity with GPL's reported figures.
The third and main experiment uses CLAMP. We build open-set splits by holding out material categories. We then contaminate test touches in two controlled ways: by adding modality-specific noise, and by mixing in contact-microphone and force traces from other touches to simulate neighbor cross-talk, at graded intensities. Training data stays clean, or is augmented for GPL as described above. We report AUROC, FPR95, open-set macro-F1, calibration gap, and, for novel clustering, adjusted Rand index and error in estimated $K$.
The fourth experiment is a small physical study. Visually matched pairs (real and replica fruit, steel and painted-foam spheres, solid and hollow blocks) are placed in clutter of increasing density. MOST localizes the objects and a CLAMP-style probe touches each one. Some physically novel items are deliberately chosen to look like known ones. Here we also compare touch noise levels estimated from the data against those predicted from vision.
The decisive checks are three. Does the equalized bulk actually fit the Marchenko–Pastur edge on real touch features? Does our calibration gap stay small beyond the augmentation range, where GPL's grows? Does either the Mahalanobis or the inverse-variance ablation close that gap? A positive answer to the first two and a negative answer to the third would establish the claim. Any other outcome tells us which simpler method suffices.
References
- GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation
- The Dyson equalizer: adaptive noise stabilization for low-rank signal detection and recovery
- Biwhitening Reveals the Rank of a Count Matrix
- Robust tactile object recognition in open-set scenarios using Gaussian prototype learning
- CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception
- UC-OWOD: Unknown-Classified Open World Object Detection
- Redefining Normal: A Novel Object-Level Approach for Multi-object Novelty Detection
- MOST: Multiple Object localization with Self-supervised Transformers for object discovery
- Robust Inference of Manifold Density and Geometry by Doubly Stochastic Scaling
- Optimally Weighted PCA for High-Dimensional Heteroscedastic Data
- A Theoretical Analysis of Robust Coding over Noisy Overcomplete Channels
- Optimal Algorithms for the Inhomogeneous Spiked Wigner Model
- Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models
- PACS: A Dataset for Physical Audiovisual CommonSense Reasoning