Robust tactile object recognition in open-set scenarios using Gaussian prototype learning

Wendong Zheng, Huaping Liu, Di Guo, Fuchun SunFrontiers in Neuroscience 2022 · DOI

TL;DR

This paper introduces a Gaussian Prototype Learning (GPL) framework designed to help robots recognize tactile objects in "open-set" environments. Unlike traditional models that force every input into a known category, this method uses probabilistic representations and adaptive thresholds to accurately classify known objects while flagging unknown ones as "novel," preventing potentially catastrophic robotic errors.

What Problem Does This Paper Address?

Most robotic tactile recognition systems operate under a "closed-world" assumption: they assume the robot will only ever encounter the specific objects it was trained on. In the real world, this is rarely true. When a closed-set model encounters an unfamiliar object, it doesn't say "I don't know"; instead, it assigns the object to a known class with high confidence. For a robot, this could mean applying the wrong grip strength to a fragile unknown object, leading to failure or damage.

While "Open-Set Recognition" (OSR) is a hot topic in computer vision, tactile data presents unique challenges. Tactile signals are often high-dimensional, dynamic time-series data with significant "misalignment" between measurements. Existing visual OSR methods often fail to account for the specific distribution characteristics of tactile data, leading to poor performance when detecting novelty.

Methods

The authors propose a unified end-to-end deep framework consisting of two main parts: a Feature Extractor (CNN-based) and a Gaussian Prototype Module.

The core innovation lies in how the model learns and categorizes features:

  1. Gaussian Prototypes: Instead of a standard Softmax layer, the model represents each known class as a "prototype" (a centroid) in feature space. These features are constrained to follow a Gaussian distribution.
  2. Triple-Loss Optimization: The model is trained using three distinct objectives:
  3. Discriminative Classification Loss: Ensures samples are close to their correct class prototype and far from others.
  4. Feature Distribution Loss (Likelihood Regularization): Explicitly forces the features of a class to be compact and follow the assumed Gaussian distribution. This "reserves" space in the feature map for unknown objects.
  5. Adaptive Threshold Learning: Instead of using a single global threshold to reject unknowns, the model learns class-specific rejection criteria based on the maximal distance of training samples from their prototypes.

During inference, if a new tactile sample falls into a "low probability region" far from all known Gaussian prototypes, the system flags it as Unknown.

Datasets

The researchers evaluated their method on two major tactile benchmarks, reorganized to simulate open-set conditions (where some categories are withheld during training):

  • LTM_108 (Haptic Texture Database): Contains 108 material instances across 9 categories (e.g., mesh, wood, fibers). The authors used tactile acceleration traces.
  • PHAC-2 (Penn Haptic Adjective Corpus): Features 60 objects across 8 categories (e.g., foam, plastic, metal). This dataset is particularly challenging because the test and training sets do not share the same object instances, even within the same category.

Key Insights and Ideas

  • Compactness is Key: The "aha moment" here is that for open-set recognition, it’s not enough to be separable; classes must be compact. By forcing known classes into tight Gaussian clusters, the model leaves "empty" territory in the feature space where unknown objects can be identified.
  • The "Human" Analogy: The authors liken their prototypes to abstract memories in the human brain. We recognize things by matching them against these mental templates; if a sensation doesn't match any template well, we perceive it as something new.
  • Adaptive vs. Global Thresholds: One-size-fits-all thresholds for "novelty" usually fail because some categories have naturally higher variance than others. Learning a specific "radius" for each class significantly reduces false positives.

Key Results

The GPL method consistently outperformed state-of-the-art OSR methods like OpenMax and $\tau$-Center across all metrics:

  • Accuracy & F1-Score: On the LTM_108 dataset, the proposed method achieved 70.76% accuracy, significantly higher than the standard Softmax (59.1%) and OpenMax (62.5%).
  • Robustness to "Openness": As the number of unknown classes increased (making the "openness" of the problem harder), the GPL method's performance remained relatively stable, whereas other methods saw sharp declines in their Macro-F1 scores.
  • Novelty Detection: The model achieved an AUC (Area Under Curve) of 0.986 on both datasets, indicating near-perfect ability to distinguish between known and unknown categories.

Limitations and Future Work

While the model excels at detecting unknowns, it doesn't yet incorporate them. The authors note that this research is a "first step" toward lifelong tactile learning.

Future work will likely focus on:

  • Active Learning: How the robot can autonomously ask for labels when it detects an unknown object.
  • Incremental Learning: Updating the model to include new classes on the fly without forgetting the old ones (avoiding "catastrophic forgetting").
  • Generalization: Testing the model in even more unstructured, real-world environments beyond controlled datasets.

Summary updated

Abstract

Tactile object recognition is crucial for effective grasping and manipulation. Recently, it has started to attract increasing attention in robotic applications. While there are many works on tactile object recognition and they also achieved promising performances in some applications, most of them are usually limited to closed world scenarios, where the object instances to be recognition in deployment are known and the same as that of during training. Since robots usually operate in realistic open-set scenarios, they inevitably encounter unknown objects. If automation systems falsely recognize unknown objects as one of the known classes based on the pre-trained model, it can lead to potentially catastrophic consequences. It motivates us to break the closed world assumption and to study tactile object recognition in realistic open-set conditions. Although several open-set recognition methods have been proposed, they focused on visual tasks and may not be suitable for tactile recognition. It is mainly due to that these methods do not take into account the special characteristic of tactile data in their models. To this end, we develop a novel Gaussian Prototype Learning method for robust tactile object recognition. Particularly, the proposed method converts feature distributions to probabilistic representations, and exploit uncertainty for tactile recognition in open-set scenarios. Experiments on the two tactile recognition benchmarks demonstrate the effectiveness of the proposed method on open-set tasks.
BibTeX
@inproceedings{zheng2022robust,
  author = {Wendong Zheng and Huaping Liu and Di Guo and Fuchun Sun},
  title = {{Robust tactile object recognition in open-set scenarios using Gaussian prototype learning}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.1070645},
}
Figure