Introduction
In open-world robotics, agents must continuously learn new object categories from a stream of data. This presents two significant challenges: catastrophic forgetting, where new knowledge overwrites previous learning, and data scarcity, particularly in the "few-shot" regime where only a handful of examples are available. While vision is a primary sense, it often fails to distinguish between objects that are visually similar but physically different (e.g., a real fruit vs. a plastic replica). Integrating tactile feedback (touch) provides a stable, viewpoint-invariant anchor, yet multimodal data is difficult to collect at scale.
To address these issues, this paper introduces the Touch-vision Incremental Few-shot (TIFS) framework. TIFS leverages "imaginary vision synthesis" to augment limited visual data and a specialized attention-maintenance mechanism to ensure the model retains its cross-modal reasoning capabilities over time.
The TIFS Framework
The TIFS architecture is designed to handle the modal imbalance and data scarcity inherent in incremental multimodal learning. It consists of four integrated components:
- Imaginary Vision Synthesis (IVS): To mitigate the few-shot problem, IVS generates synthetic visual samples through color dithering, cropping, and rotation. Crucially, touch data remains unaugmented, as physical properties like hardness are independent of visual perspective, allowing touch to serve as a stable semantic anchor.
- Touch-Vision Cross-Modal Contrastive Learning (TVCC): This module aligns visual and tactile features in a shared embedding space. It ensures that corresponding visual and tactile signals for the same object are clustered together while pushing different categories apart.
- Touch-Vision Attention Maintenance (TVAM): This is the core defense against catastrophic forgetting. By using Kullback-Leibler (KL) divergence, TVAM ensures that the spatial and temporal attention maps learned in previous steps remain consistent as the model encounters new tasks.
- Regularized Final Loss: A combination of Separated Softmax Cross-Entropy and L2 regularization prevents the model from developing a bias toward recently learned classes.
Experimental Evaluation
The framework was benchmarked on a combined set of 24 categories derived from the Touch and Go and VisGel datasets. The evaluation followed an incremental protocol consisting of 8 steps, with 3 new categories introduced at each step.
Results and Analysis
TIFS demonstrates superior performance and stability compared to state-of-the-art incremental learning methods such as iCaRL, LwF, and AFC.
- Accuracy and Stability: TIFS achieved an average accuracy of 64.59%, significantly higher than the vision-only baseline (60.76%). More importantly, it recorded the lowest Average Drop Rate (ADR) of 9.31%, indicating high stability in retaining old knowledge.
- Late-Stage Robustness: By the final incremental step, while other models saw their accuracy drop to the 30-40% range, TIFS maintained a robust 46.92% accuracy.
- Ablation Insights: Ablation studies (shown in the rightmost plot of the figure above) confirm that the combination of IVS, TVCC, TVAM, and L2 regularization provides the most consistent performance gains, particularly in later incremental steps.
Conclusion
The TIFS framework effectively addresses the dual challenges of data scarcity and catastrophic forgetting in multimodal robotic learning. By using touch as a stable anchor and preserving cross-modal attention, robots can more reliably learn and retain physical object properties in dynamic, open-world environments. Future work will explore generalizing these methods to diverse tactile sensors and additional modalities like audio.
Summary updated