Speech Recognition Using RFID Tattoos (Extended Abstract)

Jingxian Wang, Chengfeng Pan, Haojian Jin, Vaibhav Singh, Yash Jain, Jason I. Hong, Carmel Majidi, Swarun KumarIJCAI 2021 · OpenReview

Abstract

This paper presents a radio-frequency (RF) based assistive technology for voice impairments (i.e., dysphonia), which occurs in an estimated 1% of the global population. We specifically focus on acquired voice disorders where users continue to be able to make facial and lip gestures associated with speech. Despite the rich literature on assistive technologies in this space, there remains a gap for a solution that neither requires external infrastructure in the environment, battery-powered sensors on skin or body-worn manual input devices. We present RFTattoo, which to our knowledge is the first wireless speech recognition system for voice impairments using batteryless and flexible RFID tattoos. We design specialized wafer-thin tattoos attached around the user's face and easily hidden by makeup. We build models that process signal variations from these tattoos to a portable RFID reader to recognize various facial gestures corresponding to distinct classes of sounds. We then develop natural language processing models that infer meaningful words and sentences based on the observed series of gestures. A detailed user study with 10 users reveals 86% accuracy in reconstructing the top-100 words in the English language, even without the users making any sounds.

Introduction

Voice impairments, or dysphonia, significantly impact the quality of life for approximately 1% of the global population. While many individuals with these conditions retain the ability to perform the facial and lip gestures associated with speech, current assistive technologies often fall short. Camera-based systems require constant line-of-sight and fail during movement; audio-based systems are ineffective in noisy environments or for those unable to produce sound; and wearable sensors often involve bulky batteries or invasive components that are socially stigmatizing.

RFTattoo addresses these challenges by providing a batteryless, skin-friendly speech recognition system. By utilizing flexible RFID "tattoos" to track facial gestures, the system measures minute frequency shifts caused by skin stretching and tongue movement to achieve high-accuracy word recognition without requiring audible sound.

Methodology

The core of RFTattoo is a novel application of Radio Frequency Identification (RFID) technology that treats the antenna itself as a primary sensor. The system follows a multi-stage pipeline to translate physical facial movements into synthesized speech.

System architecture of RFTattoo, illustrating the pipeline from RF signal measurement to speech synthesis.

Hardware and Sensing

The researchers fabricated stretchable RFID antennas using silver-polydimethylsiloxane (Ag-PDMS) conductors. These tattoos are hypoallergenic, flexible, and thin enough to be hidden under makeup. As a user speaks, their facial muscles stretch the tattoo, changing the antenna's electrical length and shifting its resonant frequency. The system detects these shifts with millimeter-level accuracy.

To overcome the narrow bandwidth of the unlicensed 900 MHz band, the authors implemented a Multi-Chip Strategy. Multiple RFID chips are attached to a single antenna, each tuned to different frequencies. This allows the reader to "probe" the stretch value at a single frequency rather than sweeping the entire band, preventing the signal from moving outside the readable range during significant stretches.

Viseme Characterization and NLP

Beyond lip movement, RFTattoo monitors the magnitude and phase of RF signals to detect the subtle impact of tongue position on skin impedance. This capability allows the system to distinguish between visemes—sets of sounds that look identical on the lips (e.g., "p" and "m") but differ in internal articulation.

The raw gesture data is processed through a Natural Language Processing (NLP) framework. Using a Bayesian model and word co-occurrence statistics trained on the Cornell Movie Dialog Corpus, the system disambiguates phoneme sequences to construct meaningful sentences. This contextual correction is vital; while raw gesture recognition for unseen sentences may be low (35.7%), applying linguistic constraints and grammar rules improves accuracy to 53.2%.

Experimental Results

The system was evaluated through a user study with 10 participants, including individuals with temporary dysphonia. The results demonstrate the high precision of the RF-based sensing approach:

  • Stretch Accuracy: The system achieves a median accuracy of 1.4 mm in tracking tattoo deformation.
  • Viseme Recognition: It distinguishes between 11 classes of English visemes with 90% accuracy.
  • Word and Sentence Accuracy: For the top-100 most frequent English words, the system achieves 86% accuracy. For commonly used daily sentences, accuracy reaches 91%.

Discussion and Conclusion

RFTattoo represents a significant step forward in discreet assistive technology. By eliminating the need for separate strain gauges and power sources, the device becomes truly "skin-like." Furthermore, the use of RF signals provides a form of "RF Vision" that can perceive tongue movement through the skin, a data stream that traditional camera-based lip-reading systems lack.

Future work will address current limitations, such as the need for light-weight calibration upon reapplication and potential signal shadowing caused by body movements. Expanding the training datasets will also help the system better handle individual speaking styles, accents, and a broader vocabulary including proper nouns.

Summary updated

Abstract

This paper presents a radio-frequency (RF) based assistive technology for voice impairments (i.e., dysphonia), which occurs in an estimated 1% of the global population. We specifically focus on acquired voice disorders where users continue to be able to make facial and lip gestures associated with speech. Despite the rich literature on assistive technologies in this space, there remains a gap for a solution that neither requires external infrastructure in the environment, battery-powered sensors on skin or body-worn manual input devices. We present RFTattoo, which to our knowledge is the first wireless speech recognition system for voice impairments using batteryless and flexible RFID tattoos. We design specialized wafer-thin tattoos attached around the user's face and easily hidden by makeup. We build models that process signal variations from these tattoos to a portable RFID reader to recognize various facial gestures corresponding to distinct classes of sounds. We then develop natural language processing models that infer meaningful words and sentences based on the observed series of gestures. A detailed user study with 10 users reveals 86% accuracy in reconstructing the top-100 words in the English language, even without the users making any sounds.
BibTeX
@inproceedings{wang2021speech,
  author = {Jingxian Wang and Chengfeng Pan and Haojian Jin and Vaibhav Singh and Yash Jain and Jason I. Hong and Carmel Majidi and Swarun Kumar},
  title = {{Speech Recognition Using RFID Tattoos (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Akpm8y4Mag},
}
Figure