This thesis from IIIT Delhi, authored by Bikrant Bikram Pratap Maurya, tackles a critical vulnerability in modern computing: the sound of your typing. While Acoustic Side-Channel Attacks (ASCA) aren't new, they have historically been "laboratory curiosities" that fail the moment you change the keyboard or add background noise. This work introduces DECKER, a framework that makes these attacks robust enough to survive the "open world" of unseen hardware and compressed VoIP channels.
TL;DR
The paper demonstrates that acoustic keystroke attacks are a credible real-world threat, not just a lab phenomenon. By using domain-invariant representation learning and Large Language Models (LLMs), the authors' system, DECKER, can recover text from keyboards it has never "heard" before, even over noisy Zoom calls, narrowing the accuracy gap on unseen hardware by over 20 percentage points.
What Problem Does This Paper Address?
Acoustic side-channel attacks rely on the fact that every key on a keyboard produces a slightly different sound. If an attacker can record you typing, they can theoretically map those sounds back to specific keys to steal passwords or private messages.
However, there is a massive "generalization gap." Most existing models learn the keyboard's signature (the resonance of the chassis, the switch type) rather than the key's signature. Consequently, a model trained on a MacBook Pro fails completely when used against a Dell XPS. Furthermore, real-world factors like background noise, different typing styles, and the audio compression used in VoIP (Voice-over-IP) apps like Zoom or Teams typically destroy the subtle acoustic cues these attacks rely on.
Methods
The authors propose a two-stage pipeline called DECKER (Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition).
- Acoustic Level (Domain Invariance): To stop the model from overfitting to a specific device, they use:
- Keyboard Signature Normalization (KSN): A learnable inverse filter that attempts to "deconvolve" the sound, stripping away the device-specific resonance.
- Acoustic Style Randomization (ASR): A data augmentation technique that simulates thousands of "virtual" keyboards during training.
- Domain-Adversarial Training: Using a Gradient Reversal Layer (GRL) to force the model to produce embeddings that are useful for identifying keys but useless for identifying which keyboard was used.
- Supervised Contrastive Alignment: Ensuring that the sound of the "A" key on a Lenovo laptop is mapped close to the "A" key on an HP laptop in the embedding space.
- Sequence Level (LLM Assistance): Instead of treating keys as independent events, DECKER feeds the top-$k$ acoustic candidates into an LLM (like GPT-4 or FLAN-T5). The LLM uses linguistic priors to correct "acoustic typos," effectively acting as a powerful, context-aware decoder.
For VoIP-specific attacks, the authors introduce EvFlow-CTSSM, which uses a continuous-time state-space model to handle the non-linear distortions and "silence" inherent in streaming audio.
Datasets
The study introduces HEAR, the most comprehensive ASCA dataset to date. Unlike previous datasets that were often limited to a single user or device, HEAR includes:
- 53 participants across different genders and typing backgrounds.
- 37 different laptop keyboards (scissor-switch, membrane, and mechanical).
- Three recording channels: External microphone, on-device microphone, and VoIP streams (Zoom/Teams).
- Environmental variety: Quiet libraries, moderate offices, and noisy cafes.
Key Insights and Ideas
- The "Generalization Crisis": Strong baseline models lose roughly 40 points of accuracy when moved to an unseen keyboard. This confirms that standard deep learning models are essentially "fingerprinting" the device, not the keys.
- LLMs as a Risk Multiplier: The value of the LLM depends on the entropy of the target. It helps immensely with natural language and "human-like" passwords but struggles with high-entropy random strings. This highlights that human-selection bias in passwords is a primary enabler of these attacks.
- VoIP is Vulnerable: Even with aggressive audio compression and echo cancellation, keystroke transients survive. Using a Kolmogorov–Arnold Network (KAN) adaptation head, the authors were able to recover much of the accuracy lost to codec distortion.
Key Results
- Cross-Keyboard Accuracy: DECKER achieved 81.3% top-1 accuracy on keyboards it had never seen during training, compared to just 47-58% for standard unimodal baselines.
- Sentence Reconstruction: When paired with GPT-4, the system reached a 93.2% character accuracy and successfully recovered 72.8% of full sentences perfectly.
- Feasibility: The entire pipeline runs in real-time on a commodity Apple M1 laptop, processing roughly 430 keystrokes per second. This proves that an attacker doesn't need a GPU cluster; a standard laptop is sufficient.
Limitations and Future Work
- Continuous Streams: While the paper addresses "segmentation jitter," the primary evaluation still relies on pre-segmented keystrokes. Moving to fully "segmentation-free" attacks in highly overlapping, rapid typing scenarios remains a challenge.
- Extreme Noise: While it handles cafe noise, environments with strong impulsive noise (like a construction site) would likely require more advanced denoising.
- Defensive Directions: The authors suggest that since LLMs are the "multiplier," the most effective defenses might be linguistic (e.g., using high-entropy password managers) or system-level (e.g., injecting "keystroke noise" into the audio stream).
For more details, the HEAR dataset and DECKER framework are publicly documented on GitHub.
Summary updated