TL;DR
TSFFormer is a novel, purely spiking neural network architecture that achieves state-of-the-art accuracy in Human Activity Recognition (HAR) while maintaining an ultra-low energy footprint of just 0.1 pJ per operation. By combining frequency-domain Fourier attention with a biologically inspired, dual-compartment adaptive neuron, it breaks the traditional trade-off between the high accuracy of deep learning and the extreme efficiency of neuromorphic computing.
What Problem Does This Paper Address?
Human Activity Recognition (HAR) using wearable sensors (IMUs) is essential for health monitoring and sports analytics. However, current solutions face an "arithmetic-latency deadlock."
Traditional Deep Neural Networks (DNNs) are accurate but rely on dense floating-point Multiply-Accumulate (MAC) operations that drain battery life on edge devices. Spiking Neural Networks (SNNs) offer a "green" alternative by using sparse, energy-efficient Accumulate (AC) operations, but they historically struggle with three issues:
- Spatial Blind Spots: They often flatten sensor data, losing the physical relationship between different body parts.
- Neural Rigidity: Standard spiking neurons use fixed firing thresholds, making them poor at adapting to the varying rhythms of human movement.
- Gradient Degradation: Training deep SNNs over long sequences is notoriously difficult, often leading to slow convergence or poor performance.
Methods
The authors introduce TSFFormer, a purely spiking Fourier Transformer. Its architecture is built on three main pillars:
- MultiHeadFourier Attention: Instead of the standard dot-product attention (which has $O(V^2)$ complexity), this module performs global mixing in the frequency domain using the Fast Fourier Transform (FFT). This reduces complexity to $O(V \log V)$, saves ~50% GPU memory, and is naturally suited for capturing the periodic patterns inherent in human motion.
- TSLIFNode (Temporal Spiking LIF): Inspired by the dual-compartment (dendrite and soma) structure of biological neurons, this node uses dynamic thresholds. It adjusts its firing sensitivity based on historical activity—lowering the threshold during intense movement to capture detail and raising it at rest to suppress noise.
- Topology-Preserving Embedding (MS-SPS): Unlike models that flatten data, TSFFormer uses a non-downsampling patch embedding to preserve the spatial structure of sensor nodes across the body.
Datasets
The model was evaluated on 8 diverse public IMU benchmarks to ensure robustness across different scenarios:
- Multi-node: PAMAP2, TNDA-HAR, and HuGaDB (large-scale, asymmetric sensor nodes).
- Complex/Daily Activity: WISDM and OPPORTUNITY.
- Single-node: USC-HAD.
- Clinical/Elderly: HAR70 (specifically used to test the adaptive thresholding on non-stationary rhythms).
Key Insights and Ideas
- Frequency Domain is a Natural Fit for HAR: Human activities like walking or running are inherently periodic. By moving attention to the frequency domain, the model captures global dependencies more efficiently than spatial-only transformers.
- SNNs Can Beat ANNs: One of the most significant findings is that TSFFormer doesn't just match SNN performance; it outperforms the classic DeepConvLSTM (an Artificial Neural Network) on several datasets. This challenges the assumption that SNNs are inherently less capable than their non-spiking counterparts.
- Adaptive Thresholds Solve Gradient Issues: The dual-compartment TSLIF neuron acts as a powerful regularizer. By dynamically modulating thresholds, it mitigates the gradient vanishing/explosion problems common in deep SNN training, leading to a "cliff-like" drop in loss during early training stages.
Key Results
TSFFormer establishes new state-of-the-art (SOTA) benchmarks for spiking networks across all 8 tested datasets.
- Accuracy: On the WISDM dataset, it achieved 86.35% accuracy, a massive 33.92% leap over the next best SNN (SpikeTCN). It also outperformed the ANN baseline DeepConvLSTM (71.23%).
- Efficiency: The model operates with an energy footprint of 0.1 pJ, compared to the ~4.6 pJ required for standard FP32 MAC operations in DNNs.
- Training Speed: TSFFormer converges significantly faster than traditional models. As shown in the training dynamics comparison, it reaches a 50% accuracy threshold in just 5 epochs, whereas DeepConvLSTM requires 35 epochs—a 7x speedup in convergence.
Limitations and Future Work
While TSFFormer is highly effective, the authors noted a few areas for improvement:
- Single-Node Redundancy: On datasets with only a single sensor (like USC-HAD), the global frequency mixing can introduce slight redundancy, as there is no complex spatial topology to map.
- Computational Complexity of TSLIF: The dual-compartment neuron is more complex than a standard LIF neuron. While it improves accuracy and convergence, it adds a small amount of per-epoch computation time.
Future work may involve migrating this frequency-spiking hybrid structure to other multimodal time-series tasks beyond IMU data, such as EEG or financial forecasting.
Summary updated