Temporal Feedback Convolutional Recurrent Neural Networks for Keyword Spotting

Taejun Kim, Juhan Nam

Introduction

In the field of deep learning for audio processing, models have traditionally followed a "bottom-up" approach, where raw waveforms are processed through hierarchical layers to extract features for classification. While effective, this feedforward paradigm overlooks a fundamental characteristic of the human auditory system: bidirectionality. In biological hearing, the brain sends "efferent" signals back to the cochlea, allowing it to adaptively control sensitivity (gain) based on high-level context.

This paper introduces Temporal Feedback Convolutional Recurrent Neural Networks (TF-CRNNs), a novel architecture that mimics this biological feedback mechanism. By feeding high-level temporal context from a recurrent module back into low-level convolutional feature extractors, the model achieves state-of-the-art performance in speech command recognition.

Architecture and Methodology

The TF-CRNN architecture is built upon the SampleCNN framework, which processes raw audio waveforms directly. The core innovation lies in the integration of a feedback loop between the recurrent and convolutional stages.

Overview of the TF-CRNN architecture showing temporal feedback from the RNN hidden state to the convolutional blocks.

Temporal Feedback Mechanism

The model consists of a series of convolutional blocks followed by a Gated Recurrent Unit (GRU). Unlike standard CRNNs, the hidden state of the GRU at time step $t-1$ ($h^{t-1}$) is fed back into the convolutional blocks at time step $t$. This feedback signal passes through an "Excitation" block—a small fully-connected bottleneck—to generate channel-wise scaling values. These values multiply the activations in the CNN layers, effectively performing a form of dynamic gain control based on the preceding temporal context.

Block Variants

The researchers explored several variations of the convolutional blocks to determine the most effective way to integrate feedback and local statistics:

Comparison of various block architectures used in the study, including basic, SE, and temporal feedback (TF) variants.
  • Basic Block: A standard 1D-convolutional block.
  • TF-Basic Block: Integrates the temporal feedback signal ($h^{t-1}$) to scale features.
  • SE Block: A standard Squeeze-and-Excitation block that uses local channel statistics for scaling.
  • TF-SE Block: Combines both local channel statistics (via Squeeze) and global temporal feedback for feature scaling.
  • Res Block: A residual block structure used to improve gradient flow in deeper models.

Experimental Setup

The models were evaluated using the Google Speech Commands Dataset (v2), which contains approximately 100,000 one-second audio clips across 35 command categories. The researchers compared two training strategies: "many-to-one" (calculating loss only at the final time step) and "many-to-many" (calculating loss at every time step). The latter was found to be crucial for successfully training the feedback loops and preventing vanishing gradients.

Results and Discussion

The TF-CRNN variants consistently outperformed standard CNN and SENet baselines. The TF-Res-SE model, which combines residual connections with both local and temporal feedback, achieved a state-of-the-art accuracy of 96.55%, surpassing previous benchmarks like HarmonicCNN (96.39%).

Key Insights

  • Biological Mimicry: Visualization of the excitation scales revealed that the model acts as a sophisticated gain control mechanism, suppressing sensitivity during high-energy segments and amplifying it during low-energy segments.
  • Contextual Priming: The feedback mechanism allows the RNN to "prime" the CNN to look for specific discriminative features. This was particularly evident in the reduction of errors between phonetically similar words (e.g., "learn" vs. "left"), where confusion was reduced by nearly 50%.
  • Training Dynamics: The many-to-many training setup proved essential for the feedback architecture, likely by providing more frequent gradient signals that help the model learn useful temporal representations earlier in the sequence.

Conclusion

By incorporating bio-inspired temporal feedback, TF-CRNNs provide a more robust and context-aware approach to keyword spotting. The ability to dynamically adjust feature extraction based on high-level temporal context allows the model to better distinguish between similar-sounding commands. Future work will explore the scalability of this feedback mechanism to long-form speech and more complex acoustic environments.

Summary updated

BibTeX
@inproceedings{kim2022temporal,
  author = {Taejun Kim and Juhan Nam},
  title = {{Temporal Feedback Convolutional Recurrent Neural Networks for Keyword Spotting}},
  booktitle = {2022},
  year = {2022},
  doi = {10.23919/apsipaasc55919.2022.9979907},
}
Figure