k-FFNN: A priori knowledge infused Feed-forward Neural Networks

Sri Harsha Dumpala, Rupayan Chakraborty, Sunil Kumar KopparapuarXiv 2017 · OpenReview

Abstract

Recurrent Neural Networks (RNNs) are favored for sequential data due to their ability to capture temporal relationships. However, their reliance on large datasets to learn these correlations limits their effectiveness in low-resource scenarios. This paper explores the k-FFNN, a Feed-forward Neural Network (FFNN) architecture that incorporates a priori temporal knowledge during training. By injecting known temporal patterns into training labels, k-FFNN achieves performance comparable to or better than RNNs in data-sparse environments. We evaluate this approach using the MediaEval 2016 "Emotional Impact of Movies" task, demonstrating that explicit knowledge infusion can effectively substitute for architectural complexity.

Introduction: The Data Hunger of Recurrence

In deep learning, sequential tasks like speech processing or video analysis typically demand Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) units. These architectures utilize internal feedback loops to maintain a "memory" of previous inputs, allowing them to automatically discover temporal correlations.

Comparison of (a) Feed-forward Neural Network (FFNN) and (b) Recurrent Neural Network (RNN) architectures. RNNs use feedback loops to capture temporal dependencies, while FFNNs are memoryless.

While powerful, this automatic learning comes with a significant overhead: RNNs require vast amounts of data to tune their recurrent weights. In "low-resource" scenarios where data is scarce, RNNs often fail to converge on the underlying temporal structure. Conversely, standard Feed-Forward Neural Networks (FFNNs) are easier to train with limited data but are fundamentally "blind" to time, treating each input segment as an independent event. The k-FFNN addresses this gap by asking if a priori knowledge can be used to "teach" a simple FFNN about time.

Methodology: Knowledge Infusion via Label Shaping

The k-FFNN does not modify the network architecture itself. Instead, it alters the training targets to reflect known temporal dynamics.

  1. Segmentation: A long sequence (e.g., a 10-second audio clip) is divided into smaller 1-second segments.
  2. Knowledge Infusion: Instead of assigning the global sequence label to every segment, labels are modified by a function $f(i)$ representing a priori knowledge. For movie clips, emotions often follow a "fade-in/fade-out" pattern.
  3. Knowledge Functions: The authors tested several functions:
  4. Fn1 & Fn2: Smoothly scale labels at the start and end of a sequence to mimic natural emotional transitions.
  5. Fn3: A poorly-fitted control function to test the sensitivity of the model to incorrect assumptions.
  6. Training and Inference: A standard FFNN is trained on these shaped labels. During inference, the segment-level outputs are decoded back into a single sequence-level value using the inverse of the knowledge function.

Experimental Setup and Dataset

The study utilizes the MediaEval 2016 dataset, specifically the "Emotional Impact of Movies" task. This task requires predicting the shift in a viewer's emotional state—defined by Valence (positive/negative) and Arousal (exciting/calm)—after watching a short film clip.

The emotional impact task involves mapping a video stimulus to a shift in valence and arousal from an initial state to a final state.

The authors extracted 384 audio features using the openSMILE toolkit, eventually reducing them to 21 key features via WEKA for training.

Key Results and Insights

The k-FFNN was compared against standard FFNNs, RNNs, LSTMs, and Bidirectional LSTMs (BLSTMs) across varying training set sizes.

  • Superiority in Low-Resource Scenarios: With only 200 training samples, k-FFNN (using Fn1) achieved a Mean Squared Error (MSE) of 0.927, significantly outperforming the standard RNN (0.977). Its Pearson Correlation Coefficient (PCC) was 0.16, nearly double that of the RNN (0.079).
  • Convergence at Scale: As the training set size increased to the full 6814 samples, the performance gap closed. RNNs eventually learned the temporal patterns that k-FFNN was "told" at the start. This confirms that k-FFNN is a highly efficient alternative when data is limited.
  • Sensitivity to Knowledge Quality: The success of k-FFNN is tied to the accuracy of the a priori knowledge. When the "incorrect" Fn3 function was used, performance dropped below that of a standard FFNN, highlighting the importance of domain expertise.

Conclusion

The k-FFNN demonstrates that explicit mathematical knowledge can serve as a powerful substitute for architectural complexity. By "baking" temporal expectations into the training process, researchers can deploy simpler, more efficient models that excel in data-constrained environments. Future work may explore learnable knowledge functions that can adapt to diverse data sequences without requiring manual tuning.

Summary updated

Abstract

Recurrent neural network (RNN) are being extensively used over feed-forward neural networks (FFNN) because of their inherent capability to capture temporal relationships that exist in the sequential data such as speech. This aspect of RNN is advantageous especially when there is no a priori knowledge about the temporal correlations within the data. However, RNNs require large amount of data to learn these temporal correlations, limiting their advantage in low resource scenarios. It is not immediately clear (a) how a priori temporal knowledge can be used in a FFNN architecture (b) how a FFNN performs when provided with this knowledge about temporal correlations (assuming available) during training. The objective of this paper is to explore k-FFNN, namely a FFNN architecture that can incorporate the a priori knowledge of the temporal relationships within the data sequence during training and compare k-FFNN performance with RNN in a low resource scenario. We evaluate the performance of k-FFNN and RNN by extensive experimentation on MediaEval 2016 audio data ("Emotional Impact of Movies" task). Experimental results show that the performance of k-FFNN is comparable to RNN, and in some scenarios k-FFNN performs better than RNN when temporal knowledge is injected into FFNN architecture. The main contributions of this paper are (a) fusing a priori knowledge into FFNN architecture to construct a k-FFNN and (b) analyzing the performance of k-FFNN with respect to RNN for different size of training data.
BibTeX
@misc{dumpala2017k,
  author = {Sri Harsha Dumpala and Rupayan Chakraborty and Sunil Kumar Kopparapu},
  title = {{k-FFNN: A priori knowledge infused Feed-forward Neural Networks}},
  year = {2017},
  eprint = {1704.07055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.07055},
}
Figure