The effects of applying filters on EEG signals for classifying developers’ code comprehension

Lucian José Gonçales, Kleinner Farias, Lucas Silveira Kupssinskü, Matheus SegalottoJournal of Applied Research and Technology 2021 · DOI

Abstract

EEG signals are a relevant indicator for measuring aspects related to human factors in Software Engineering. EEG is used in software engineering to train machine learning techniques for a wide range of applications, including classifying task difficulty, and developers’ level of experience. The EEG signal contains noise such as abnormal readings, electrical interference, and eye movements, which are usually not of interest to the analysis, and therefore contribute to the lack of precision of the machine learning techniques. However, research in software engineering has not evidenced the effectiveness when applying these filters on EEG signals. The objective of this work is to analyze the effectiveness of filters on EEG signals in the software engineering context. As literature did not focus on the classification of developers’ code comprehension, this study focuses on the analysis of the effectiveness of applying EEG filters for training a machine learning technique to classify developers' code comprehension. A Random Forest (RF) machine learning technique was trained with filtered EEG signals to classify the developers' code comprehension. This study also trained another random forest classifier with unfiltered EEG data. Both models were trained using 10-fold cross-validation. This work measures the classifiers' effectiveness using the f-measure metric. This work used the t-test, Wilcoxon, and U Mann Whitney to analyze the difference in the effectiveness measures (f-measure) between the classifier trained with filtered EEG and the classifier trained with unfiltered EEG. The tests pointed out that there is a significant difference after applying EEG filters to classify developers' code comprehension with the random forest classifier. The conclusion is that the use of EEG filters significantly improves the effectivity to classify code comprehension using the random forest technique.

Introduction

In the field of Software Engineering (SE), understanding the cognitive state of developers is crucial for improving productivity and software quality. Researchers are increasingly turning to Electroencephalography (EEG) to monitor neural activity during programming tasks. The ultimate goal is to develop intelligent systems capable of detecting when a developer is experiencing high cognitive load, struggling with complex logic, or maintaining a "flow state."

However, a significant challenge in using EEG within SE contexts is the inherent "noise" in the data. Raw EEG signals capture not only brain activity but also physiological artifacts such as muscle movements, eye blinks (EOG), and heartbeats (ECG), as well as external electrical interference. While signal filtering is standard practice in clinical neuroscience, its specific impact on machine learning models within the SE domain has often been assumed rather than empirically validated. This study addresses this gap by investigating whether rigorous filtering actually improves the performance of classifiers designed to assess developers' code comprehension.

Methodology

The research followed a structured three-phase approach: EEG monitoring, data pre-processing, and data classification.

Overview of the research methodology including monitoring, pre-processing, and classification stages.

Data Collection

The study involved 35 participants, ranging from students to professional developers. Each participant performed 10 Java-based code comprehension tasks (e.g., calculating salaries or managing data structures). EEG data was captured using the Emotiv Epoc+ wireless headset, which provides 14 channels of neural data. Task outputs (correct, incorrect, or unanswered) served as the ground truth labels for the classification models.

Signal Pre-processing and Filtering

To isolate the neural signals relevant to code comprehension, the researchers implemented a multi-stage filtering pipeline.

The detailed EEG signal filtering pipeline used to remove noise and artifacts.

The filtering process consisted of three primary steps:

  1. Abnormal Signal Removal: Samples with amplitudes exceeding $\pm8 \mu V$ were discarded, as these typically represent sensor malfunctions or extreme physical movement rather than brain activity.
  2. Bandpass Filtering: A Finite Impulse Response (FIR) filter with a Hamming window was applied to retain frequencies between 1Hz and 40Hz. This effectively removed low-frequency baseline drift and high-frequency noise from electrical equipment.
  3. Artifact Removal via ICA: Independent Component Analysis (Fast ICA) was used to identify and remove eye-blink patterns. Since the headset lacked a dedicated EOG channel, the F8 electrode was utilized as a reference point for identifying these artifacts.

Classification

The study compared two Random Forest (RF) classifiers: one trained on the filtered EEG data (EEGf) and another on the unfiltered, raw data (EEGu). To ensure a fair comparison, the dataset was balanced using under-sampling techniques. Both models were evaluated using 10-fold cross-validation.

Results

The empirical results demonstrate a clear advantage for the filtered data across multiple performance metrics.

  • Effectiveness (F-measure): The model trained on filtered data achieved an average F-measure of 71%, representing a significant improvement over the 60% achieved by the unfiltered model.
  • Correlation (MCC): The Matthews Correlation Coefficient improved from 0.41 to 0.56, indicating a more robust and reliable classification.
  • Statistical Significance: Statistical analyses, including t-tests, Wilcoxon, and Mann-Whitney U tests, all yielded p-values significantly lower than 0.05 (e.g., $p = 0.000001$ for the t-test). This confirms that the performance gains from filtering are statistically significant and not due to random variation.

Discussion and Conclusion

This research provides critical empirical evidence for the necessity of signal filtering in SE-related EEG studies. By systematically removing noise, the researchers achieved a 10-11% boost in classification accuracy for code comprehension tasks.

A key insight from this work is the effective use of the F8 channel as a proxy for eye-blink detection in the absence of dedicated EOG hardware. This demonstrates how software-based processing can mitigate hardware limitations. However, the authors also highlight a trade-off: the computational cost of these filtering steps. While they improve accuracy, the processing lag may pose challenges for real-time developer support tools.

Future work will explore the impact of these filters on other machine learning architectures, such as Neural Networks, and investigate methods to optimize the pipeline for real-time applications in diverse, non-laboratory environments.

Summary updated

Abstract

EEG signals are a relevant indicator for measuring aspects related to human factors in Software Engineering. EEG is used in software engineering to train machine learning techniques for a wide range of applications, including classifying task difficulty, and developers’ level of experience. The EEG signal contains noise such as abnormal readings, electrical interference, and eye movements, which are usually not of interest to the analysis, and therefore contribute to the lack of precision of the machine learning techniques. However, research in software engineering has not evidenced the effectiveness when applying these filters on EEG signals. The objective of this work is to analyze the effectiveness of filters on EEG signals in the software engineering context. As literature did not focus on the classification of developers’ code comprehension, this study focuses on the analysis of the effectiveness of applying EEG filters for training a machine learning technique to classify developers' code comprehension. A Random Forest (RF) machine learning technique was trained with filtered EEG signals to classify the developers' code comprehension. This study also trained another random forest classifier with unfiltered EEG data. Both models were trained using 10-fold cross-validation. This work measures the classifiers' effectiveness using the f-measure metric. This work used the t-test, Wilcoxon, and U Mann Whitney to analyze the difference in the effectiveness measures (f-measure) between the classifier trained with filtered EEG and the classifier trained with unfiltered EEG. The tests pointed out that there is a significant difference after applying EEG filters to classify developers' code comprehension with the random forest classifier. The conclusion is that the use of EEG filters significantly improves the effectivity to classify code comprehension using the random forest technique.
BibTeX
@article{goncales2021effects,
  author = {Lucian José Gonçales and Kleinner Farias and Lucas Silveira Kupssinskü and Matheus Segalotto},
  title = {{The effects of applying filters on EEG signals for classifying developers’ code comprehension}},
  journal = {Journal of Applied Research and Technology},
  year = {2021},
  doi = {10.22201/icat.24486736e.2021.19.6.1299},
}
Figure