Brain2Object: Printing Your Mind from Brain Signals with Spatial Correlation Embedding

Xiang Zhang, Lina Yao, Chaoran Huang, Salil S. Kanhere, Dalin ZhangarXiv 2018 · OpenReview

TL;DR

This paper introduces Brain2Object, an end-to-end system that translates visually-evoked brain signals (EEG) into commands for a 3D printer, allowing users to create physical replicas of objects they are observing. By combining advanced signal processing with deep learning, Brain2Object demonstrates a novel application of Brain-Computer Interfaces (BCI) for digital fabrication, achieving high accuracy in object recognition.

What Problem Does This Paper Address?

The paper tackles the challenge of directly translating human thought, specifically visual perception, into physical creation. While EEG has been used for communication and control, its application in directly fabricating objects based on visual observation is novel. A key hurdle is the low signal-to-noise ratio (SNR) of non-invasive EEG, which makes accurate decoding of brain patterns difficult. This work aims to overcome these limitations by developing a robust system that can reliably interpret EEG signals associated with visual object recognition and use this information to control a 3D printer.

Methods

The Brain2Object system employs a multi-stage approach:

  • Multi-class Common Spatial Pattern (CSP): This technique is used for spatial filtering to enhance the signal-to-noise ratio and extract discriminative features from the EEG data. CSP aims to maximize the variance between different object categories.
  • Dynamical Graph Representation (DGR): To capture the complex spatial relationships between EEG channels, the authors propose DGR. This method represents the brain network as a weighted undirected graph, where channels are vertices and their connections are edges. The adjacency matrix is dynamically updated during training, allowing the model to learn the evolving spatial dependencies.
  • Convolutional Neural Network (CNN): A CNN is used as the classifier. It takes the DGR-transformed EEG signals as input and learns higher-level spatial features for accurate object recognition. The CNN architecture consists of a convolutional layer followed by three fully-connected layers.

The system is designed as an end-to-end pipeline, from EEG signal acquisition to 3D printing.

Datasets

The authors evaluated their system using two datasets:

  • EEG-P (Public Dataset): This dataset is derived from the BCI2000 EEG system and contains 64 channels. It was collected while subjects performed visual stimuli related to actions (left hand, right hand, both hands, both feet), labeled 1-4. A subset of 20 subjects with 28,000 samples per subject was used.
  • EEG-L (Local Dataset): This dataset was collected by the authors using an Emotiv EPOC+ headset with 14 channels. It includes images of 4 objects: a car, a boat, Pinkie Pie Pony, and Mario. 8 healthy participants were recruited for data collection.

The EEG-P dataset is noted for its higher resolution and more subjects, while EEG-L was collected to demonstrate the system's practicality with a more accessible headset.

Key Insights and Ideas

  • Bridging the Gap between Mind and Matter: The core innovation is the direct translation of visual perception into physical object creation, showcasing a significant leap in BCI applications beyond communication.
  • Leveraging Spatial-Temporal Dynamics: The DGR approach is crucial for capturing the intricate spatial relationships between EEG channels, which traditional methods might overlook, leading to more robust feature extraction.
  • End-to-End System Design: The seamless integration of EEG acquisition, signal processing, object recognition, and 3D printing demonstrates a practical, real-world application of BCI technology.
  • Adaptability of the Model: The system's ability to achieve high accuracy on both a large public dataset and a smaller, locally collected dataset suggests its generalizability.

Key Results

  • High Recognition Accuracy: Brain2Object achieved a recognition accuracy of 92.58% on the EEG-P dataset and 75.23% on the EEG-L dataset.
  • Outperformance of Baselines: The proposed method consistently outperformed a wide range of baseline and state-of-the-art approaches, including KNN, RF, SVM, and standalone CNNs.
  • Acceptable Latency: The system demonstrated competitive latency, with an overall end-to-end latency of around 2 seconds in the online demonstration, which includes signal acquisition and transmission.
  • Online Demonstration: A working prototype successfully demonstrated the system's ability to print a replica of an observed object in real-time.

Limitations and Future Work

  • EEG Signal Quality: The performance of the pre-trained model is significantly influenced by the quality of EEG data. Noisy and low-resolution signals, as seen in the EEG-L dataset, lead to lower accuracy. Future work should focus on developing classification methods robust to such signals.
  • Hardware Adaptability: The model's adaptability across different EEG acquisition devices (e.g., Emotiv, NeuroSky, OpenBCI) with varying characteristics needs further investigation.
  • Object Repository Size: The current object repository is limited to four items. Expanding this to recognize a wider range of objects is a key future direction, which will require algorithms capable of discriminating between a larger number of classes. The authors note that accuracy decreases with an increased number of categories.
  • Online vs. Offline Performance: The online demonstration showed lower accuracy than offline evaluations, attributed to factors like user mental state, electrode conductivity variations, headset mounting, and maintaining concentration.
  • Future Research: The authors suggest exploring algorithms to improve multi-class classification performance and investigating the trade-off between accuracy and latency, potentially by reducing channel count while maintaining performance.

Summary updated

Abstract

Electroencephalography (EEG) signals are known to manifest differential patterns when individuals visually concentrate on different objects. In this work, we present an end-to-end digital fabrication system, Brain2Object, to print the 3D object that an individual is observing by decoding visually-evoked brain signals. We propose a unified training framework that combines multi-class Common Spatial Pattern and Convolutional Neural Networks to support the backend computation. We learn the dynamical graph representations of brain signals to accurately capture the structural information among EEG channels. A user-friendly interface is developed as the system front end. Brain2Object presents a streamlined end-to-end workflow that can serve as a template for deeper integration of BCI technologies to assist with our routine activities. The proposed system is evaluated extensively using offline experiments and through an online demonstrator. The experimental results show that our approach can achieve the recognition accuracy of 92.58% on a benchmark dataset and 75.23% on a locally collected dataset. Moreover, our method consistently outperforms a wide range of baseline and state-of-the-art approaches. The proof-of-concept corroborates the practicality of our approach and illustrates the ease with which such a system could be deployed.
BibTeX
@misc{zhang2018brain,
  author = {Xiang Zhang and Lina Yao and Chaoran Huang and Salil S. Kanhere and Dalin Zhang},
  title = {{Brain2Object: Printing Your Mind from Brain Signals with Spatial Correlation Embedding}},
  year = {2018},
  eprint = {1810.02223},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.02223},
}
Figure