Introduction
In traditional machine learning, the "feature extraction" step is often the most labor-intensive. Researchers typically rely on hand-crafted features like Mel-frequency cepstral coefficients (MFCCs) for audio or Histograms of Oriented Gradients (HOG) for computer vision. While effective, these require significant domain expertise and may inadvertently discard useful information present in the raw signal.
End2You is an open-source Python toolkit built on TensorFlow that enables end-to-end multimodal profiling. By training directly on raw data—such as audio, video, and physiological signals—it eliminates the need for manual feature engineering and provides a flexible framework for building complex, multisensorial deep learning models. While deep learning has popularized automatic feature extraction, End2You fills the gap for a unified, generic toolkit specifically designed for end-to-end multimodal profiling.
Methods
The End2You workflow is designed for modularity and ease of use, following a pipeline from raw data to prediction.
The pipeline consists of four primary stages:
- Data Preparation: Raw information is converted into the
.tfrecordformat for efficient processing. - Feature Extraction: The toolkit provides pre-defined models for different modalities, including a 2-block CNN for audio and a ResNet-50 for video. Physiological signals are handled via RNNs or FCNs.
- Sequence Modelling: To capture temporal context, features are passed through GRU or LSTM layers.
- Fusion: Users can concatenate outputs from different modality-specific models and stack further RNN or FCN layers on top to perform multimodal prediction.
Experimental Setup
The toolkit was evaluated using the RECOLA database (Remote Collaborative and Affective Interactions), which served as the benchmark for the AVEC 2016 challenge. This dataset includes four modalities: Audio, Video, Electro-cardiogram (ECG), and Electro-dermal activity (EDA). The objective was to predict two time-continuous emotional dimensions: arousal (intensity) and valence (positivity/negativity).
Results and Discussion
The toolkit was benchmarked against the AVEC 2016 baseline and the challenge winner using the Concordance Correlation Coefficient (CCC).
Key findings include:
- Audio: End2You achieved a CCC of .669 for arousal, outperforming the baseline (.648).
- Video: For valence, End2You reached .561 CCC, significantly beating the baseline (.507).
- Multimodal: While End2You performed well, it did not surpass the AVEC 2016 winner. This is likely due to the winner's use of specific hand-crafted physiological features (like heart rate variability) which provide more information than the raw ECG/EDA signals used by End2You.
The results demonstrate that "self-learning" features directly from raw data can yield performance comparable to, or better than, expert-engineered features. Furthermore, the toolkit confirms that audio models generally perform better for arousal, while video models are superior for valence.
Conclusion
End2You provides a "Lego-like" ability to stack different architectures for different modalities and fuse them into a single end-to-end trainable system. While there is a performance gap in the physiological domain compared to highly specialized hand-crafted features, the toolkit's flexibility is its primary value. Future work includes adding support for text information, incorporating more diverse models, and implementing automatic face recognition and extraction directly from raw video files.
Summary updated