End2You - The Imperial Toolkit for Multimodal Profiling by End-to-End Learning

Panagiotis Tzirakis, Stefanos Zafeiriou, Björn W. SchullerarXiv 2018 · arXiv

Introduction

In traditional machine learning, the "feature extraction" step is often the most labor-intensive. Researchers typically rely on hand-crafted features like Mel-frequency cepstral coefficients (MFCCs) for audio or Histograms of Oriented Gradients (HOG) for computer vision. While effective, these require significant domain expertise and may inadvertently discard useful information present in the raw signal.

End2You is an open-source Python toolkit built on TensorFlow that enables end-to-end multimodal profiling. By training directly on raw data—such as audio, video, and physiological signals—it eliminates the need for manual feature engineering and provides a flexible framework for building complex, multisensorial deep learning models. While deep learning has popularized automatic feature extraction, End2You fills the gap for a unified, generic toolkit specifically designed for end-to-end multimodal profiling.

Methods

The End2You workflow is designed for modularity and ease of use, following a pipeline from raw data to prediction.

The End2You workflow pipeline from raw input to prediction.

The pipeline consists of four primary stages:

  1. Data Preparation: Raw information is converted into the .tfrecord format for efficient processing.
  2. Feature Extraction: The toolkit provides pre-defined models for different modalities, including a 2-block CNN for audio and a ResNet-50 for video. Physiological signals are handled via RNNs or FCNs.
  3. Sequence Modelling: To capture temporal context, features are passed through GRU or LSTM layers.
  4. Fusion: Users can concatenate outputs from different modality-specific models and stack further RNN or FCN layers on top to perform multimodal prediction.

Experimental Setup

The toolkit was evaluated using the RECOLA database (Remote Collaborative and Affective Interactions), which served as the benchmark for the AVEC 2016 challenge. This dataset includes four modalities: Audio, Video, Electro-cardiogram (ECG), and Electro-dermal activity (EDA). The objective was to predict two time-continuous emotional dimensions: arousal (intensity) and valence (positivity/negativity).

Results and Discussion

The toolkit was benchmarked against the AVEC 2016 baseline and the challenge winner using the Concordance Correlation Coefficient (CCC).

Performance comparison on the RECOLA database (CCC metrics).

Key findings include:

  • Audio: End2You achieved a CCC of .669 for arousal, outperforming the baseline (.648).
  • Video: For valence, End2You reached .561 CCC, significantly beating the baseline (.507).
  • Multimodal: While End2You performed well, it did not surpass the AVEC 2016 winner. This is likely due to the winner's use of specific hand-crafted physiological features (like heart rate variability) which provide more information than the raw ECG/EDA signals used by End2You.

The results demonstrate that "self-learning" features directly from raw data can yield performance comparable to, or better than, expert-engineered features. Furthermore, the toolkit confirms that audio models generally perform better for arousal, while video models are superior for valence.

Conclusion

End2You provides a "Lego-like" ability to stack different architectures for different modalities and fuse them into a single end-to-end trainable system. While there is a performance gap in the physiological domain compared to highly specialized hand-crafted features, the toolkit's flexibility is its primary value. Future work includes adding support for text information, incorporating more diverse models, and implementing automatic face recognition and extraction directly from raw video files.

Summary updated

Abstract

We introduce End2You -- the Imperial College London toolkit for multimodal profiling by end-to-end deep learning. End2You is an open-source toolkit implemented in Python and is based on Tensorflow. It provides capabilities to train and evaluate models in an end-to-end manner, i.e., using raw input. It supports input from raw audio, visual, physiological or other types of information or combination of those, and the output can be of an arbitrary representation, for either classification or regression tasks. To our knowledge, this is the first toolkit that provides generic end-to-end learning for profiling capabilities in either unimodal or multimodal cases. To test our toolkit, we utilise the RECOLA database as was used in the AVEC 2016 challenge. Experimental results indicate that End2You can provide comparable results to state-of-the-art methods despite no need of expert-alike feature representations, but self-learning these from the data "end to end".
BibTeX
@misc{tzirakis2018end,
  author = {Panagiotis Tzirakis and Stefanos Zafeiriou and Björn W. Schuller},
  title = {{End2You - The Imperial Toolkit for Multimodal Profiling by End-to-End Learning}},
  year = {2018},
  eprint = {1802.01115},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.01115},
}
Figure