Unlabeled Multimodal Datasets for Robust Emotion Recognition

Despite the vast literature on emotion recognition, intra- and inter-subject variability and emotional cultural differences are still outstanding challenges that limit the state-of-the-art model’s generalization ability and robustness to out-of-training distribution data. We argue that potential solution to these problems could be based on the use of unlabeled large-scale datasets available online, in particular those providing multi-modal streams, whose availability is increasing. The aim of this work is to explore the use of multi-modal large datasets, with both EEG and Eye-tracking data streams, to increase the robustness of an emotion recognition downstream task. Three data sets on different scales, with data from different numbers of subjects (117, 47, and 16 subjects) for different pretext tasks (gaze estimation, attention type recognition, and emotion recognition), were used for self-supervised pretraining of a deep learning model and compared with the performance obtained under fully supervised training with a small emotion recognition dataset, SEED-IV (15 subjects). The use of unlabeled multimodal datasets has shown promising results to improve emotion recognition robustness using Eye-related data, although further research is needed to fully benefit from the unprecedented amount of data available in the near future.