Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation
Issued Date
2022-09-01
Resource Type
eISSN
25042289
Scopus ID
2-s2.0-85138994749
Journal Title
Big Data and Cognitive Computing
Volume
6
Issue
3
Rights Holder(s)
SCOPUS
Bibliographic Citation
Big Data and Cognitive Computing Vol.6 No.3 (2022)
Suggested Citation
Wongpatikaseree K., Singkul S., Hnoohom N., Yuenyong S. Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation. Big Data and Cognitive Computing Vol.6 No.3 (2022). doi:10.3390/bdcc6030079 Retrieved from: https://repository.li.mahidol.ac.th/handle/20.500.14594/84256
Title
Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation
Author(s)
Author's Affiliation
Other Contributor(s)
Abstract
Language resources are the main factor in speech-emotion-recognition (SER)-based deep learning models. Thai is a low-resource language that has a smaller data size than high-resource languages such as German. This paper describes the framework of using a pretrained-model-based front-end and back-end network to adapt feature spaces from the speech recognition domain to the speech emotion classification domain. It consists of two parts: a speech recognition front-end network and a speech emotion recognition back-end network. For speech recognition, Wav2Vec2 is the state-of-the-art for high-resource languages, while XLSR is used for low-resource languages. Wav2Vec2 and XLSR have proposed generalized end-to-end learning for speech understanding based on the speech recognition domain as feature space representations from feature encoding. This is one reason why our front-end network was selected as Wav2Vec2 and XLSR for the pretrained model. The pre-trained Wav2Vec2 and XLSR are used for front-end networks and fine-tuned for specific languages using the Common Voice 7.0 dataset. Then, feature vectors of the front-end network are input for back-end networks; this includes convolution time reduction (CTR) and linear mean encoding transformation (LMET). Experiments using two different datasets show that our proposed framework can outperform the baselines in terms of unweighted and weighted accuracies.