Novel Spatio-Temporal Continuous Sign Language Recognition Using an Attentive Multi-Feature Network
Issued Date
2022-09-01
Resource Type
ISSN
14248220
Scopus ID
2-s2.0-85137557544
Pubmed ID
36080911
Journal Title
Sensors
Volume
22
Issue
17
Rights Holder(s)
SCOPUS
Bibliographic Citation
Sensors Vol.22 No.17 (2022)
Suggested Citation
Aditya W., Shih T.K., Thaipisutikul T., Fitriajie A.S., Gochoo M., Utaminingrum F., Lin C.Y. Novel Spatio-Temporal Continuous Sign Language Recognition Using an Attentive Multi-Feature Network. Sensors Vol.22 No.17 (2022). doi:10.3390/s22176452 Retrieved from: https://repository.li.mahidol.ac.th/handle/20.500.14594/83626
Title
Novel Spatio-Temporal Continuous Sign Language Recognition Using an Attentive Multi-Feature Network
Other Contributor(s)
Abstract
Given video streams, we aim to correctly detect unsegmented signs related to continuous sign language recognition (CSLR). Despite the increase in proposed deep learning methods in this area, most of them mainly focus on using only an RGB feature, either the full-frame image or details of hands and face. The scarcity of information for the CSLR training process heavily constrains the capability to learn multiple features using the video input frames. Moreover, exploiting all frames in a video for the CSLR task could lead to suboptimal performance since each frame contains a different level of information, including main features in the inferencing of noise. Therefore, we propose novel spatio-temporal continuous sign language recognition using the attentive multi-feature network to enhance CSLR by providing extra keypoint features. In addition, we exploit the attention layer in the spatial and temporal modules to simultaneously emphasize multiple important features. Experimental results from both CSLR datasets demonstrate that the proposed method achieves superior performance in comparison with current state-of-the-art methods by 0.76 and 20.56 for the WER score on CSL and PHOENIX datasets, respectively.