Hybrid Deep Learning Approach For Stress Detection Model Through Speech Signal

Phie Chyan - Hasanuddin University, Gowa, 92171, Indonesia
Andani Achmad - Hasanuddin University, Gowa, 92171, Indonesia
Ingrid Nurtanio - Hasanuddin University, Gowa, 92171, Indonesia
Intan Areni - Hasanuddin University, Gowa, 92171, Indonesia

Citation Format:

DOI: http://dx.doi.org/10.30630/joiv.7.4.02026


Stress is a psychological condition that requires proper treatment due to its potential long-term effects on health and cognitive faculties. This is particularly pertinent when considering pre- and early-school-age children, where stress can yield a range of adverse effects. Furthermore, detection in children requires a particular approach different from adults because of their physical and cognitive limitations. Traditional approaches, such as psychological assessments or the measurement of biosignal parameters prove ineffective in this context. Speech is also one of the approaches used to detect stress without causing discomfort to the subject and does not require prerequisites for a certain level of cognitive ability. Therefore, this study introduced a hybrid deep learning approach using supervised and unsupervised learning in a stress detection model. The model predicted the stress state of the subject and provided positional data point analysis in the form of a cluster map to obtain information on the degree using CNN and GSOM algorithms. The results showed an average accuracy and F1 score of 94.7% and 95%, using the children's voice dataset. To compare with the state-of-the-art, model were tested with the open-source DAIC Woz dataset and obtained average accuracy and F1 scores of 89% and 88%. The cluster map generated by GSOM further underscored the discerning capability in identifying stress and quantifying the degree experienced by the subjects, based on their speech patterns


stress detection; speech processing; deep learning; CNN; GSOM

Full Text:



E. S. Epel et al., "More than a feeling: A unified view of stress measurement for population science," Front. Neuroendocrinol., vol. 49, no. December 2017, pp. 146–169, 2018, doi: 10.1016/j.yfrne.2018.03.001.

M. Bucci, S. S. Marques, D. Oh, and N. B. Harris, "Toxic Stress in Children and Adolescents," Adv. Pediatr., vol. 63, no. 1, pp. 403–428, 2016, doi: 10.1016/j.yapd.2016.04.002.

M. Kaczmarek and S. Trambacz-Oleszak, "School-related stressors and the intensity of perceived stress experienced by adolescents in Poland," Int. J. Environ. Res. Public Health, vol. 18, no. 22, 2021, doi: 10.3390/ijerph182211791.

N. Garmezy, A. S. Masten, and A. Tellegen, "The study of stress and competence in children: a building block for developmental psychopathology.," Child Dev., vol. 55, no. 1, pp. 97–111, 1984, doi: 10.1111/j.1467-8624.1984.tb00276.x.

M. Rohmadi, M. Sudaryanto, C. Ulya, H. Akbariski, and U. Putri, "Case Study: Exploring Golden Age Students' Ability and Identifying Learning Activities in Kindergarten," Proc. First Brawijaya Int. Conf. Soc. Polit. Sci. BSPACE, 26-28 November, 2019, Malang, East Java, Indonesia., 2020, doi: 10.4108/eai.26-11-2019.2295218.

H. Yaribeygi, Y. Panahi, H. Sahraei, T. P. Johnston, and A. Sahebkar, "The impact of stress on body function: A review.," EXCLI J., vol. 16, pp. 1057–1072, 2017.

P. Morgado and J. Cerqueira, "The Impact of Stress on Cognition and Motivation," Front. Behav. Neurosci., 2018, doi: 10.1038/mp.2015.196.

M. Solmi et al., "Age at onset of mental disorders worldwide: a large-scale meta-analysis of 192 epidemiological studies," Mol. Psychiatry, vol. 27, no. 1, pp. 281–295, 2022, doi: 10.1038/s41380-021-01161-7.

M. Mohler-kuo, S. Dzemaili, S. Foster, L. Werlen, and S. Walitza, "Stress and mental health among children/adolescents, their parents, and young adults during the first COVID-19 lockdown in Switzerland," Int. J. Environ. Res. Public Health, vol. 18, no. 9, 2021, doi: 10.3390/ijerph18094668.

Y. Choi, Y. M. Jeon, L. Wang, and K. Kim, "A biological signal-based stress monitoring framework for children using wearable devices," Sensors (Switzerland), vol. 17, no. 9, pp. 1–16, 2017, doi: 10.3390/s17091936.

T. Y. Kim, L. Měsíček, and S. H. Kim, "Modeling of Child Stress-State Identification Based on Biometric Information in Mobile Environment," Mob. Inf. Syst., vol. 2021, 2021, doi: 10.1155/2021/5531770.

K. E. Smith and S. D. Pollak, "Early life stress and development: potential mechanisms for adverse outcomes," J. Neurodev. Disord., vol. 12, no. 1, p. 34, 2020, doi: 10.1186/s11689-020-09337-y.

Y. S. Can, N. Chalabianloo, D. Ekiz, J. Fernandez-Alvarez, G. Riva, and C. Ersoy, "Personal Stress-Level Clustering and Decision-Level Smoothing to Enhance the Performance of Ambulatory Stress Detection with Smartwatches," IEEE Access, vol. 8, pp. 38146–38163, 2020, doi: 10.1109/ACCESS.2020.2975351.

K. Kyriakou et al., "Detecting moments of stress from measurements of wearable physiological sensors," Sensors (Switzerland), vol. 19, no. 17, 2019, doi: 10.3390/s19173805.

S. Gedam and S. Paul, "A Review on Mental Stress Detection Using Wearable Sensors and Machine Learning Techniques," IEEE Access, vol. 9, pp. 84045–84066, 2021, doi: 10.1109/ACCESS.2021.3085502.

M. Chauhan, S. V. Vora, and D. Dabhi, "Effective stress detection using physiological parameters," Proc. 2017 Int. Conf. Innov. Information, Embed. Commun. Syst. ICIIECS 2017, vol. 2018-Janua, pp. 1–6, 2017, doi: 10.1109/ICIIECS.2017.8275853.

P. Chyan, A. Andani, I. Nurtanio, and I. Areni, "A Deep Learning Approach for Stress Detection Through Speech with Audio Feature Analysis," in The 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE-2022), IEEE, 2022, pp. 269–273.

G. M. Slavich, S. Taylor, and R. W. Picard, "Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations," stress, vol. 22, no. 4, pp. 408–413, 2019, doi: 10.1080/10253890.2019.1584180.

S. Paulmann, D. Furnes, A. M. Bøkenes, and P. J. Cozzolino, "How psychological stress affects emotional prosody," PLoS One, vol. 11, no. 11, pp. 1–21, 2016, doi: 10.1371/journal.pone.0165022.

K. Pisanski and P. Sorokowski, "Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress," Perception, vol. 50, no. 1, pp. 80–87, 2021, doi: 10.1177/0301006620978378.

K. Tomba, J. Dumoulin, E. Mugellini, O. A. Khaled, and S. Hawila, "Stress detection through speech analysis," in ICETE 2018 - Proceedings of the 15th International Joint Conference on e-Business and Telecommunications, 2018. doi: 10.5220/0006855803940398.

H. K. Shin, H. Han, K. Byun, and H. G. Kang, "Speaker-invariant Psychological Stress Detection Using Attention-based Network," 2020 Asia-Pacific Signal Inf. Process. Assoc. Annu. Summit Conf. APSIPA ASC 2020 - Proc., no. December, pp. 308–313, 2020.

R. Dillon and A. Ni Teoh, "Real-time Stress Detection Model and Voice Analysis: An Integrated VR-based Game for Training Public Speaking Skills," IEEE Conf. Games, pp. 1–4, 2021.

I. Madhavi, S. Chamishka, R. Nawaratne, V. Nanayakkara, D. Alahakoon, and D. De Silva, "A Deep Learning Approach for Work-Related Stress Detection from Audio Streams in Cyber-Physical Environments," IEEE Symp. Emerg. Technol. Fact. Autom. ETFA, vol. 2020-Septe, pp. 929–936, 2020, doi: 10.1109/ETFA46521.2020.9212098.

A. König et al., "Measuring stress in health professionals over the phone using automatic speech analysis during the COVID-19 pandemic: Observational Pilot study," J. Med. Internet Res., vol. 23, no. 4, pp. 1–14, 2021, doi: 10.2196/24191.

E. Rejaibi, A. Komaty, F. Meriaudeau, S. Agrebi, and A. Othmani, "MFCC-based Recurrent Neural Network for automatic clinical depression recognition and assessment from speech," Biomed. Signal Process. Control, vol. 71, pp. 1–14, 2022, doi: 10.1016/j.bspc.2021.103107.

G. Douzas, F. Bacao, J. Fonseca, and M. Khudinyan, "Imbalanced learning in land cover classification: Improving minority classes' prediction accuracy using the geometric SMOTE algorithm," Remote Sens., vol. 11, no. 24, 2019, doi: 10.3390/rs11243040.

Vandana, N. Marriwala, and D. Chaudhary, "A hybrid model for depression detection using deep learning," Meas. Sensors, vol. 25, no. November 2022, p. 100587, 2023, doi: 10.1016/j.measen.2022.100587.

N. Rafique, L. I. Al-Asoom, R. Latif, A. Al Sunni, and S. Wasi, "Comparing levels of psychological stress and its inducing factors among medical students," J. Taibah Univ. Med. Sci., vol. 14, no. 6, pp. 488–494, 2019, doi: 10.1016/j.jtumed.2019.11.002.

N. F. Narvaez Linares, V. Charron, A. J. Ouimet, P. R. Labelle, and H. Plamondon, "A systematic review of the Trier Social Stress Test methodology: Issues in promoting study comparison and replicable research," Neurobiol. Stress, vol. 13, p. 100235, 2020, doi: 10.1016/j.ynstr.2020.100235.

Q. Ren, Y. Li, and D. G. Chen, "Measurement invariance of the Kessler Psychological Distress Scale (K10) among children of Chinese rural-to-urban migrant workers," Brain Behav., vol. 11, no. 12, pp. 1–10, 2021, doi: 10.1002/brb3.2417.

J. Gratch et al., "The distress analysis interview corpus of human and computer interviews," Proc. 9th Int. Conf. Lang. Resour. Eval. Lr. 2014, pp. 3123–3128, 2014.

A. Défossez, G. Synnaeve, and Y. Adi, "Real-time speech enhancement in the waveform domain," Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, 2020.

H. P. Shi, J. H. Cao, and X. Liu, "Blind source separation for non-stationary signal based on time-frequency analysis," Proc. - 2011 4th Int. Conf. Intell. Networks Intell. Syst. ICINIS 2011, pp. 45–48, 2011, doi: 10.1109/ICINIS.2011.12.