Enhancing Motoric Impulsivity Detection in Children through Deep Learning and Body Keypoint Recognition

Fahmy F. Dalimarta - Universitas Dian Nuswantoro, Semarang 50131, Indonesia
Pulung N. Andono - Universitas Dian Nuswantoro, Semarang 50131, Indonesia
Moch. A. Soeleman - Universitas Dian Nuswantoro, Semarang 50131, Indonesia
Zainal A. Hasibuan - Universitas Dian Nuswantoro, Semarang 50131, Indonesia


Citation Format:



DOI: http://dx.doi.org/10.62527/joiv.9.1.2779

Abstract


Quantifying motoric impulsivity in pediatric settings is crucial for safeguarding children and for devising effective intervention strategies. Existing quantitative techniques, such as accelerometry, have been utilized to assess it, but they often prove insufficient for accurately differentiating impulsive movements from regular ones. Conventional assessment methods are frequently used and rely on subjective assessments, which hinders the accurate characterization of impulsive behavior. To address this research gap, our study introduced an innovative objective approach using computer vision and deep learning techniques. We utilized MediaPipe to track precise body movement data from a child. The data were then analyzed using a Bidirectional Long Short-Term Memory (Bi-LSTM) network to process sequential information and recognize patterns indicative of impulsivity. Our approach successfully distinguished impulsive movements, marked by rapid changes in position and inconsistent movement velocities, from typical behavioral patterns with an accuracy rate of 98.21%. This research demonstrates the effectiveness of combining computer vision and deep learning to measure motoric impulsivity more precisely and impartially than prevailing qualitative techniques. Our model quantifies behaviors, enabling the development of improved safety protocols and targeted interventions in educational and recreational settings. This research has broader implications, suggesting a framework for future studies on pediatric motion analysis and behavioral assessment.

Keywords


Motoric impulsivity; quantitative; Bi-LSTM; keypoints; pose estimator

Full Text:

PDF

References


J. C. Corona, “Role of Oxidative Stress and Neuroinflammation in Attention-Deficit/Hyperactivity Disorder,” Antioxidants, vol. 9, no. 11, p. 1039, Oct. 2020, doi: 10.3390/antiox9111039.

O. Grimm et al., “Impulsivity and Venturesomeness in an Adult ADHD Sample: Relation to Personality, Comorbidity, and Polygenic Risk,” Front Psychiatry, vol. 11, Dec. 2020, doi: 10.3389/fpsyt.2020.557160.

E. K. Edmiston et al., “Assessing Relationships Among Impulsive Sensation Seeking, Reward Circuitry Activity, and Risk for Psychopathology: A Functional Magnetic Resonance Imaging Replication and Extension Study,” Biol Psychiatry Cogn Neurosci Neuroimaging, vol. 5, no. 7, pp. 660–668, Jul. 2020, doi: 10.1016/j.bpsc.2019.10.012.

T. M. K. ElSehrawy, E. A. Elela, G. A. M. Hassan, M. El Missiry, S. A. Nabi, and M. F. Soliman, “A study of emotional intelligence in an Egyptian sample of offspring of patients with schizophrenia,” Middle East Current Psychiatry, vol. 29, no. 1, p. 48, Dec. 2022, doi: 10.1186/s43045-022-00216-x.

X. Chen and S. Li, “Serial mediation of the relationship between impulsivity and suicidal ideation by depression and hopelessness in depressed patients,” BMC Public Health, vol. 23, no. 1, p. 1457, Jul. 2023, doi: 10.1186/s12889-023-16378-0.

J. Salles et al., “Indirect effect of impulsivity on suicide risk through self-esteem and depressive symptoms in a population with treatment-resistant depression: A FACE-DR study,” J Affect Disord, vol. 347, pp. 306–313, Feb. 2024, doi: 10.1016/j.jad.2023.11.063.

T. Veliki, Z. Užarević, and S. Dubovicki, “Self-Evaluated ADHD Symptoms as Risk Adaptation Factors in Elementary School Children,” Drustvena istrazivanja, vol. 28, no. 3, pp. 503–522, Oct. 2019, doi: 10.5559/di.28.3.07.

A. Bandyopadhyay et al., “Behavioural difficulties in early childhood and risk of adolescent injury,” Arch Dis Child, vol. 105, no. 3, pp. 282–287, Mar. 2020, doi: 10.1136/archdischild-2019-317271.

J. Pereira, P. Vagos, A. Fonseca, H. Moreira, M. C. Canavarro, and D. Rijo, “The Children’s Revised Impact of Event Scale: Dimensionality and Measurement Invariance in a Sample of Children and Adolescents Exposed to Wildfires,” J Trauma Stress, vol. 34, no. 1, pp. 35–45, Feb. 2021, doi: 10.1002/jts.22634.

B. Wolff, E. Sciberras, J. He, G. Youssef, V. Anderson, and T. J. Silk, “The Role of Sleep in the Relationship Between ADHD Symptoms and Stop Signal Task Performance,” J Atten Disord, vol. 25, no. 13, pp. 1881–1894, Nov. 2021, doi: 10.1177/1087054720943290.

J. C. Vázquez, O. Martin de la Torre, J. López Palomé, and D. Redolar-Ripoll, “Effects of Caffeine Consumption on Attention Deficit Hyperactivity Disorder (ADHD) Treatment: A Systematic Review of Animal Studies,” Nutrients, vol. 14, no. 4, p. 739, Feb. 2022, doi: 10.3390/nu14040739.

G. Abbadessa et al., “Digital therapeutics in neurology,” J Neurol, vol. 269, no. 3, pp. 1209–1224, Mar. 2022, doi: 10.1007/s00415-021-10608-4.

I. Weygers, M. Kok, M. Konings, H. Hallez, H. De Vroey, and K. Claeys, “Inertial Sensor-Based Lower Limb Joint Kinematics: A Methodological Systematic Review,” Sensors, vol. 20, no. 3, p. 673, Jan. 2020, doi: 10.3390/s20030673.

A. Jalal, M. A. K. Quaid, S. B. ud din Tahir, and K. Kim, “A Study of Accelerometer and Gyroscope Measurements in Physical Life-Log Activities Detection Systems,” Sensors, vol. 20, no. 22, p. 6670, Nov. 2020, doi: 10.3390/s20226670.

D. Kobsar et al., “Wearable Inertial Sensors for Gait Analysis in Adults with Osteoarthritis—A Scoping Review,” Sensors, vol. 20, no. 24, p. 7143, Dec. 2020, doi: 10.3390/s20247143.

C. K. Conners, J. Pitkanen, and S. R. Rzepa, “Conners 3rd Edition (Conners 3; Conners 2008),” in Encyclopedia of Clinical Neuropsychology, New York, NY: Springer New York, 2011, pp. 675–678. doi: 10.1007/978-0-387-79948-3_1534.

G. J. DuPaul, T. J. Power, A. D. Anastopoulos, and R. C. Reid, “Adhd Rating Scale-IV: Checklists, Norms, and Clinical Interpretation,” 1998. [Online]. Available: https://api.semanticscholar.org/CorpusID:141673166

G. J. DuPaul and G. Stoner, ADHD in the schools: Assessment and intervention strategies, 3rd ed. New York, NY, US: The Guilford Press, 2014.

J. Barth, J. W. Klaesner, and C. E. Lang, “Relationships between accelerometry and general compensatory movements of the upper limb after stroke,” J Neuroeng Rehabil, vol. 17, no. 1, p. 138, Dec. 2020, doi: 10.1186/s12984-020-00773-4.

M. S. Islam et al., “Using AI to measure Parkinson’s disease severity at home,” NPJ Digit Med, vol. 6, no. 1, p. 156, Aug. 2023, doi: 10.1038/s41746-023-00905-9.

K. Luxem et al., “Open-source tools for behavioral video analysis: Setup, methods, and best practices,” Elife, vol. 12, Mar. 2023, doi: 10.7554/eLife.79305.

H. Imai et al., “A lack of specific motor patterns between rhythmic/non-rhythmic masticatory muscle activity and bodily movements in sleep bruxism,” J Prosthodont Res, vol. 65, no. 3, p. JPR_D_20_00012, 2021, doi: 10.2186/jpr.JPR_D_20_00012.

W. Baccinelli et al., “Movidea: A Software Package for Automatic Video Analysis of Movements in Infants at Risk for Neurodevelopmental Disorders,” Brain Sci, vol. 10, no. 4, p. 203, Mar. 2020, doi: 10.3390/brainsci10040203.

A. G. Boyarov, O. O. Vlasov, and I. S. Siparov, “Methodology for Determining Time Intervals by Video Recordings,” Theory and Practice of Forensic Science, vol. 17, no. 2, pp. 58–69, Aug. 2022, doi: 10.30764/1819-2785-2022-2-58-69.

J. H. Hsiao, H. Lan, Y. Zheng, and A. B. Chan, “Eye movement analysis with hidden Markov models (EMHMM) with co-clustering,” Behav Res Methods, vol. 53, no. 6, pp. 2473–2486, Dec. 2021, doi: 10.3758/s13428-021-01541-5.

A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional LSTM and other neural network architectures,” Neural Networks, vol. 18, no. 5–6, pp. 602–610, Jul. 2005, doi: 10.1016/j.neunet.2005.06.042.

Ü. Atila and F. Sabaz, “Turkish lip-reading using Bi-LSTM and deep learning models,” Engineering Science and Technology, an International Journal, vol. 35, p. 101206, Nov. 2022, doi: 10.1016/j.jestch.2022.101206.

A.-A. Liu, Z. Shao, Y. Wong, J. Li, Y.-T. Su, and M. Kankanhalli, “LSTM-based multi-label video event detection,” Multimed Tools Appl, vol. 78, no. 1, pp. 677–695, Jan. 2019, doi: 10.1007/s11042-017-5532-x.

F. Carrara, P. Elias, J. Sedmidubsky, and P. Zezula, “LSTM-based real-time action detection and prediction in human motion streams,” Multimed Tools Appl, vol. 78, no. 19, pp. 27309–27331, Oct. 2019, doi: 10.1007/s11042-019-07827-3.

R. Rijayanti, M. Hwang, and K. Jin, “Detection of Anomalous Behavior of Manufacturing Workers Using Deep Learning-Based Recognition of Human–Object Interaction,” Applied Sciences, vol. 13, no. 15, p. 8584, Jul. 2023, doi: 10.3390/app13158584.

M. A. Soeleman, C. Supriyanto, D. P. Prabowo, and P. N. Andono, “Video Violence Detection Using LSTM and Transformer Networks Through Grid Search-Based Hyperparameters Optimization,” International Journal of Safety and Security Engineering, vol. 12, no. 05, pp. 615–622, Nov. 2022, doi: 10.18280/ijsse.120510.

W. Ullah, A. Ullah, I. U. Haq, K. Muhammad, M. Sajjad, and S. W. Baik, “CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks,” Multimed Tools Appl, vol. 80, no. 11, pp. 16979–16995, May 2021, doi: 10.1007/s11042-020-09406-3.

G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artif Intell Rev, vol. 53, no. 8, pp. 5929–5955, Dec. 2020, doi: 10.1007/s10462-020-09838-1.

Google, “MediaPipe Pose Landmark.” Accessed: Jan. 05, 2024. [Online]. Available: https://developers.google.com/mediapipe/solutions/vision/pose_landmarker

F. F. Dalimarta, Z. A. Hasibuan, P. N. Andono, Pujiono, and M. A. Soeleman, “Lower Body Detection and Tracking with AlphaPose and Kalman Filters,” in Proceedings - 2021 International Seminar on Application for Technology of Information and Communication: IT Opportunities and Creativities for Digital Innovation and Communication within Global Pandemic, iSemantic 2021, 2021. doi: 10.1109/iSemantic52711.2021.9573221.

J. Qiu, X. Yan, W. Wang, W. Wei, and K. Fang, “Skeleton-Based Abnormal Behavior Detection Using Secure Partitioned Convolutional Neural Network Model,” IEEE J Biomed Health Inform, vol. 26, no. 12, pp. 5829–5840, Dec. 2022, doi: 10.1109/JBHI.2021.3137334.