Design and Development of a System for Monitoring Student Attention and Concentration during Learning using CNN Model and Face Landmark Detection

Syamsul Arifin - Institut Teknologi Sepuluh Nopember, ITS Campus, Raya ITS, Surabaya, 60111, Indonesia
Aulia Aisjaha - Institut Teknologi Sepuluh Nopember, ITS Campus, Raya ITS, Surabaya, 60111, Indonesia
Azzezza Fatima - Institut Teknologi Sepuluh Nopember, ITS Campus, Raya ITS, Surabaya, 60111, Indonesia
Haniah Mahmudah - Institut Teknologi Sepuluh Nopember, ITS Campus, Raya ITS, Surabaya, 60111, Indonesia


Citation Format:



DOI: http://dx.doi.org/10.62527/joiv.9.1.2897

Abstract


Mobile learning media has been wide and provides a tendency for lecturers to identify students' concentration levels in online classes. To bring the class into active learning, efforts are needed from lecturers and educational institutions to return students' concentration to the ongoing learning process. In this paper, a monitoring and alarm system is designed to increase student concentration and combines two elements of statistical analysis to validate CNN models that recognize face emotions in real time while learning. The research was carried out by recording face data using a camera, extracting digital features, and analyzing facial features. The results of the analysis are used as data input for the decision-making system regarding the level of concentration. The concentration level will be used to activate alarms and send them via chat so that students can focus on learning.The system is created by merging facial expression recognition (FER) and decision-making with a convolutional neural network. The system using a face landmark via camera V2 and a Raspberry Pi 4 performed with the Haar-Cascade classifier, extracting facial features. Face detection via camera is performed using the Haar-Cascade classifier, which extracts facial features. The results of CNN model face detection with landmark features showed good results, with weighted average performance of precision, recall, and F1-score close to 0.99. According to the implementation results, the average number of facial expressions identified in drowsy and neutral states. The device can alert lecturers to how frequently drowsy detects students within a 10-minute interval.


Keywords


Mobile learning; atention; monitoring; CNN model; face landmark

Full Text:

PDF

References


M. A. Almaiah, M. M. Alamri, and W. Al-Rahmi, “Applying the UTAUT Model to Explain the Students’ Acceptance of Mobile Learning System in Higher Education,” IEEE Access, vol. 7, pp. 174673–174686, 2019, doi: 10.1109/ACCESS.2019.2957206.

S. Criollo-C, A. Guerrero-Arias, Á. Jaramillo-Alcázar, and S. Luján-Mora, “Mobile learning technologies for education: Benefits and pending issues,” Appl. Sci., vol. 11, no. 9, 2021, doi: 10.3390/app11094111.

M. Uther, “Mobile learning—trends and practices,” Educ. Sci., vol. 9, no. 1, pp. 10–12, 2019, doi: 10.3390/educsci9010033.

S. Qun, “The Development of Mobile Education Resource Database under the Concept of Ubiquitous Learning,” Proc. - 2021 13th Int. Conf. Meas. Technol. Mechatronics Autom. ICMTMA 2021, pp. 725–728, 2021, doi: 10.1109/ICMTMA52658.2021.00167.

C. Demazière et al., “Enhancing higher education through hybrid and flipped learning: Experiences from the GRE@T-PIONEeR project,” Nucl. Eng. Des., vol. 421, no. February, 2024, doi: 10.1016/j.nucengdes.2024.113028.

K. Alhumaid, M. Habes, and S. A. Salloum, “Examining the Factors Influencing the Mobile Learning Usage during COVID-19 Pandemic: An Integrated SEM-ANN Method,” IEEE Access, vol. 9, pp. 102567–102578, 2021, doi: 10.1109/ACCESS.2021.3097753.

M. N. Hasnine, H. T. T. Bui, T. T. T. Tran, H. T. Nguyen, G. Akçapõnar, and H. Ueda, “Students’ emotion extraction and visualization for engagement detection in online learning,” Procedia Comput. Sci., vol. 192, pp. 3423–3431, 2021, doi: 10.1016/j.procs.2021.09.115.

S. Hangaragi, T. Singh, and N. Neelima, “Face Detection and Recognition Using Face Mesh and Deep Neural Network,” Procedia Comput. Sci., vol. 218, pp. 741–749, 2022, doi: 10.1016/j.procs.2023.01.054.

G. Kaur et al., “Face mask recognition system using CNN model,” Neurosci. Informatics, vol. 2, no. 3, p. 100035, 2022, doi: 10.1016/j.neuri.2021.100035.

D. Bhagat, A. Vakil, R. K. Gupta, and A. Kumar, “Facial Emotion Recognition (FER) using Convolutional Neural Network (CNN),” Procedia Comput. Sci., vol. 235, no. 2023, pp. 2079–2089, 2024, doi: 10.1016/j.procs.2024.04.197.

D. Wang, H. Yu, D. Wang, and G. Li, “Face recognition system based on CNN,” Proc. - 2020 Int. Conf. Comput. Inf. Big Data Appl. CIBDA 2020, pp. 470–473, 2020, doi: 10.1109/CIBDA50819.2020.00111.

B. R. Ilyas, B. Mohammed, M. Khaled, and K. Miloud, “Enhanced Face Recognition System Based on Deep CNN,” Proc. - 2019 6th Int. Conf. Image Signal Process. their Appl. ISPA 2019, 2019, doi: 10.1109/ISPA48434.2019.8966797.

D. Ciraolo, M. Fazio, R. S. Calabrò, M. Villari, and A. Celesti, “Facial expression recognition based on emotional artificial intelligence for tele-rehabilitation,” Biomed. Signal Process. Control, vol. 92, no. May 2023, p. 106096, 2024, doi: 10.1016/j.bspc.2024.106096.

C. N. Duong, K. G. Quach, I. Jalata, N. Le, and K. Luu, “MobiFace: A Lightweight Deep Learning Face Recognition on Mobile Devices,” 2019 IEEE 10th Int. Conf. Biometrics Theory, Appl. Syst. BTAS 2019, 2019, doi: 10.1109/BTAS46853.2019.9185981.

R. Ravi, S. V. Yadhukrishna, and R. Prithviraj, “A Face Expression Recognition Using CNN LBP,” Proc. 4th Int. Conf. Comput. Methodol. Commun. ICCMC 2020, no. Iccmc, pp. 684–689, 2020, doi: 10.1109/ICCMC48092.2020.ICCMC-000127.

J. Wang, R. Cao, P. N. Chakravarthula, X. Li, and S. Wang, “A critical period for developing face recognition,” Patterns, vol. 5, no. 2, p. 100895, 2024, doi: 10.1016/j.patter.2023.100895.

Y. Wang, M. Cao, Z. Fan, and S. Peng, “Learning to Detect 3D Facial Landmarks via Heatmap Regression with Graph Convolutional Network,” Proc. 36th AAAI Conf. Artif. Intell. AAAI 2022, vol. 36, pp. 2595–2603, 2022, doi: 10.1609/aaai.v36i3.20161.

D. Cakir and N. Arica, “Cascading CNNs for facial action unit detection,” Eng. Sci. Technol. an Int. J., vol. 47, no. October, p. 101553, 2023, doi: 10.1016/j.jestch.2023.101553.

S. Dwijayanti, R. R. Abdillah, H. Hikmarika, Hermawati, Z. Husin, and B. Y. Suprapto, “Facial Expression Recognition and Face Recognition Using a Convolutional Neural Network,” 2020 3rd Int. Semin. Res. Inf. Technol. Intell. Syst. ISRITI 2020, pp. 621–626, 2020, doi: 10.1109/ISRITI51436.2020.9315513.

C. Ashwini and V. Sellam, “An optimal model for identification and classification of corn leaf disease using hybrid 3D-CNN and LSTM,” Biomed. Signal Process. Control, vol. 92, no. February, p. 106089, 2024, doi: 10.1016/j.bspc.2024.106089.

K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Glob. Transitions Proc., vol. 3, no. 1, pp. 91–99, 2022, doi: 10.1016/j.gltp.2022.04.020.

N. Deshpande, F. Nunnari, and E. Avramidis, “Fine-tuning of Convolutional Neural Networks for the Recognition of Facial Expressions in Sign Language Video Samples,” 7th Work. Sign Lang. Transl. Avatar Technol. Junction Vis. Textual Challenges Perspect. SLTAT 2022 - as part Int. Conf. Lang. Resour. Eval. Lr. 2022 - Proc., no. June, pp. 29–38, 2022.

A. Nasayreh et al., “Jordanian banknote data recognition: A CNN-based approach with attention mechanism,” J. King Saud Univ. - Comput. Inf. Sci., vol. 36, no. 4, p. 102038, 2024, doi: 10.1016/j.jksuci.2024.102038.

D. Yang et al., “An efficient multi-task learning CNN for driver attention monitoring,” J. Syst. Archit., vol. 148, no. September 2023, p. 103085, 2024, doi: 10.1016/j.sysarc.2024.103085.

G. Rajeshkumar et al., “Smart office automation via faster R-CNN based face recognition and internet of things,” Meas. Sensors, vol. 27, no. November 2022, p. 100719, 2023, doi: 10.1016/j.measen.2023.100719.

C. R. Kumar, S. N, M. Priyadharshini, D. G. E, and K. R. M, “Face recognition using CNN and siamese network,” Meas. Sensors, vol. 27, no. March, p. 100800, 2023, doi: 10.1016/j.measen.2023.100800.

J. Wan et al., “Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network,” IEEE Trans. Cybern., vol. 53, no. 6, pp. 3546–3560, 2023, doi: 10.1109/TCYB.2021.3131569.

R. Verma, N. Bhardwaj, A. Bhavsar, and K. Krishan, “Towards facial recognition using likelihood ratio approach to facial landmark indices from images,” Forensic Sci. Int. Reports, vol. 5, no. October 2021, p. 100254, 2022, doi: 10.1016/j.fsir.2021.100254.

R. Salhab and W. Daher, “University Students’ Engagement in Mobile Learning,” Eur. J. Investig. Heal. Psychol. Educ., vol. 13, no. 1, pp. 202–216, 2023, doi: 10.3390/ejihpe13010016.

P. González-Gaspar et al., “Analixity: An open source, low-cost analysis system for the elevated plus maze test, based on computer vision techniques,” Behav. Processes, vol. 193, no. November, 2021, doi: 10.1016/j.beproc.2021.104539.

C. Hughes and A. Akkari, “Education needs a refocus so that all learners reach their full potential,” no. March, 2021.

K. A. Adamson and S. Prion, “Reliability: Measuring Internal Consistency Using Cronbach’s α,” Clin. Simul. Nurs., vol. 9, no. 5, pp. e179–e180, 2013, doi: 10.1016/j.ecns.2012.12.001.