Music Recommendation Based on Facial Expression Using Deep Learning

- Kurniawan - Faculty of Science Technology, Universitas Bina Darma, Palembang, Indonesia
Tri Basuki Kurniawan - Postgraduate Program, Universitas Bina Darma, Palembang, Indonesia
Deshinta Arrova Dewi - Faculty of Data Science and Information Technology, INTI International University, Nilai, Malaysia
Mohd Zaki Zakaria - Faculty of Computer & Mathematic Sciences, University Technology Mara, Malaysia
Zainuri Saringat - Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Johor, Malaysia
Ardian Firosha - Department of Information Technology, Politeknik Negeri Padang, Padang, Indonesia


Citation Format:



DOI: http://dx.doi.org/10.62527/joiv.9.1.3794

Abstract


Music's profound impact on human emotions is essential for creating personalized experiences in entertainment and therapeutic settings. This study introduces a cutting-edge music recommendation system that utilizes facial expression analysis to tailor music suggestions according to the user's emotional state. Our approach integrates a haar-cascade classifier for real-time face detection with a Convolutional Neural Network (CNN) that classifies emotions into seven distinct categories: happiness, sadness, anger, fear, disgust, surprise, and neutrality. This emotionally aware system recommends music tracks corresponding to the user's current emotional condition to enhance mood regulation and overall listener satisfaction. The effectiveness of our system was evaluated through rigorous testing, where the CNN model demonstrated a high degree of accuracy. Notably, the model achieved an overall accuracy of 84.44% in recognizing facial expressions. Precision, recall, and F1 scores consistently exceeded 84%, indicating robust performance across diverse emotional states. These results underscore the system's capability to accurately interpret and respond to complex emotional cues through tailored music suggestions. Integrating advanced deep learning techniques for face and emotion recognition enables our recommendation system to adapt dynamically to the user's emotional fluctuations. This responsiveness ensures a highly personalized music listening experience that reflects the user's feelings and potentially enhances their emotional well-being. By bridging the gap between static user profiles and the dynamic nature of human emotions, our system sets a new standard for personalized technology in music recommendation, promising significant improvements in user engagement and satisfaction.

Keywords


Facial expression recognition; music recommendation system; emotion analysis; process innovation

Full Text:

PDF

References


P. Agrahari, A. Singh Tanwar, B. Das, and P. Kunekar, “Musical therapy using facial expressions,” Int. Res. J. Eng. Technol., 2020.

N. Ahmad, “Impact of music on mood: Empirical investigation,” J. Educ. Pract., vol. 5, no. 21, 2015.

A. Jaya Mabel Rani, M. S. Nivetha, N. M. Jothi Swaroopan, and K. Hari Kumar, “Face emotion based music recommendation system using modified convolution neural network,” in 2023 IEEE Int. Conf. Res. Methodol. Knowl. Manag., Artif. Intell. Telecommun. Eng. (RMKMATE), 2023, doi: 10.1109/RMKMATE59243.2023.10368948.

A. Aziz and M. Fayyaz, “Comparison of content based and collaborative filtering in recommendation systems,” 2021.

M. Piórkowska and M. Wrobel, “Basic emotions,” in Encyclopedia of Personality and Individual Differences, 2017, doi: 10.1007/978-3-319-28099-8_495-1.

R. Zhang, S. Tu, and Z. Sun, “A hybrid music recommendation method based on music genres and collaborative filtering,” in 2022 IEEE Int. Conf. Dependable, Auton. Secure Comput., Pervasive Intell. Comput., Cloud Big Data Comput., Cyber Sci. Technol. Congr. (DASC/PiCom/CBDCom/CyberSciTech), 2022, pp. 1–6, doi:10.1109/DASC/PiCom/CBDCom/Cy55231.2022.9927924.

B. Yu, Q. Hu, L. Hao, and R. Li, “Music recommendation system based on collaborative filtering algorithm,” in 2023 IEEE Int. Conf. Sens., Electron. Comput. Eng. (ICSECE), 2023, pp. 308–313, doi:10.1109/ICSECE58870.2023.10263531.

A. Niyazov, E. Mikhailova, and O. Egorova, “Content-based music recommendation system,” in 2021 29th Conf. Open Innov. Assoc. (FRUCT), May 2021, pp. 274–279, doi:10.23919/FRUCT52173.2021.9435533.

M. Chemeque Rabel, “Content-based music recommendation system: A comparison of supervised machine learning models and music features,” M.S. thesis, School Electr. Eng. Comput. Sci. (EECS), KTH Royal Inst. Technol., Stockholm, Sweden, 2020. [Online]. Available: https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-288534.

A. Van Den Oord, S. Dieleman, and B. Schrauwen, “Deep content-based music recommendation,” in Adv. Neural Inf. Process. Syst., 2013, pp. 2643–2651.

B. Mahesh, “Machine learning algorithms—A review,” Int. J. Sci. Res., vol. 7, no. 3, pp. 1–6, 2018, doi: 10.21275/ART20203995.

J. Alzubi, A. Nayyar, and A. Kumar, “Machine learning from theory to algorithms: An overview,” J. Phys.: Conf. Ser., vol. 1142, no. 1, p. 012012, 2018, doi: 10.1088/1742-6596/1142/1/012012.

H. Meng, F. Yuan, Y. Wu, and T. Yan, “Facial expression recognition algorithm based on fusion of transformed multilevel features and improved weighted voting SVM,” Math. Probl. Eng., vol. 2021, pp. 1–12, 2021, doi: 10.1155/2021/6639598.

P. Chandrakala, B. S. Assistant Professor, and M. A. Kumar, “Real time face detection and face recognition using OpenCV and Python,” J. Eng. Sci., vol. 13, 2022.

A. Mishra, H. Chandrasekaran, B. Dhamecha, and R. Bhise, “Real time face detection,” Asian J. Res. Soc. Sci. Humanit., 2020.

M. Sakthimohan, G. Elizabeth Rani, M. Navaneethakrishnan, K. Janani, V. Nithva, and R. Pranav, “Detection and recognition of face using deep learning,” in 2023 Int. Conf. Intell. Syst. Commun., IoT Secur. (ICISCoIS), 2023, pp. 72–76, doi:10.1109/ICISCoIS56541.2023.10100435.

S. Begaj, A. O. Topal, and M. Ali, “Emotion recognition based on facial expressions using convolutional neural network (CNN),” in 2020 Int. Conf. Comput., Netw., Telecommun. Eng. Sci. Appl. (CoNTESA), Dec. 2020, pp. 58–63, doi: 10.1109/CoNTESA50436.2020.9302866.

R. Ravi, S. V. Yadhukrishna, and R. Prithviraj, “A face expression recognition using CNN LBP,” in 2020 4th Int. Conf. Comput. Methodol. Commun. (ICCMC), 2020, pp. 684–689, doi:10.1109/ICCMC48092.2020.ICCMC-000127.

G. M. Foody and A. Mathur, “A relative evaluation of multiclass image classification by support vector machines,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 6, pp. 1335–1343, Jun. 2004, doi:10.1109/TGRS.2004.827257.

R. K. Mishra, G. Y. S. Reddy, and H. Pathak, “The understanding of deep learning: A comprehensive review,” Math. Probl. Eng., vol. 2021, pp. 1–16, 2021, doi: 10.1155/2021/5548884.

A. Mathew, P. Amudha, and S. Sivakumari, “Deep learning techniques: An overview,” in Adv. Intell. Syst. Comput., vol. 1141, 2021, pp. 599–608, doi: 10.1007/978-981-15-3383-9_54.

S. I. Mohammad, N. S. G. Abdelrasheed, A. S. Minasova, A. Vasudevan, and N. Shavkatov, “A gauge into emotional intelligence enhancement in CALL and the effects on oral skills, personal best goals, and self-efficacy among EFL learners,” Comput.-Assist. Lang. Learn. Electron. J., vol. 25, no. 4, pp. 552–577, 2024.