The Use of Hyperparameter Tuning in Model Classification: A Scientific Work Area Identification

Nadya Alinda Rahmi - Universitas Putra Indonesia YPTK, Padang, Padang, 25221, Indonesia
Sarjon Defit - Universitas Putra Indonesia YPTK, Padang, Padang, 25221, Indonesia
- Okfalisa - Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru, 28293 Indonesia


Citation Format:



DOI: http://dx.doi.org/10.62527/joiv.8.4.3092

Abstract


This research aims to investigate the effectiveness of hyperparameter tuning, particularly using Optuna, in enhancing the classification performance of machine learning models on scientific work reviews. The study focuses on automating the classification of academic papers into eight distinct fields: decision support systems, information technology, data science, technology education, artificial intelligence, expert systems, image processing, and information systems. The research dataset comprises reviews of scientific papers ranging from 150 to 500 words, collected from the repository of Universitas Putra Indonesia YPTK Padang. The classification process involved the application of the TF-IDF method for feature extraction, followed using various machine learning algorithms including SVM, MNB, KNN, and RF, with and without the integration of SMOTE for data balancing and Optuna for hyperparameter optimization. The results show that combining SMOTE with Optuna significantly improves the accuracy, precision, recall, and F1-score of the models, with the SVM algorithm achieving the highest accuracy at 90%. Additionally, the research explored the effectiveness of ensemble methods, revealing that hard voting combined with SMOTE and Optuna provided substantial improvements in classification performance. These findings underscore the importance of hyperparameter tuning and data balancing in optimizing machine learning models for text classification tasks. The implications of this research are broad, suggesting that the methodologies developed can be applied to various text classification tasks in different domains. Future research should consider exploring other hyperparameter tuning techniques and ensemble methods to further enhance model performance across diverse datasets.

Keywords


Machine Learning, Voting, Optuna, SMOTE, Scientific Work Area.

Full Text:

PDF

References


J. Wong, T. Manderson, M. Abrahamowicz, D. L. Buckeridge, and R. Tamblyn, “Can Hyperparameter Tuning Improve the Performance of a Super Learner?,” Epidemiology, vol. 30, no. 4, pp. 521–531, Jul. 2019, doi: 10.1097/ede.0000000000001027.

O. A. Montesinos López, A. Montesinos López, and J. Crossa, Multivariate Statistical Machine Learning Methods for Genomic Prediction. Springer International Publishing, 2022. doi: 10.1007/978-3-030-89010-0.

L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, Nov. 2020, doi: 10.1016/j.neucom.2020.07.061.

J.-P. Lai, Y.-L. Lin, H.-C. Lin, C.-Y. Shih, Y.-P. Wang, and P.-F. Pai, “Tree-Based Machine Learning Models with Optuna in Predicting Impedance Values for Circuit Analysis,” Micromachines, vol. 14, no. 2, p. 265, Jan. 2023, doi: 10.3390/mi14020265.

S. Hadianti, “Optimization of The Machine Learning Approach using Optuna in Heart Disease Prediction,” Journal Medical Informatics Technology, pp. 59–64, Sep. 2023, doi: 10.37034/medinftech.v1i3.15.

A. M. Vincent and P. Jidesh, “An improved hyperparameter optimization framework for AutoML systems using evolutionary algorithms,” Scientific Reports, vol. 13, no. 1, Mar. 2023, doi:10.1038/s41598-023-32027-3.

M. K. Anam, T. A. Fitri, A. Agustin, L. Lusiana, M. B. Firdaus, and A. T. Nurhuda, “Sentiment Analysis for Online Learning using The Lexicon-Based Method and The Support Vector Machine Algorithm,” Ilkom Jurnal Ilmiah, vol. 15, no. 2, pp. 290–302, Aug. 2023, doi:10.33096/ilkom.v15i2.1590.290-302.

J. Wu, X. Y. Chen, H. Zhang, L. D. Xiong, H. Lei, and S. H. Deng, “Hyperparameter optimization for machine learning models based on Bayesian optimization,” Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, Mar. 2019, doi:10.11989/JEST.1674-862X.80904120.

K. E. Hoque and H. Aljamaan, “Impact of Hyperparameter Tuning on Machine Learning Models in Stock Price Forecasting,” IEEE Access, vol. 9, pp. 163815–163830, 2021, doi: 10.1109/access.2021.3134138.

E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, “Hyperparameter Tuning for Machine Learning Algorithms Used for Arabic Sentiment Analysis,” Informatics, vol. 8, no. 4, p. 79, Nov. 2021, doi:10.3390/informatics8040079.

Y. Ali, E. Awwad, M. Al-Razgan, and A. Maarouf, “Hyperparameter Search for Machine Learning Algorithms for Optimizing the Computational Complexity,” Processes, vol. 11, no. 2, p. 349, Jan. 2023, doi: 10.3390/pr11020349.

A. Tikaningsih, P. Lestari, A. Nurhopipah, I. Tahyudin, E. Winarto, and N. Hassa, “Optuna Based Hyperparameter Tuning for Improving the Performance Prediction Mortality and Hospital Length of Stay for Stroke Patients,” Telematika, vol. 17, no. 1, pp. 1–16, Feb. 2024, doi:10.35671/telematika.v17i1.2816.

M. Arifin, W. Widowati, and F. Farikhin, “Optimization of Hyperparameters in Machine Learning for Enhancing Predictions of Student Academic Performance,” Ingénierie des systèmes d information, vol. 28, no. 3, pp. 575–582, Jun. 2023, doi:10.18280/isi.280305.

G. Ramadhan, “Comparative Analysis of ADASYN-SVM and SMOTE-SVM Methods on the Detection of Type 2 Diabetes Mellitus,” Scientific Journal of Informatics, vol. 8, no. 2, pp. 276–282, Nov. 2021, doi: 10.15294/sji.v8i2.32484.

R. Shahid et al., “Predicting Customer Sentiment in Social Media Interactions: Analyzing Amazon Help Twitter Conversations Using Machine Learning,” International Journal of Advanced Science Computing and Engineering, vol. 6, no. 2, pp. 52–56, Jul. 2024, doi:10.62527/ijasce.6.2.211.

A. Mohammed and R. Kora, “A comprehensive review on ensemble deep learning: Opportunities and challenges,” Journal of King Saud University - Computer and Information Sciences, vol. 35, no. 2, pp. 757–774, Feb. 2023, doi: 10.1016/j.jksuci.2023.01.014.

R. Arifudin, D. Indra Wijaya, B. Warsito, and A. Wibowo, “Voting Classifier Technique and Count Vectorizer with N-gram to Identify Hate Speech and Abusive Tweets in Indonesian,” Scientific Journal of Informatics, vol. 10, no. 4, p. 469, 2023, doi: 10.15294/sji.v10i4.46633.

S. Karlos, G. Kostopoulos, and S. Kotsiantis, “A Soft-Voting Ensemble Based Co-Training Scheme Using Static Selection for Binary Classification Problems,” Algorithms, vol. 13, no. 1, p. 26, Jan. 2020, doi: 10.3390/a13010026.

W. Ccoya and E. Pinto, "Comparative analysis of libraries for the sentimental analysis," 2023. [Online]. Available: https://arxiv.org/abs/2307.14311. [Accessed: Sep. 19, 2024].

T. A. Assegie, “An optimized K-Nearest Neighbor based breast cancer detection,” Journal of Robotics and Control (JRC), vol. 2, no. 3, 2021, doi: 10.18196/jrc.2363.

D. C. E. Saputra, A. Ma’arif, and K. Sunat, “Optimizing Predictive Performance: Hyperparameter Tuning in Stacked Multi-Kernel Support Vector Machine Random Forest Models for Diabetes Identification,” Journal of Robotics and Control (JRC), vol. 4, no. 6, pp. 896–904, Jan. 2024, doi: 10.18196/jrc.v4i6.20898.

J. Zhang, Y. Li, F. Shen, Y. He, H. Tan, and Y. He, “Hierarchical text classification with multi-label contrastive learning and KNN,” Neurocomputing, vol. 577, p. 127323, Apr. 2024, doi:10.1016/j.neucom.2024.127323.

A. Zamsuri, S. Defit, and G. W. Nurcahyo, “Classification of Multiple Emotions in Indonesian Text Using The K-Nearest Neighbor Method,” Journal of Applied Engineering and Technological Science (JAETS), vol. 4, no. 2, pp. 1012–1021, Jun. 2023, doi: 10.37385/jaets.v4i2.1964.

M. Y. Baihaqi, E. Halawa, R. A. S. Syah, A. Nurrahma, and W. Wijaya, “Emotion Classification in Indonesian Language: A CNN Approach with Hyperband Tuning,” Jurnal Buana Informatika, vol. 14, no. 02, pp. 137–146, Oct. 2023, doi: 10.24002/jbi.v14i02.7558.

M. K. Anam, M. I. Mahendra, W. Agustin, R. Rahmaddeni, and N. Nurjayadi, “Framework for Analyzing Netizen Opinions on BPJS Using Sentiment Analysis and Social Network Analysis (SNA),” Intensif: Jurnal Ilmiah Penelitian dan Penerapan Teknologi Sistem Informasi, vol. 6, no. 1, pp. 11–28, Feb. 2022, doi:10.29407/intensif.v6i1.15870.

A. Handayanto, P. D. Yulianti, and K. Latifah, “Covid-19 Impact on Divorce Case Using Sentiment Analysis,” SAINTEKS, vol. 18, no. 2, pp. 145–154, 2021.

A. Majid, D. Nugraha, and F. D. Adhinata, “Sentiment Analysis on Tiktok Application Reviews Using Natural Language Processing Approach,” Journal of Embedded System Security and Intelligent System, vol. 4, no. 1, pp. 32–38, 2023, doi: 10.26858/jessi.v4i1.41897.

K. K. Agustiningsih, E. Utami, and M. A. Alsyaibani, “Sentiment Analysis of COVID-19 Vaccines in Indonesia on Twitter Using Pre-Trained and Self-Training Word Embeddings,” Jurnal Ilmu Komputer dan Informasi, vol. 15, no. 1, pp. 39–46, Feb. 2022, doi:10.21609/jiki.v15i1.1044.

E. Escobar-Linero, F. Luna-Perejón, L. Muñoz-Saavedra, J. L. Sevillano, and M. Domínguez-Morales, “On the feature extraction process in machine learning. An experimental study about guided versus non-guided process in falling detection systems,” Engineering Applications of Artificial Intelligence, vol. 114, p. 105170, Sep. 2022, doi: 10.1016/j.engappai.2022.105170.

Z. Zhang, Y. Lei, J. Xu, X. Mao, and X. chang, “TFIDF-FL: Localizing Faults Using Term Frequency-Inverse Document Frequency and Deep Learning,” IEICE Transactions on Information and Systems, vol. E102.D, no. 9, pp. 1860–1864, Sep. 2019, doi:10.1587/transinf.2018edl8237.

Okfalisa, L. Handayani, D. J. P., M. Affandes, M. Fauzi, dan Saktioto, “Coronary Hearth Disease Using Support Vector Machine,” J. Eng. Sci. Technol., vol. 16, no. 2, pp. 1370–1385, 2021.