Test Case Prioritization for Software Product Line: A Systematic Mapping Study

Muhammad Idham - Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia
Shahliza Abd Halim - Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia
Dayang Norhayati Abang Jawawi - Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia
Zalmiyah Zakaria - Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia
Aldo Erianda - Politeknik Negeri Padang, Padang, West Sumatera, Indonesia
Nachnoer Arss - Faculty of Engineering, Universitas Negeri Padang, Padang, Indonesia


Citation Format:



DOI: http://dx.doi.org/10.30630/joiv.7.3-2.1340

Abstract


Combinatorial explosion remains a common issue in testing. Due to the vast number of product variants, the number of test cases required for comprehensive coverage has significantly increased. One of the techniques to efficiently tackle this problem is prioritizing the test suites using a regression testing method. However, there is a lack of comprehensive reviews focusing on test case prioritization in SPLs. To address this research gap, this paper proposed a systematic mapping study to observe the extent of test case prioritization usage in Software Product Line Testing. The study aims to classify various aspects of SPL-TCP (Software Product Line – Test Case Prioritization), including methods, criteria, measurements, constraints, empirical studies, and domains. Over the last ten years, a thorough investigation uncovered twenty-four primary studies, consisting of 12 journal articles and 12 conference papers, all related to Test Case Prioritization for SPLs. This systematic mapping study presents a comprehensive classification of the different approaches to test case prioritization for Software Product Lines. This classification can be valuable in identifying the most suitable strategies to address specific challenges and serves as a guide for future research works. In conclusion, this mapping study systematically classifies different approaches to test case prioritization in Software Product Lines. The results of this study can serve as a valuable resource for addressing challenges in SPL testing and provide insights for future research.

Keywords


Software testing; software product line; system product line; test case prioritization; systematic mapping study

Full Text:

PDF

References


I. D. C. Machado, J. D. McGregor, Y. C. Cavalcanti, and E. S. De Almeida, “On strategies for testing software product lines: A systematic literature review,†Inf Softw Technol, vol. 56, no. 10, pp. 1183–1199, 2014, doi: 10.1016/j.infsof.2014.04.002.

S. Yoo and M. Harman, “Regression testing minimization, selection and prioritization: a survey,†SOFTWARE TESTING, VERIFICATION AND RELIABILITY, pp. 1–22, 2010, doi: 10.1002/stvr.

E. J. Adwan and B. A. Alsaeed, “Cloud Computing Adoption in the Financial Banking Sector- A Systematic Literature Review (2011-2021),†International Journal of Advanced Science Computing and Engineering, vol. 4, no. 1, pp. 48–55, Apr. 2022, doi: 10.30630/ijasce.4.1.73.

W. R. Malatji, R. VanEck, and T. Zuva, “A Systematic Review of the Adoption of eHealth Cloud-based Technology Applications during COVID-19,†International Journal of Advanced Science Computing and Engineering, vol. 3, no. 2, pp. 53–64, Oct. 2021, doi: 10.30630/ijasce.3.2.44.

J. Lee, S. Kang, and P. Jung, “Test coverage criteria for software product line testing: Systematic literature review,†Inf Softw Technol, vol. 122, no. December 2019, p. 106272, 2020, doi: 10.1016/j.infsof.2020.106272.

R. S. Wahono, “A Systematic Literature Review of Software Defect Prediction: Research Trends, Datasets, Methods and Frameworks,†Journal of Software Engineering, vol. 1, no. 1, pp. 1–16, 2015.

P. A. Da Mota Silveira Neto, I. Do Carmo MacHado, J. D. McGregor, E. S. De Almeida, and S. R. De Lemos Meira, “A systematic mapping study of software product lines testing,†Inf Softw Technol, vol. 53, no. 5, pp. 407–423, 2011, doi: 10.1016/j.infsof.2010.12.003.

M. Z. Sahid, A. Azim, A. Ghani, and S. Baharom, “Combinatorial Interaction Testing of Software Product Lines : A Mapping Study,†2016, doi: 10.3844/jcssp.2016.379.398.

M. Khatibsyarbini, M. A. Isa, D. N. A. Jawawi, and R. Tumeng, “Test case prioritization approaches in regression testing: A systematic literature review,†Information and Software Technology, vol. 93. Elsevier B.V., pp. 74–93, Jan. 01, 2018. doi: 10.1016/j.infsof.2017.08.014.

Y. Lou, J. Chen, L. Zhang, and D. Hao, A Survey on Regression Test-Case Prioritization, 1st ed., vol. 113. Elsevier Inc., 2019. doi: 10.1016/bs.adcom.2018.10.001.

M. Sahak, S. A. Halim, D. N. A. Jawawi, and M. A. Isa, “Evaluation of software product line test case prioritization technique,†Int J Adv Sci Eng Inf Technol, vol. 7, no. 4–2 Special Issue, pp. 1601–1608, 2017, doi: 10.18517/ijaseit.7.4-2.3403.

A. B. Sanchez, S. Segura, and A. Ruiz-Cortes, “A Comparison of test case prioritization criteria for software product lines,†Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation, ICST 2014, pp. 41–50, 2014, doi: 10.1109/ICST.2014.15.

S. Kumar and Rajkumar, “Test case prioritization techniques for software product line: A survey,†Proceeding - IEEE International Conference on Computing, Communication and Automation, ICCCA 2016, pp. 884–889, 2017, doi: 10.1109/CCAA.2016.7813841.

S. Kitchenham, B & Charters, “Guidelines for performing systematic literature reviews in software engineering,†Technical report, Ver. 2.3 EBSE Technical Report. EBSE, vol. 1, no. January 2007, pp. 1–54, 2007, [Online]. Available: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.471&rep=rep1&type=pdf

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic Mapping Studies in Software Engineering,†in BASE - Revista de Administração e Contabilidade da Unisinos, Jun. 2008, pp. 1–10. doi: 10.14236/ewic/EASE2008.8.

M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences, vol. 29, no. 2. Wiley, 2006. doi: 10.1002/9780470754887.

A. Bajaj and O. P. Sangwan, “A Systematic Literature Review of Test Case Prioritization Using Genetic Algorithms,†IEEE Access, vol. 7, pp. 126355–126375, 2019, doi: 10.1109/ACCESS.2019.2938260.

R. Mukherjee and K. S. Patnaik, “A survey on different approaches for software test case prioritization,†Journal of King Saud University - Computer and Information Sciences, vol. 33, no. 9, pp. 1041–1054, 2021, doi: 10.1016/j.jksuci.2018.09.005.

M. Sahak, D. N. A. Jawawi, and S. A. Halim, “An Experiment of Different Similarity Measures on Test Case Prioritization for Software Product Lines,†Journal of Telecommunication, Electronic and Computer Engineering, vol. 9, no. 3-4 Special Issue, pp. 177–185, 2017.

C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. Le Traon, “Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines,†IEEE Transactions on Software Engineering, vol. 40, no. 7, pp. 650–670, 2014, doi: 10.1109/TSE.2014.2327020.

R. A. Sulaiman, D. N. A. Jawawi, and S. A. Halim, “A dissimilarity with dice-jaro-winkler test case prioritization approach for model-based testing in software product line,†KSII Transactions on Internet and Information Systems, vol. 15, no. 3, pp. 932–951, 2021, doi: 10.3837/tiis.2021.03.007.

U. Markiegi, A. Arrieta, L. Etxeberria, and G. Sagardui, Dynamic test prioritization of product lines: An application on configurable simulation models, no. 0123456789. Springer US, 2021. doi: 10.1007/s11219-021-09571-0.

S. Kumar, M. Mittal, and V. K. Yadav, “Cost¬-effective product prioritisation technique for software product line testing,†Engineering Systems Modelling and Simulation, vol. x, no. x.

J. A. P. Lima, W. D. F. Mendonça, S. R. Vergilio, and W. K. G. Assunção, “Learning-based prioritization of test cases in continuous integration of highly-configurable software,†in Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A, New York, NY, USA: ACM, Oct. 2020, pp. 1–11. doi: 10.1145/3382025.3414967.

S. Kumar, R. Kumar, and M. Mittal, “A hybrid approach to perform test case prioritisation and reduction for software product line testing,†International Journal of Vehicle Autonomous Systems, vol. 15, no. 3–4, pp. 197–224, 2020, doi: 10.1504/IJVAS.2020.116439.

I. Hajri, A. Goknil, F. Pastore, and L. C. Briand, “Automating system test case classification and prioritization for use case-driven testing in product lines,†Empir Softw Eng, vol. 25, no. 5, pp. 3711–3769, 2020, doi: 10.1007/s10664-020-09853-4.

S. Kumar, Rajkumar, and M. Rani, “Collaborative Filtering-based Test Case Prioritization and Reduction for Software Product-Line Testing,†IEEE Region 10 Annual International Conference, Proceedings/TENCON, vol. 2019-Octob, pp. 498–503, 2019, doi: 10.1109/TENCON.2019.8929705.

S. A. Halim, D. N. A. Jawawi, and M. Sahak, “Similarity Distance Measure and Prioritization Algorithm for Test Case Prioritization in Software Product Line Testing,†Journal of Information and Communication Technology, vol. 1, no. 1, pp. 57–75, 2019.

A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria, “Search-Based test case prioritization for simulation-Based testing of cyber-Physical system product lines,†Journal of Systems and Software, vol. 149, pp. 1–34, Mar. 2019, doi: 10.1016/j.jss.2018.09.055.

M. Al-Hajjaji, T. Thüm, M. Lochau, J. Meinicke, and G. Saake, “Effective product-line testing using similarity-based product prioritization,†Softw Syst Model, vol. 18, no. 1, pp. 499–521, 2019, doi: 10.1007/s10270-016-0569-2.

H. Akimoto, Y. Isogami, T. Kitamura, N. Noda, and T. Kishi, “A Prioritization Method for SPL Pairwise Testing Based on User Profiles,†Proceedings - Asia-Pacific Software Engineering Conference, APSEC, vol. 2019-Decem, pp. 118–125, 2019, doi: 10.1109/APSEC48747.2019.00025.

X. Devroey et al., “Statistical prioritization for software product line testing: an experience report,†Softw Syst Model, vol. 16, no. 1, pp. 153–171, 2017, doi: 10.1007/s10270-015-0479-8.

M. Al-Hajjaji, S. Lity, R. Lachmann, T. Thüm, I. Schaefer, and G. Saake, “Delta-Oriented Product Prioritization for Similarity-Based Product-Line Testing,†Proceedings - 2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design, VACE 2017, pp. 34–40, 2017, doi: 10.1109/VACE.2017.8.

M. Al-Hajjaji, J. Kruger, S. Schulze, T. Leich, and G. Saake, “Efficient product-line testing using cluster-based product prioritization,†Proceedings - 2017 IEEE/ACM 12th International Workshop on Automation of Software Testing, AST 2017, pp. 16–22, 2017, doi: 10.1109/AST.2017.7.

Z. Akbari, S. Khoshnevis, and M. Mohsenzadeh, “A Method for Prioritizing Integration Testing in Software Product Lines Based on Feature Model,†International Journal of Software Engineering and Knowledge Engineering, vol. 27, no. 4, pp. 575–600, 2017, doi: 10.1142/S0218194017500218.

J. A. Parejo, A. B. Sánchez, S. Segura, A. Ruiz-Cortés, R. E. Lopez-Herrejon, and A. Egyed, “Multi-objective test case prioritization in highly configurable systems: A case study,†Journal of Systems and Software, vol. 122, pp. 287–310, 2016, doi: 10.1016/j.jss.2016.09.045.

R. Lachmann, S. Lity, M. Al-Hajjaji, F. Fürchtegott, and I. Schaefer, “Fine-grained test case prioritization for integration testing of delta-oriented software product lines,†FOSD 2016 - Proceedings of the 7th International Workshop on Feature-Oriented Software Development, co-located with SPLASH 2016, vol. 1, no. 212, pp. 1–10, 2016, doi: 10.1145/3001867.3001868.

R. Lachmann, S. Lity, S. Lischke, S. Beddig, S. Schulze, and I. Schaefer, “Delta-oriented test case prioritization for integration testing of software product lines,†Proceedings of the 19th International Conference on Software Product Line, pp. 81–90, 2015, doi: 10.1145/2791060.2791073.

H. Wu, C. Nie, and F. C. Kuo, “Test suite prioritization by switching cost,†Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014, pp. 133–142, 2014, doi: 10.1109/ICSTW.2014.15.

S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan, and M. Liaaen, “Multi-objective test prioritization in software product line testing: An industrial case study,†ACM International Conference Proceeding Series, vol. 1, pp. 32–41, 2014, doi: 10.1145/2648511.2648515.

R. E. Lopez-Herrejon, J. Ferrer, F. Chicano, E. N. Haslinger, A. Egyed, and E. Alba, “A parallel evolutionary algorithm for prioritized pairwise testing of software product lines,†GECCO 2014 - Proceedings of the 2014 Genetic and Evolutionary Computation Conference, pp. 1255–1262, 2014, doi: 10.1145/2576768.2598305.

X. Devroey et al., “Towards Statistical Prioritization for Software Product Lines Testing Categories and Subject Descriptors,†VaMoS ’14 Proceedings of the Eighth International Workshop on Variability Modelling of Software-Intensive Systems, p. Article No. 10, 2014.

M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake, “Similarity-based prioritization in software product-line testing,†ACM International Conference Proceeding Series, vol. 1, pp. 197–206, 2014, doi: 10.1145/2648511.2648532.