On Determining the Most Effective Technique Available in Software Testing

Authors: Qasim Zafar, Matthew Anderson, Esteban Garcia, Steven Drager

Abstract:

Software failures can present an enormous detriment to people's lives and cost millions of dollars to repair when they are unexpectedly encountered in the wild. Despite a significant portion of the software development lifecycle and resources are dedicated to testing, software failures are a relatively frequent occurrence. Nevertheless, the evaluation of testing effectiveness remains at the forefront of ensuring high-quality software and software metrics play a critical role in providing valuable insights into quantifiable objectives to assess the level of assurance and confidence in the system. As the selection of appropriate metrics can be an arduous process, the goal of this paper is to shed light on the significance of software metrics by examining a range of testing techniques and metrics as well as identifying key areas for improvement. In doing so, this paper presents a method to compare the effectiveness of testing techniques with heterogeneous output metrics. Additionally, through this investigation, readers will gain a deeper understanding of how metrics can help to drive informed decision-making on delivering high-quality software and facilitate continuous improvement in testing practices.

Keywords: Software testing, software metrics, testing effectiveness, black box testing, random testing, adaptive random testing, combinatorial testing, fuzz testing, equivalence partition, boundary value analysis, white box testings.

References:

[1] W. Dijkstra, "Notes on Structured Programming," Technological University Eindhoven T.H. Report 70-WSK-03, Second edition, April 1970.
[2] “ISO/IEC/IEEE International Standard - Systems and software engineering--Vocabulary," in ISO/IEC/IEEE 24765:2017(E), vol., no., pp.1-541.
[3] V. Vukovic, J. Djurkovic, M. Sakal, & L. Rakovic, “An Empirical Investigation of Software Testing Methods and Techniques in the Province of Vojvodina,” Tehnicki Vjesnik-Technical Gazette, 2020.
[4] B.W. Boehm, “The High Cost of Software”, in Practical Strategies for Developing Large Software Systems, E. Horowitz (editor), Addison-Wesley, Reading, MA, 1975.
[5] DoD Instruction 5200.44: “Protection of Mission Critical Functions to Achieve Trusted Systems and Networks (TSN),” Nov 5, 2012.
[6] B. Beizer, Software Testing Techniques. London: International Thompson Computer Press, 1990.
[7] T. Y. Chen, F. Kuo, H. Liu, & W. C. Wong, “Does Adaptive Random Testing Deliver a Higher Confidence than Random Testing?”, the Eighth International Conference on Quality Software, 2008.
[8] D. R. Kuhn and D. R. Wallace, “Software fault interactions and implications for software testing,” IEEE Trans. Softw. Eng., vol. 30, no. 6, pp. 418–421, Jun. 2004.
[9] Y. Lei, R. N. Kacker, & D. R. Kuhn, "ACTS: A combinatorial test generation tool" IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), 2013.
[10] D. R. Kuhn, R. N. Kacker and Y. Lei. “Combinatorial coverage as an aspect of test quality,” 2015.
[11] H. Wu, C. Nie, J. Petke, Y. Jia and M. Harman, "An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing" in IEEE Transactions on Software Engineering, vol. 46, no. 03, pp. 302-320, 2020.
[12] L. S. Ghandehari, J. Czerwonka, Y. Lei, S. Shafiee, R. Kacker, and R. Kuhn, “An empirical comparison of combinatorial and random testing,” in Proc. IEEE Int. Conf. Softw. Testing Verification Validation Workshops, 2014, pp. 68–77.
[13] R. Kuhn, M. S. Raunak and R. Kacker, "Combinatorial Coverage for Assured Autonomy," IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Charlotte, NC, USA, 2022, pp. 357-358.
[14] Y. Singh, Software Testing. Cambridge University Press, 2012.
[15] A. Takanen, J. D. DeMott, & C. Miller, Fuzzing for Software Security Testing and Quality Assurance, 2018.
[16] J. W. Hollén, P. S. Zacarias. “Exploring Code Coverage in Software Testing and its Correlation with Software Quality; A Systematic Literature Review,” 2015.