The Effect of Program Type on Mutation Testing: Comparative Study

Authors: B. Falah, N. E. Abakouy

Abstract:

Due to its high computational cost, mutation testing has been neglected by researchers. Recently, many cost and mutants’ reduction techniques have been developed, improved, and experimented, but few of them has relied the possibility of reducing the cost of mutation testing on the program type of the application under test. This paper is a comparative study between four operators’ selection techniques (mutants sampling, class level operators, method level operators, and all operators’ selection) based on the program code type of each application under test. It aims at finding an alternative approach to reveal the effect of code type on mutation testing score. The result of our experiment shows that the program code type can affect the mutation score and that the programs using polymorphism are best suited to be tested with mutation testing.

Keywords: Equivalent mutant, killed mutant, mutation score, mutation testing, program code type.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1339053

References:

[1] B. Falah. “An Approach to Regression Test Selection Based on Complexity Metrics”, Scholar’s Press, ISBN-10: 3639518683, ISBN-13: 978-3639518689, Pages: 136, October 28, 2013.
[2] B. Falah, K. Magel. “Test Case Selection Based on a Spectrum of Complexity Metrics”. Ph.D. Dissertation. North Dakota State University, Fargo, ND, USA. Advisor(s) Kenneth Magel. 978-1-124-62588-1.
[3] B. Falah, K. Magel, O. El Ariss. “A Complex Based Regression Test Selection Strategy”, Computer Science & Engineering: An International Journal (CSEIJ), Vol.2, No.5, October 2012.
[4] B. Beizer. Software testing techniques (2nd ed.). ISBN:0-442-20672-0, Van Nostrand Reinhold Co.,New York, NY, USA, 1990.
[5] R. N. Charette. Why software fails, Spectrum IEEE, Volume 42, Issue 9, Pages 42-49, September, 2005.
[6] M. Plol, M. Piattini, and I. G. Rogriguez, “Decreasing the Cost of Mutaion Teting with the Second Order Mutants,” Software Testing, Verification and Reliability, vol. 19, pp. 111-131, June 2009.
[7] Y. Jia and M. Harman, An analysis and Survey of the Development of Mutation Testing, IEEE transactions on Software Engineering, Volume 37, Issue 5, pages 649-678, 2011.
[8] P. R. Mateo, M. P. Usaola, Mutant Execution Cost Reduction through MUSIC (MUtant Schema Improved with extra Code), ICST '12 Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pages 664-672.
[9] B. H. Smith, L. Williams. Should Software Testers Use Mutation Analysis to Augment a Test Set? Journal of Systems and Software, Volume 82 Issue 11, pages 1819-1832, Elsevier Science Inc. New York, NY, USA, November, 2009.
[10] A. P. Mathur and W. E. Wong, “An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria,” Purdue University, West Lafayette, Indiana, Technique Report, 1993.
[11] W. E. Wong, “On Mutation and Data Flow,” PhD Thesis, Purdue University, West Lafayette, Indiana, 1993.
[12] M. Sahinoglu and E. H. Spafford, “A Bayes Sequential Statistical Procedure for Approving Software Products,” in Proceedings of the IFIP Conference on Approving Software Products (ASP’90). Garmisch Partenkirchen, Germany: Elsevier Science, pp. 43–56, September 1990.
[13] Y-S. Ma, J. Offutt, and Y. R. Kwon, MuJava: An Automated Class Mutation System, Journal of Software Testing, Verification and Reliability, 2005, pages. 97 - 133.
[14] A. J. Offutt, Y. S. Ma, and Y. R. Kwon, “An Experimental Mutation System for Java,” ACM SIGSOFT Software Engineering Notes (SECTION: Workshop on empirical research in software testing papers), 29(5), Sep. 2004.
[15] Y. S. Ma, Y. R. Kwon, and J. Offutt, “Inter-Class Mutation Operators for Java,” In Proc. of ISSRE 2002, pages 352–363, IEEE Computer Society Press, Nov. 2002.
[16] A. Jefferson Offutt and K. N. King, “A Fortran Language System for Mutation-Based Software Testing,” Software Practice and Experience, 21(7):686–718, July 1991.
[17] E. S. Mresa, L. Bottaci. (1999). Efficiency of mutation operators and selective mutation strategies: An empirical study, Journal of Software Testing, Verification and Reliability, Volume 9, Issue 4, pages 205–232, December 1999.
[18] Y-S. Ma and J. Offut, Description of Method Level Mutation Operators for Java, December 2005, http://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf
[19] Y-S. Ma and J. Offutt, Description of Class Mutation Operators for Java, December 2005, http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf
[20] Muclipse: An Open Source Mutation Testing Plug-in for Eclipse, July 2008, http://muclipse.sourceforge.net
[21] Bernhard J. M. Grun, David Schuler, and Andreas Zeller. The Impact of Equivalent Mutants. Proc. 2nd International Workshop on Multi-Core Software Engineering (IWMSE), Pages 49-55, May 2009.