An Experiment for Assessment of a “Functional Scenario-based“ Test Case Generation Method

Authors: Cencen Li, Shaoying Liu, Shin Nakajima

Abstract:

Specification-based testing enables us to detect errors in the implementation of functions defined in given specifications. Its effectiveness in achieving high path coverage and efficiency in generating test cases are always major concerns of testers. The automatic test cases generation approach based on formal specifications proposed by Liu and Nakajima is aimed at ensuring high effectiveness and efficiency, but this approach has not been empirically assessed. In this paper, we present an experiment for assessing Liu-s testing approach. The result indicates that this testing approach may not be effective in some circumstances. We discuss the result, analyse the specific causes for the ineffectiveness, and describe some suggestions for improvement.

Keywords: experiment, functional scenario, specification-based, testing.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1333434

References:

[1] Shaoying Liu, and Shin Nakajima. A Decompositional Approach to Automatic Test Case Generation Based on Formal Specification. Fourth IEEE International Conference on Secure Software Integration and Reliability Improvement, pages 147-155, 2010.
[2] J. Dick, and A. Faivre. Automating the Generation and Sequencing of Test Cases from Model-based Specifications. In Proceedings of FME -93: Industrial-Strength Formal Methods, pages 268-284, Odense, Denmark, 1993. Springer-Verlag Lecture Notes in Computer Science Volume 670.
[3] Shaoying Liu. Formal Engineering for Industrial Software Development Using the SOFL Method. Springer-Verlag, ISBN 3-540-20602-7, 2004.
[4] M. C. Gaudel and P. Le Gall. Testing Data Types Implementation from Algebraic Specifications. In R. Hierons, J. Bowen, and M. Harman, editors, Formal Methods and Testing, pages 209-239. LNCS 4949, Springer-Verlag, 2008.
[5] M. Broy, B. Jonsson, J. -P. Katoen, M. Leucker, and A. Pretschner (eds.). Model-based Testing of Reactive Systems. LNCS 3472, Springer-Verlag, 2005.
[6] Yoonsik Cheon, and Carmen Avila. Automating Java Program Testing Using OCL and AspectJ. 7th International Conference on Information Technology, pages 1020-1025, 2010.
[7] Anton Michlmayr, Pascal Fenkam, and Schahram Dustdar. Specification- Based Unit Testing of Publich/Subscribe Applications. Proceedings of the 26th IEEE International Conference on Distributed Computing Systems Workshops, pages 34-34, 2006.
[8] S. Khurshid, and D. Marinov. TestEra: Specification-based Testing of Java Program using SAT. Automated Software Engineering, 11(4), pages 403-434, 2004.
[9] S. Khurshid, D. Marinov, I. Shlyakhter, and D. Jackson. A Case for Efficient Solution Enumeration. Proceedings of SAT2003, pages 297- 298. LNCS 2919, Springer-Verlag, 2003.
[10] Aritra Bandyopadhyay, and Sudipto Ghosh. Test Input Generation using UML Sequence and State Machines Models. International Conference on Software Testing Verification and Validation, pages 121-130, 2009.
[11] Gordon Fraser, and Angelo Gargantini. Experiments on the Test Case Length in Specification Based Test Case Generation. ICSE Workshop on Automation of Software Test, pages 18-26, 2009.
[12] Shaoying Liu, and Shin Nakajima. A "Vibration" Method for Automatically Generating Test Cases Based on Formal Specifications. 18th Asia-Pacific Software Engineering Conference, pages 5-8, 2011.