Assertion-Driven Test Repair Based on Priority Criteria

Authors: Ruilian Zhao, Shukai Zhang, Yan Wang, Weiwei Wang

Abstract:

Repairing broken test cases is an expensive and challenging task in evolving software systems. Although an automated repair technique with intent-preservation has been proposed, it does not take into account the association between test repairs and assertions, leading a large number of irrelevant candidates and decreasing the repair capability. This paper proposes a assertion-driven test repair approach. Furthermore, a intent-oriented priority criterion is raised to guide the repair candidate generation, making the repairs closer to the intent of the test. In more detail, repair targets are determined through post-dominance relations between assertions and the methods that directly cause compilation errors. Then, test repairs are generated from the target in a bottom-up way, guided by the the intent-oriented priority criteria. Finally, the generated repair candidates are prioritized to match the original test intent. The approach is implemented and evaluated on the benchmark of 4 open-source programs and 91 broken test cases. The result shows that the approach can fix 89% (81/91) broken test cases, which are more effective than the existing intent-preserved test repair approach, and our intent-oriented priority criteria work well.

Keywords: Test repair, test intent, software test, test case evolution.

References:

[1] B. Daniel, V. Jagannath, D. Dig, and D. Marinov, “ReAssert: Suggesting repairs for broken unit tests,” ASE2009 - 24th IEEE/ACM International Conference on Automated Software Engineering, pp. 433–444, 2009.
[2] A. M. Memon and M. L. Soffa, “Regression testing of GUIs,” Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, vol. 28, no. 5, pp. 118–127, 2003.
[3] J. Imtiaz, S. Sherin, M. U. Khan, and M. Z. Iqbal, “A systematic literature review of test breakage prevention and repair techniques,” Information and Software Technology, vol. 113, no. May, pp. 1–19, 2019. (Online). Available: https://doi.org/10.1016/j.infsof.2019.05.001
[4] B. Daniel, D. Dig, T. Gvero, V. Jagannath, J. Jiaa, D. Mitchell, J. Nogiec, S. H. Tan, and D. Marinov, “ReAssert: A tool for repairing broken unit tests,” in Proceedings - International Conference on Software Engineering, 2011, pp. 1010–1012.
[5] M. Mirzaaghaei, F. Pastore, and M. Pezz´e, “Automatically repairing test cases for evolving method declarations,” in IEEE International Conference on Software Maintenance, ICSM. IEEE, sep 2010, pp. 1–5. (Online). Available: http://ieeexplore.ieee.org/document/5609549/
[6] M. Mirzaaghaei, F. Pastore, and M. Pezz`e, “Supporting test suite evolution through test case adaptation,” Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012, vol. 2, pp. 231–240, 2012.
[7] Y. Xu, B. Huang, G. Wu, and M. Yuan, “Using genetic algorithms to repair JUnit test cases,” Proceedings - Asia-Pacific Software Engineering Conference, APSEC, vol. 1, pp. 287–294, 2014.
[8] L. S. Pinto, S. Sinha, and A. Orso, “Understanding myths and realities of test-suite evolution,” Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE 2012, vol. 1, pp. 1–11, 2012.
[9] X. Li, M. D’Amorim, and A. Orso, “Intent-preserving test repair,” Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019, pp. 217–227, 2019.
[10] R. A. Assi, W. Masri, and F. Zaraket, “UCov: a user-defined coverage criterion for test case intent verification,” Software Testing Verification and Reliability, vol. 26, no. 6, pp. 460–491, 2016.
[11] T. Xie, D. Marinov, and D. Notkin, “Rostra: A framework for detecting redundant object-oriented unit tests,” Proceedings - 19th International Conference on Automated Software Engineering, ASE 2004, pp. 196–205, 2004.
[12] D. van Bruggen. JavaParser : Analyse, transform and generate your Java codebase. (Online). Available: http://javaparser.org/
[13] W. Visser, C. S. Pˇasˇareanu, and S. Khurshid, “Test input generation with Java PathFinder,” in ISSTA 2004 - Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, 2004, pp. 97–107.
[14] B. Daniel, T. Gvero, and D. Marinov, “On test repair using symbolic execution,” ISSTA’10 - Proceedings of the 2010 International Symposium on Software Testing and Analysis, pp. 207–217, 2010.
[15] H. A. Nguyen, T. T. Nguyen, T. N. Nguyen, and H. V. Nguyen, “Interaction-based tracking of program entities for test case evolution,” Proceedings - 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017, pp. 433–443, 2017.
[16] I. Balaban, F. Tip, and R. Fuhrer, “Refactoring support for class library migration,” ACM SIGPLAN Notices, vol. 40, no. 10, pp. 265–279, 2005.
[17] H. Zhong, S. Thummalapenta, T. Xie, L. Zhang, and Q. Wang, “Mining API mapping for language migration,” Proceedings - International Conference on Software Engineering, vol. 1, pp. 195–204, 2010.
[18] A. T. Nguyen, H. A. Nguyen, T. T. Nguyen, and T. N. Nguyen, “Statistical learning approach for mining API usage mappings for code migration,” in ASE 2014 - Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, 2014, pp. 457–468.
[19] Y. Yuan and W. Banzhaf, “ARjA: Automated repair of Java programs via multi-objective genetic programming,” arXiv, vol. 46, no. 10, pp. 1040–1067, 2017.
[20] J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Towards practical program repair with on-demand candidate generation,” pp. 12–23, 2018.
[21] S. McMaster and A. M. Memon, “An extensible heuristic-based framework for GUI test case maintenance,” IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2009, pp. 251–254, 2009.
[22] S. Zhang, H. Lü, and M. D. Ernst, “Automatically repairing broken workflows for evolving GUI applications,” in 2013 International Symposium on Software Testing and Analysis, ISSTA 2013 - Proceedings, 2013, pp. 45–55.
[23] X. Li, N. Chang, Y. Wang, H. Huang, Y. Pei, L. Wang, and X. Li, “ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications,” Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017, pp. 161–171, 2017.
[24] S. R. Choudhary, D. Zhao, H. Versee, and A. Orso, “WATER: Web application TEst repair,” in 2011 International Workshop on End-to-End Test Script Engineering, ETSE 2011 - Proceedings, 2011, pp. 24–29.
[25] M. Hammoudi, G. Rothermel, and A. Stocco, “WATERFALL: An incremental approach for repairing record-replay tests of web applications,” in Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, vol. 13-18-Nove, 2016, pp. 751–762.
[26] A. Stocco, R. Yandrapally, and A. Mesbah, “Visual web test repair,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York, NY, USA: ACM, oct 2018, pp. 503–514. (Online). Available: https://dl.acm.org/doi/10.1145/3236024.3236063