Factors Affecting Test Automation Stability and Their Solutions

Authors: Nagmani Lnu

Abstract:

Test automation is a vital requirement of any organization to release products faster to their customers. In most cases, an organization has an approach to developing automation but struggles to maintain it. It results in an increased number of flaky tests, reducing return on investments and stakeholders’ confidence. Challenges grow in multiple folds when automation is for User Interface (UI) behaviors. This paper describes the approaches taken to identify the root cause of automation instability in an extensive payments application and the best practices to address that using processes, tools, and technologies, resulting in a 75% reduction of effort.

Keywords: Automation stability, test stability, flaky test, test quality, test automation quality.

References:

[1] B. Shea, “Software testing gets new respect,” InformationWeek, July 2000.
[2] “Time Estimation for software testing,” devmio - Software Know-How, 08-Feb-2016. (Online). Available: https://devm.io/testing/time-estimation-for-software-testing-128078#:~:text=Statistically %20speaking%2C%20testing%20occupies%2020,as%2035%20to%2050%20percent (Accessed: 18-Jan-2023).
[3] Zippia. "16 Amazing Agile Statistics (2023): What Companies Use Agile Methodology" Zippia.com. Nov. 27, 2022, https://www.zippia.com/advice/agile-statistics/
[4] R. Krish, “Test automation framework – challenges in the ever changing technology scenario,” Test Automation Framework – Challenges In The Ever Changing Technology Scenario - page 4. (Online). Available: https://www.siliconindia.com/guestcontributor/guestarticle/383/test-automation-framework-%E2%80%93-challenges-in-the-ever-changing-technology-scenario.html (Accessed: 26-Jan-2023).
[5] “How to find flaky selenium test suite,” LambdaTest, 06-Sep-2022. (Online). Available: https://www.lambdatest.com/blog/flaky-selenium-test-suite/ (Accessed: 26-Jan-2023).
[6] Owain Parry, Gregory Kapfhammer, Michael Hilton, and Phil McMinn, “A survey of flaky tests a survey of flaky tests,” A Survey of Flaky Tests. (Online). Available: https://dl.acm.org/doi/fullHtml/10.1145/3476105 (Accessed: 26-Jan-2023).
[7] Vinodjo, “Test analytics - azure pipelines,” Azure Pipelines | Microsoft Learn. (Online). Available: https://learn.microsoft.com/en-us/azure/devops/pipelines/test/test-analytics?view=azure-devops (Accessed: 26-Jan-2023).
[8] steved0x, “Associate automated tests with test cases azure test plans,” Azure Test Plans | Microsoft Learn. (Online). Available: https://learn.microsoft.com/en-us/azure/devops/test/associate-automated-test-with-test-case?view=azure-devops (Accessed: 26-Jan-2023).
[9] K. Rungta, “Xpath in selenium,” Guru99, 21-Dec-2022. (Online). Available: https://www.guru99.com/xpath-selenium.html (Accessed: 26-Jan-2023).
[10] M. Fayaz, “Azure devops vs AWS Devops,” Cloud Training Program. (Online). Available: https://k21academy.com/amazon-web-services/aws-devops-vs-azure-devops/ (Accessed: 26-Jan-2023).
[11] Chcomley, “What is azure devops? - azure DevOps,” Azure DevOps | Microsoft Learn. (Online). Available: https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops?view=azure-devops (Accessed: 26-Jan-2023)
[12] Qentelli, “It Is Automation, Not Automagic: Avoiding Failures in Test Automation Projects,” https://www.qentelli.com/thought-leadership/insights/it-automation-not-automagic-avoiding-test-automation-failures (Accessed: 23-Dec-2023)