Test Automation ROI Primer

Software Testing Automation has gained significant acceptance in enterprises in recent years. Many organizations struggle to identify the value drivers in a test-automation purchase and might take decisions based on partial data. 

This article is an attempt to help people understand why automation would make sense, how to understand the various platforms out there, how to balance the economics and present an ROI to ensure proper communication of the expectations.

Let us start by understanding the need for testing automation –

Software is consistent – Any software will consistently perform actions as per the instructions. While the instruction set might be suspect, the software would be consistent irrespective of the day of the week, the temperature outside or any of the factors that impact humans.

Software is exact – Logical structures have no room for common sense.  So, the onus on the developer to understand intent and implications is high.  Given that majority of enterprise software, today consists of a myriad set of options, the chances that a particular combination of options (an option set) might work contrary to intentions is high. 

The software will evolve – Any software that you use is amorphous and would continue to evolve as long as there are customers and/or developers that wish to tinker. Change is typically due to innovation, competition, regulation, or adaptation. 

Software is fragile – New features and options must be delicately introduced, considering the existing options and their usage.  It is very much possible to unintentionally “break” an “option-set” while bringing in some new capabilities. 

Software Testing – Considering these attributes, it is imperative that software is tested – when it is designed, built, deployed, or modified. The discipline of software testing has been evolving in parallel to software development, albeit in the shadows. Considering the repetitive nature of software testing, a lot of research and effort has been spent on reducing the “drudgery” of the repetitive tasks by adopting automation in the practice of testing.  Automated testing is a large part of the public debate and we have explored why exploratory and automated testing needs to coexist in our article here.    

In this article, we would attempt to share how test automation supports the mission of consistently deploying reliable software and how one could explore the ROI of these efforts. This is important as Test Automation is primarily a tool to improve efficiency and not a tool to improve the effectiveness of your tests.  

What Constitutes Test Automation

The practice of software testing essentially is a stimulus-response study. You provide stimulus to the software (“Application under-test” or AUT) and validate if its response meets a predetermined criterion. 

The profession of software testing is to

  1. identify various stimulus-response conditions (test-cases) that could be encountered during the use of the software
  2. plan and execute a sequence of such test-cases
  3. capture and report software behavior,

Test Automation is an aid to a software tester, where the execution and reporting of a certain subset of cases are delegated to the automation software. 

 Automation allows teams to focus on “net-new”

As software gets enriched, the testing burden shifts from testing “new” feature to testing the “existing” features.  In the graph, we have shown 5 sprints where each bar represents newsprint.  The “new” functionality in each sprint is shown in “blue” and the “existing” features from previous sprints are in “grey”. You would notice that while the volume of new features is relatively steady, the volume of “existing” features has grown with every release.    

While leaders are happy to spend time/effort on testing the blue portions, the repetitive testing of the grey portions is a shoe that pinches. Re-testing of functionality that has been available for a few releases represents a unique set of challenges across all stakeholders. From a tester’s perspective, it is repetitive and therefore boring. It is very likely to only get a cursory look-over and not an in-depth review.  From a sponsor’s perspective, it represents an ever-increasing volume of work that often has a low-return element.  Therefore, we risk under-testing and letting defects slip into “deemed-stable” portions of our software.  Test-automation represents a middle path, where the code can be tested for a larger sample of usage conditions at a lower cost-time construct where only the exceptions are picked-up for further evaluation. 

What benefits can I expect? What benefits should I not expect?

The investment in the right automation tool once set up offers potential benefits like:

Coverage: You could achieve a higher test coverage / larger sampling of the software functionality, faster and at a lower execution effort, than what you would get with pure manual efforts. 

Speed:  Automated tests can run faster and round the clock.  These could reduce your overall time-to-test significantly, giving more time to development teams to fix any potential issues. 

Consistency: Test automation tools offer maximum consistency. The same test case runs multiple times without even a minor deviation.

Effort:  Test Automation can be set up to run with little or no manual intervention. 

The Caveats

Setup: Setting up tests on any automation tool is a laborious task.  It requires planning and foresight that only experience would bring. Knowledge of how data changes across time, what variables would need to be dynamically updated and how to validate expected outcomes etc. require experience and time. 

Maintenance:  Once you have set up a large library of cases to be automatically executed, it should be smooth sailing; or so we think!!  The challenge of test automation (and therefore where money is generally made or lost) is how easy it would be to maintain this comprehensive library when your underlying application changes. When the “Application-Under-Test” changes, you would have to identify all aspects of your automation that consequently need a change.  

Test Automation – Understanding Costs & Benefits

Companies are increasingly investing in test automation considering the increased frequency of changes and anticipated cost-benefit. The investment in the right automation tool is worth it, as it offers potential benefits like:

Coverage: Test automation offers high test coverage with nearly 100% results, which otherwise cannot be achieved by manual testing.

Consistency: Test automation tools offer maximum consistency. The same test case was run multiple times without deviation.

Speed: Automation is multiple times greater than the manual testing speed. 

Effort: The manual effort in executing a test automation tool is almost negligible.

Test-automation solutions are varied and bring different categories of over-bearing costs

The cost of the test automation can be categorized as:

  • Acquisition cost
  • Maintenance cost

Acquisition cost: There are some free test automation tools available in the market. The free versions mostly perform basic functions and are not recommended for thorough end-to-end or regression testing.

To buy an efficient test automation tool, you will require to make some investment. A licensed and open-source automation tool may start with as low as $4000, depending on the brand and functionality it offers. The total acquisition cost is often inclusive of the tool cost and construction cost. Before investing in the tool, it is recommended to do thorough research and understand the right tool that will complement your requirements perfectly.

Maintenance cost: The maintenance cost of automated tests drives expenses up. Maintenance may involve a range of factors:

  • Process
  • User interface
  • Data
  • Environment

However, maintenance of automated tests may cost you a bit, statistics reveal the fact that maintenance can be a lot cheaper affair than creating new test scripts. Furthermore, frequent maintenance will keep the system efficient and seamless. Maintenance is a good option while considering the cost model of the project, as manual testing or implementing new automation tools can be expensive and time-consuming.

Diagram Description automatically generated with medium confidence

Test Automation ROI

Now that we have given you an understanding of the costs and benefits of test automation, we go further deep into exploring the ROI of these efforts. Companies are constantly investing in test automation, but investment in test automation without proper calculation of ROI would be a half-baked effort, which is unacceptable for the companies.

At the beginning of this article, we have stated that test Automation is primarily a tool to improve efficiency, and the calculation of ROI determines the efficiency of your test automation process. Test automation is a one-time investment. Configuring the test automation at the initial stage of project inception can maximize the benefits in the long run. By calculating ROI, companies can decide to invest in test automation.

Calculating ROI does not necessarily mean that we would focus on the overall cost, which is required to evaluate the quality of the product. ROI calculation also includes the parameters like late releases, defects overlooked, and lack of quality.

One of the biggest challenges that the companies face is investing a large sum of money for conducting the end-to-end testing of software, which often they are unable to recoup. Once the companies decide what test sets or subsets they must automate, they will generate a high return on investment.

Due to poor decision-making, companies may spend a lot to adopt automation at the later stage of development, and if the project fails, they may incur a high loss. ROI-based automation testing resolves this issue. The Quality Assurance team decides which section must be automated based on the clients’ environment, system, and machine. They can do a cost comparison analysis of manual test runs and automation test runs and determine the types of scripts that can be automated.

Following are the significant metrics, which can be considered while calculating automated testing ROI.

  • Measure product quality – Ensuring the overall quality of the product by identifying defects and mean time to detection of defects.
  • Test execution speed – Comparing the test run duration percentage between automated and manual testing.
  • Thorough cost and resource analysis – Analyzing cost and resources are important as the test automation process involves software and hardware costs, prioritizing defects based on their severity, operational costs, the requirement of new resources, training costs, and more.

With the metrics mentioned above, we can calculate Automated Testing ROI. The cost that companies bear on resources, tools used, setting up the testing framework and more fall under companies’ investments. The net gain that the companies receive over their investments would be the ROI for the automation testing.

Following are the parameters required for performing ROI calculation.

Automating new tests – There are always a certain amount of costs involved in automating new tests. Before initiating the automation of new tests, companies must consider the predictable costs of developing, executing, and maintaining the tests. Based on the test categorization, you would segregate the test to automate or run manually.

Automate regression testing – Adding new features or changes in software would introduce new bugs in the existing modules. It makes regression testing one of the most expensive phases of testing. It is a necessary step to calculate the cost of automating regression testing.

Managing Multiple Environments – As companies develop products, they must be convinced that their product works equally in different environments. Hence, companies set up multiple platforms, devices, and browsers to ensure the success of their products. For organizations, it is necessary to calculate the ROI while setting up test environments.

Minimizing Defect Leakage – There are instances where bugs were detected at the production phase, as they were not resolved at the development stage. It often leads to defect leakage in the production phase, making it an expensive issue for organizations. Calculating ROI is necessary for companies to control such overhaul expenditures in certain instances.

Reusable test cases – Using reusable test cases saves time and effort by avoiding task duplication. When you already have created test cases, you can reuse them in multiple scenarios. Hence this will impact while calculating ROI.

Protecting product knowledge – Companies may still gain control over their systems and software, but they need to take a calculative risk on their resources. If an engineer leaves, the company must look for a replacement and provide training for the job. It may impact the company’s long-term automation testing ROI.

Ways to improve Automated Testing ROI

Instead of automating all the tasks at hand, it is best to automate repetitive tasks. Automating the applications, which are not steady, may not show an accurate result. Moreover, it may increase your team’s effort and affect while calculating ROI.

Automation testing works best for regression testing. Hence, when new tests and integrated with the older ones and applications are tested across various platforms, automation testing can be implemented to save time, effort, and overall costs. Further, if you need to complete the test run in a shorter time, you can run parallel testing like a smoke test.

Changes are introduced frequently in software. Hence, the developers and testers are relying on exploratory testing which is widely used in agile models. The focus of exploratory testing is discovering, investigating, and learning through the process. The testers here own the freedom and responsibility to think and note down ideas before the test execution. As exploratory testing is strategic, it helps in improving automated testing ROI.

The testing process has evolved, and today shift-left testing is a popular step in the software development lifecycle. Shift-left testing allows you to detect defects in the early stage, which helps in reducing effort, time, and money. You could also use test management tools to identify redundant test cases at the early stage of the development process. Finally, keep thorough documentation of the process so that the newly hired resources can pick up where your previous resources have left.

Conclusion

The company adopts automation testing to improve its test efficiency and test coverage. Of all the possible methods, which could be applicable for calculating the automated testing ROI, the company must adopt the strategies, which work best for them and suffice their requirements.

When it is about improving the quality of your product, the right methodology would be to test your applications simultaneously while developing them. If defects are detected at the postproduction stage, the company will suffer monetary loss, which may be unacceptable for any organization. Hence, a little strategy, the right tool, and differentiation between manual and automation testing can successfully help you implement test automation ROI.

5 challenges of testing ERP systems in the banking sector and ways to overcome the challenges

There are multiple reasons why the banking and financial industry needs enterprise resource planning (ERP) systems. ERP system facilitates agile delivery in the broader spectrum of banking and financial services and solutions. The system is used to understand the implementation at a large scale. The ERP systems offer solutions to multiple issues in the banking industry and help with thorough insights for upcoming projects. The ERP system helps in resource planning and increases the efficiency and productivity of an organization. The organizations have seen visible results by implementing ERP systems due to their benefits like,

  1. Improves business reporting – ERP creates one integrated database for all business processes and provides real-time information and comprehensive business reports.
  2. Improves customer services – ERP systems offer better accessibility to customer information with an improved response time, on-time delivery, and order accuracy.
  3. Better inventory costs – ERP systems include only the most essential inventory and avoid common problems like higher overhead costs and longer customer fulfilment times.
  4. Improves cash flow – ERP systems offer better invoicing. It is a superior collection tool that enhances cash flow. Faster cash inflow allows adequate investment for the business.
  5. Improves cost savings – ERP system allows cost-saving with improved inventory planning, customer services, procurement, and vendor relationship management.
  6. Improves data and cloud Security – ERP system has dedicated security resources, which prevent installing malicious software. It improves data and cloud security as the data is spread across multiple servers.
  7. Improves business process – ERP system helps improve mundane or manual tasks and implements smarter workflows allowing you to work more efficiently.
  8. Manages supply chain – ERP systems effectively forecast demand and lean inventory. It reduces the production bottleneck and offers greater transparency in your business.

However, the organization cannot always guarantee the successful implementation of ERP systems. ERP implementation can fail at its pilot phase due to multiple reasons. Poor project management, data inaccuracy or quality issues, implementation without adequate planning, ineffective consultation, inability to reduce the implementation cost, and more are a few reasons for ERP implementation failures. ERP implementation depends largely on the organizational approach. The organization may invest time and money only to see process stagnancy with little to no significant growth.

Testing of ERP systems in the banking and financial industry

ERP systems in banks and financial institutions are a procedure to plan and are commonly used by banks and financial institutions because of time constraints. ERP implementation works best with the right planning and strategy. It enhances the productivity and effectiveness of the process and people, supports business operations, and helps in business decision-making. ERP systems in banks and financial institutions can be tested at the core level and implementation level. The core team validates the static functionality of ERP systems, while the implementation team validates the dynamic and tailored features and functionalities.

The changes in features and functionalities, as we know in the banking industry, are extremely frequent. A small change in the system features can affect the workflow of multiple modules. The changes must be recorded and validated accordingly to ensure that none of the changes alters the system configurations. A system with a massive amount of data requires test automation. The organization analyzes the requirements and designs the automation test framework based on organizational infrastructure.

Challenges of testing ERP systems

  1. Frequent changes and inaccurate data – The ERP system implementation is a long process during which the system performance can fluctuate a great deal. The entire implementation process involves multiple risks that must be addressed in time. Banking systems are vulnerable and open to frequent changes. Due to the frequency of changes in the banking structure, the update must be constant and continuous, which also requires system receptibility. The changes must reflect in the system without delay. If there is a delay in data reflection to the system, there will be a discrepancy in available data that leads to data inaccuracy.
  2. Integration of new data with old legacy systems – Banking ERP systems have a broader scope. The dataset entered in the bank ERP systems is used in the various processes throughout the organization. The data is used in Marketing & Sales, Accounting & Finance, Supply Chain Management, Human Resources and many more. The data is stored centrally in the old legacy systems and referred to by various organizational verticals for different purposes. The main challenge here is it often becomes hard to test these legacy systems due to the absence of adequate tools. As the legacy systems store a high volume of data, the end-to-end testing of the legacy systems becomes necessary. It is a time-consuming and tedious task to test the legacy systems. In the case of banks migrating data from legacy systems to new ERP platforms, there can be a delay in the entire testing project leading because of slow data migration and unavailability of data, which can lead to multiple errors in the systems and banks’ internal processes.
  3. Implementing ERP in rush and without planning – This challenge is common when banks start their operation for the first time. Due to rising demand and customer requirements, banks and financial institutions are sometimes in a rush to implement ERP systems and critical aspects of the ERP systems can be overlooked. The urgency of implementing the ERP systems always leads banks to ignore the critical aspects of ensuring system quality. It also leads to implementing ERP software without adequate planning. If the systems are implemented in rush it might lead to multiple performance issues, data integration and security, data migration issues and more.
  4. Lack of project knowledge – Lack of project knowledge can lead the team to witness challenges of testing ERP systems. The team must have adequate project knowledge and training to learn about the project requirements. Adequate communication and training about project requirements make the ERP testing process easy and convenient for the team. With adequate project knowledge, the testing team can validate various aspects of ERP implementation at the various phases across the business operation.
  5. Lack of technical knowledge and absence of specialized team – ERP testing in banks is not a mission-critical practice, which leads backs to ignore a few critical aspects of ERP implementations. Banks and financial organizations execute testing ERP implementation along with other crucial testing projects. And often, the management team do not feel the need to delegate the most competent team. Banks also do not deploy a specialized team for testing the ERP platform. Banks cannot guarantee the success of ERP implementation testing due to a lack of technical knowledge and a specialized testing team.

Ways to overcome the challenges of testing ERP systems

All the components and modules of the ERP platform must pass through system integration testing. ERP being a central data system is a data source for multiple verticals. If the source data is not integrated efficiently, the processes within the organizations will not receive adequate and accurate data creating an immense discrepancy in the organizations. Banks’ data and information require utmost security and safety against data theft and manipulation. Hence, security testing is critical to ensure the safety of sensitive data.

Due to multiple changes incorporated in the Bank ERP systems, there can be a few alterations in the user interface design. It can also lead to confusion among the users. Hence with each implementation, it is necessary to ensure that the changes meet user requirements. System efficiency is crucial even with effective changes. An adequate module validation can ensure that even with several changes, implementations, and interpretations, the design flow of the interface remains intact. System integration and usability testing ensure that users can use the systems without disruptions.

It is necessary to validate the performance of the ERP system. The ERP system’s accuracy and speed determine its performance. It is crucial to validate the systems of their load, stress, capacity, volume, and scalability to ensure the reliability and stability of the system even with its extreme load. The performance of banking systems comes under heavy scrutiny from users or customers. Hence, testing the software before installation is essential to always ensure seamless performance.

Even a slight change can impact on the system’s performance, and as far as banking systems are concerned, the changes are extremely frequent. Regression testing ensures that the minor changes do not affect another subsequent module. As modules are interdependent, the poor performance of one module can heavily impact another. Through testing, the team ensures that all the modules and features run smoothly without any errors.

Conclusion

Testing ERP systems has drawbacks, but since ERP systems are not among the most critical functioning systems, banks and financial institutions tend to delay the testing of ERP systems or keep it long on their bucket list. It does not reduce the complexities of ERP system testing but aggravates it. The most pertinent approach is to address the issue by testing simultaneously with the implementation process. The most accurate approach is to build an appropriate implementation strategy and plan to avoid resolving the issues at a later stage. It is always a good idea to prevent rather than fix the occurrence.