From Automation to Autonomous Testing

CIO Talk – Hexaware Blogcast: Welcome listeners! This is Sanjog, your host, and the topic for conversation is From Automation to Autonomous Testing

With digitization, we are in a race. The race with our competitors to stay ahead in meeting and exceeding customer expectations. It requires that we keep innovating our products and services using technology, and then take them to the customers as quickly as possible, in some cases, in a matter of a few hours.

For such cases, the traditional approach to testing that is dependent on human intervention simply cannot keep up. Some companies are evaluating a move from test automation to autonomous testing that takes advantage of AI/ML to make testing less dependent on human intervention and self-learning.

So, how does autonomous testing work? Is it ready for the real world? How can organizations transition from test automation to autonomous testing with confidence?

To discuss this, I have with me Tony Mohanty and Nagendra BS. Tony is the Senior Vice President and Global Head of Digital Assurance and Nagendra BS is the Vice President and Head of Practice and Solutions, at Hexaware, a consulting firm focused on transforming IT solutions and solving complex business problems using a combination of human creativity and intellect. Their three-pronged strategy of Automate Everything^®, Cloudify Everything^®, and Transform Customer Experiences^® fast-tracks enterprises into the digital era.

Hello Tony and Nagendra…Thank you for joining us.

Tony & Nagendra says, “Thanks for having me.”

If you wish to directly listen to the Podcast: From Automation to Autonomous Testing, Click here

Sanjog: So Tony, the test automation discipline is not new. Most organizations have already deployed it and are reaping benefits from related investments. Why should anyone drop what’s working well, and consider autonomous testing?

1. Automation, per se, has a limited connotation and is usually associated with execution of tests from a quality assurance perspective; whereas Autonomous testing has a far wider impact across all phases of the testing lifecycle with the aim of eliminating human intervention in all the related tasks. And this is becoming increasingly more relevant because:
2. <Describe the Business Problem>
   1. 1. The pace of change driven by digital transformation is rapidly increasing as new generations of agile competitors emerge and customers come to expect more rapid updates to products. Testing cannot be slowing it down!
      2. Beyond the current level of automation that exists today, what other levers do we have for increasing the speed of delivery and reducing human intervention? How do we increase the efficiency and reduce the cost of quality? Autonomous testing is the answer to these questions. I will explain it with an example:
      3. One of our airline customers in North America has implemented a high level of test automation and operates in a DevOps delivery model with CI/CT and daily builds. However, there are still many test activities that are still done manually which continues to have a dependency on people, and hence there is an associated cost. For instance,
         1. Impact Analysis of the requirement changes for their critical applications like flight reservation systems were done manually
         2. Maintenance of automation scripts for every change requires quite a bit of effort from the team
         3. There was no proactive learning from different data sources like production logs to pre-empt issues
         4. Usually the entire regression pack is run in every cycle which is not optimal. Ideally, we should be able to pinpoint which test scripts to execute for a given change and execute the same, thus saving time and cost in every cycle.
      4. This is the same story with most of the enterprises in the industry, where majority of the testing related activities and decision making still happens manually
3. <Describe the Opportunity> While these are some of the current challenges, this also presents a big opportunity for us:
   1. To support this pace of change, we believe that the next level of transformation for software testing is to shift from Test Automation to Autonomous Testing leveraging AI/ML to make testing the fastest cog in the DevOps chain
   2. By making testing virtually independent of human intervention, this has the potential to result in incremental savings of anywhere between 30 to 70% on the current QA spend, depending on the current test automation maturity levels of the organizations
   3. The Analyst reports also corroborate this shift – According to Gartner’s “Competitive Landscape for Application Testing Services” published recently, it states that “By 2021, intelligent automation will generate an additional 20% savings over what is achievable today in application testing services for end users”
   4. According to Forrester’s Developer survey a year back, 37% of organizations said that they will use AI and ML to test faster and to increase quality. 61% of the respondents said that they were using AI/ML algorithms to prevent incidents in production, and 50% of them said that they use AI/ML for augmenting tester’s capabilities

Today, there are at least a dozen start-ups like Autonomiq, Functionize, Algoshack just to name a few who are building products to enable autonomous testing. They have been able to raise millions of dollars in funding from various PE and VC firms which is also a testimony to the potential rise of Autonomous Testing.

Hope this clarifies why organizations need to shift their focus to Autonomous testing and the opportunity in front of all of us.

Sanjog: OK, granted, autonomous testing does sound like a great concept and perhaps can help organizations keep up with the pace of change. But, why hasn’t it taken off? What’s holding us back? Nagendra – what are your views?

As part of intelligent test automation, different elements of autonomous testing are already in place, but not as an end-to-end solution that would make complete testing function independent of human intervention. There are many reasons for this.

One of the biggest reasons is, Testing is not seen as a strategic growth and efficiency enabler in the organizations, and this discourages from any investments to transform the testing function. The other reason is skepticism around the solution being too futuristic and organizations for whatever reasons may not have been able to get the returns on their investments made on test automation.

Also, till recently we did not have the kind of access to AI-enabled technology solutions and platforms like Tensorflow, Kera, Theano and similar platforms which helps in democratizing AI and this was another constraint.

While many of these are ground realities that we have to deal with, another constraint that was holding us back from taking off is also the unavailability of a unified platform to support autonomous testing across all testing types, across all the phases of the testing lifecycle and layers of an application which is what Hexaware has embarked upon.

Let me also share some of the recent conversations with our customers on this topic. At the start of this year, we had hosted many customers across varied industries like airline, banking, insurance, and retail at our campus to whom we presented our vision to move from test automation to autonomous testing. While we had some customers, who bought into this and offered to run some pilots, there were also few customers who were skeptical about this solution.

Some of them are customers whom we have been engaged with for more than 8 to 10 years. While we acknowledge that their reservations are legitimate which are specific to their environments, we are also working with them to address their concerns since we believe this opportunity has greater upside to our customers

Sanjog: So, Nagendra – What’s the proof that Autonomous Testing would work? Has it been tested in the field with real use cases? What are the results and related learning?

As Tony mentioned earlier, one of the proof points from a business potential point of view is the fact that startups focusing on autonomous testing solutions have been able to raise millions of dollars in funding from various Private Equity and VC firms.

While this would give some level of confidence that autonomous testing is real, we are also finding sponsors at the CIO level in our customer organizations who are encouraging to deploy these point solutions.

We recently implemented one of the point solution for a manufacturing services company. Through this solution, we were able to do a seamless conversion of existing manual test assets into corresponding automation scripts with little or no manual intervention.

We have seen test analytics solutions implemented in the industry that use Python-based ML libraries to predict defect patterns of the future releases based on the data of previous releases.

Another example that I can quote is to support testing in Behavior Driven Development or BDD mode of SDLC. We have implemented a solution leveraging Python-based natural language tool kit that can import a Gherkin language feature file and generate corresponding automation scripts without any human intervention.

Similarly, there are many other use cases where we have seen Python-based AI/ML libraries and algorithms like logistic regression, reinforcement learning, clustering being used for solving challenges around automation script maintenance and impact analysis.

Through our experience of implementing different point solutions and elements of autonomous testing, we are now absolutely convinced that AI/ML is real and can make a difference to the way the testing is performed. In fact, AI will not eliminate manual testers completely, but will augment their skills.

We also believe it is critical to have a C-level Sponsorship for the success of this transformation and there needs to be top down approach to adopt the change as it requires collaboration between different IT teams outside of QA involving development, infrastructure, release management teams to make this happen.

Finally, we see a need to have a unified platform that can orchestrate end-to-end testing activities without any human intervention. This platform should be able to integrate seamlessly with existing or third-party automation solutions and if necessary, have its features exposed as services or APIs for external consumption.

Sanjog: Tony, what are the tenets of an ideal solution to enable Autonomous Testing most effectively? How well do currently available solutions, including Hexaware’s offering, meet that benchmark?

1. The key tenets of an ideal solution to enable autonomous testing are:
   1. A comprehensive maturity assessment framework covering use cases, persona, and activities across testing lifecycle to evaluate customer’s current autonomous testing maturity and provide a roadmap for implementation
   2. This needs an integrated test orchestration platform with a plug and play architecture that enables customers to either go with a vendor’s solution or integrate their existing or third-party automation solutions. Or even have an option to consume the platform’s features as services through API calls
   3. Data is the key for this platform to deliver results and we believe that there are 4 key tasks from a data perspective to operationalize the test orchestration platform:
   4. 1. Acquisition of massive data (Acquire) generated during various phases of the application lifecycle categorized as Voice of Customer, Voice of Machines, Voice of Tests, Voice of Business and Voice of Developer,
   5. 2. Analysis of the acquired data (Analyze),
   6. 3. Developing inferences (Agree) and
   7. 4. taking decisions (Act) Leveraging AI technologies
2. Hexaware’s dedicated consulting team performs a detailed due diligence over a period of up to 3 weeks leveraging our ATMA (Autonomous Test Maturity Assessment) framework which has 5 levels of autonomous testing maturity and provides a detailed report and roadmap. On a lighter note, the acronym ATMA is quite relevant in this context because this will be the soul of the test maturity roadmap for organizations
3. While there are existing point solutions in the market, Hexaware has an Integrated platform in the form of ATOP – Autonomous Test Orchestration Platform that uses Machine Learning, Deep Learning and NLP to enable this transition from automation to autonomous. It can be the one-stop solution for all testing needs of our customers
4. As part of ATPP implementation for one of our customers, we have brought to life the production log analysis use case (voice of machine use case) which enables us to automatically identify the issues from production logs and generate corresponding automation scripts to strengthen the test pack
5. Another use case is around voice of customer which is to analyze the sentiments expressed by the end users on portals/ social media. The solution goes through the customer reviews from various sources, classifies the positive and negative reviews, clusters the predicted negative reviews using module names to identify which modules were prone to issues and generates test cases for those modules

As part of our roadmap for ATOP, we have identified more than 50 such use cases across functional and non-functional testing to enable autonomous testing for testing the UI, Service layer and the Data layers of every application.

Nagendra, how should organizations start on the test automation to autonomous testing journey, all along ensuring the quality and accuracy of the results produced, and minimizing risk during the initial implementation as well as when fully operationalized?

The first thing that we would recommend enterprise leaders is to recognize the fact that there is an opportunity to tap by thinking beyond automation. The journey from automation to autonomous would take anywhere between 12 to 24 months depending on their current level of QA and Automation maturity.

An assessment of the current maturity level of autonomous testing and baselining of existing metrics must be done to arrive at a detailed roadmap for implementation.

The roadmap should detail various aspects like tasks that will be done in-house  vs leveraging partners, clarity on which are the testing activities that can be completely made autonomous vs activities that will still have to be done manually, what is the approach for building unified orchestration platform, what is the approach for acquiring data across various tools in the eco-system and other relevant details to make both functional and non-functional testing autonomous across all phases of testing lifecycle and all the layers of an application.

From an implementation point of view, we would recommend picking one or two pilot project/programs which are mature enough to take up activities beyond test automation and show the initial proof points before we take up enterprise-wide implementation. There must be an organization change management team created to drive this transformation.

While the QA function would be the owner of this transformation, we would recommend a “skin in the game approach” both for stakeholders within the organization and the partners who would be supporting this transformation.

We also recommend having a well-defined engagement model that would enable organizations to measure partners on the outcomes with attached SLAs/KPIs and at the same time provide necessary ownership to partners so that they can drive the activities. Have a dashboard that provides real-time insights on the expected outcomes, performance against corresponding SLAs/KPIs and accelerate the whole transformation through Organization Change Management (OCM) coaches.

Finally, this transformation will not be successful without the right people on the ground to deliver. Enable workforce transformation through SDETs: Software Development Engineer in Test and take a pragmatic approach for implementing autonomous testing by leveraging existing assets and complement the same with partner capabilities.

Finally, Tony – Transforming from test automation to autonomous testing seems like a significant effort, which may require help from a partner with specialized expertise and experience. Since it is such a new discipline, how should an organization go about selecting the right partner for this effort? If an organization selects Hexaware as a partner, besides just bringing your technology platform, how can your team help in ensuring success, reducing risk, and maximizing business outcome?

1. 1. For selecting a partner for this initiative, an organization should look at potential partners for whom Software Testing is one of the strategic businesses in the partner’s company
   2. The partners should have solid experience and credentials in delivering quality engineering solutions across technologies and industries
   3. They should have Demonstrable solutions that use AI/ML in testing and corresponding case studies and client testimonials
   4. They should be willing to put their skin in the game by committing outcomes upfront both from a delivery perspective and commercially as well
   5. And if you look at Hexaware as an organization, we are a Global IT services company. We have been delivering Quality Assurance and Engineering services under the Digital Assurance umbrella for more than two decades now
   6. Nearly 20% of the organization’s revenues is from Digital Assurance services and this is a strategic business for us. There is a razor-sharp focus to help customers succeed in their transformation journey in our DNA
   7. Compared to our peers in the industry, we have disproportionally invested more than 5% of each unit’s revenues towards maturing service offerings and driving innovation via a central Innovation Lab that supports rapid solution development
   8. We have a well-established Partner Ecosystem with established players and niche startups that accelerate our whole journey towards Autonomous testing
   9. We have referenceable case studies of both automation and autonomous testing solutions with multiple clients across industries, for e.g.,
      • An autonomous testing led managed test service for a multi-billion dollar North American Airline, end-to-end automation for a large European insurance company, extreme automation for a leading secondary mortgage provider in North America, etc.
   10. Career test analysts with 75% of them being SDETs ensuring DevOps and Agile Readiness
   11. Brain box – We have a crowd sourced platform for Customer Value Adds where every Hexawarian can contribute ideas for eliminating manual work through automation in their respective engagements
   12. With an excellent leadership team that is very well aligned towards our strategy for the future with focus on Automate Everything^®, Cloudify Everything^® and Transform Customer Experiences^®, we have built enormous trust over the years both with existing clients (some of them being with us for 20 years or more) and new customers we have onboarded recently.

With such credentials and experience, we are pretty well positioned to lead our customers all the way in the journey of Autonomous testing.

Once again, thank you, Tony and Nagendra, for sharing your thoughts and insights about how an organization can transition from test automation to autonomous testing to keep up with the pace of business change.

And listeners—I invite you to find related conversations on our website at CIOTalkNetwork.com