Mobile Test Automation Supports Digital Agility

When speaking with customers about mobile development and mobile application testing, we routinely find our customers describing what we refer to as the Mobile Device “Fan Effect”, which is essentially the rapid expansion of work required to safely test and deploy a new mobile app to the market.  Often the discussion centers around the complexities of managing risk, managing internal customers, and finally about managing the multitude of devices (i.e., now required in today’s market to be sure the app is ‘fully’ tested) in an orderly fashion.  Similar to the old browser proliferation a decade ago, mobile testing has similar challenges when considering the range of OS types, device types, and screen sizes.  But as we’ve experienced first-hand from delivering mobile app solutions to market over the last 4 years, the mobile landscape makes the device proliferation problem much more complex.  For many of our clients, 500-1,000 test cases is an average size for a moderately complex mobile/web based solution.  But when trying to cover the range of devices that represent what the majority of bank customers use, the number of cases quickly multiplies.   As we’ve seen through this “fan effect” phenomenon, those original 500-1,000 test cases quickly fans out to 2,000-5,000!

There is a great report on Android Device and OS fragmentation that captures the essence and scale of this “Fan Effect”.  Our team has seen the effects of this fragmentation first hand as has spent time crafting a solution to streamline the automation and testing of a wide range of manufactured devices, screen sizes, operating systems, and form factors including tablets, smart phones, and feature phones.  Our goal with mobile test automation is to dramatically improve the ability to deliver a high-quality, well tested mobile app solution safely to market, all the while reducing the cost of executing the requisite number of tests to ensure that the mobile banking app being deployed to market has been thoroughly validated.

Our ongoing work in test automation and our development of the Blackstone Quality Assurance Framework (QAF) has provided a tangible solution to solving this mobile device proliferation problem.  We have seen a reduction in average test case execution time from 8 minutes to 30-45 seconds for automatable cases.  That is a time savings of 90%+.   And through the use of concurrent device testing approaches have proven out the ability to further dramatically reduce time to execute test runs.

In one particular case for a top 25 bank, we achieved a level of mobile test automation that allowed the team to test in a half of a day (i.e., ~4 hour period) what would normally have taken a week (~40 hours) for a manual tester to execute.  In this example alone, we were able to reduce our test headcount while executing more test cases week-over-week than was previously achievable.   Building this kind of test automation efficiency will become a necessary part of mobile release planning if banks are looking to keep pace with the continued demand for new mobile features from its customers as well as the continued proliferation of new mobile devices in the market.


– Clay and Ken