When Poor Device Coverage Shreds Your App And Website Testing Strategy

    By Amy Montague, Monday 24 February 2020

    An internal QA function can be stretched to breaking point by the challenge of testing in a vast and constantly shifting digital landscape. Poor device coverage can mean you are always on the back-foot when it comes to website testing, sacrificing either speed or thoroughness in your approach - and very likely both.

    Digital fragmentation is a huge challenge for quality

    Where once a small and dedicated team of internal testers could bring all the necessary insight and hardware to deliver their testing strategy in a relatively short amount of time - now it’s practically impossible for all but the biggest brands to maintain the required hardware and keep it properly updated with the software that reflects the fragmented state of the market place.

    The new reality: 4 billion people on 9000+ device types

    The sheer variety of hardware and software combinations that could now be accessing website or app is mind-boggling.  

    According to Browserstack, every year across the world more than 4 billion people access the web using more than 9000 distinct devices, 21 different operating systems and 8 major engines that power hundreds of different browsers. 

    That’s an estimated 63,000 possible browser/platform/device combinations that could be accessing a website at any given time.  

    Of course, that’s not to say that every part of every single site needs to be tested with all of these combinations - and the bugs or usability issues related to them given equal weighting. 

    A good testing strategy needs to be underpinned by analytics - an understanding of who is (and who is likely to be) using your site and with what devices, operating systems and browsers.  These insights will help you prioritise your areas of focus.  

    But the devices you need to focus on will also change according to the different territories where you are active and  likely to launch.

    Even so, the size of the challenge to get decent test coverage is considerable, particularly in the mobile world.  Achieving more than 70% test coverage in most regions requires a sizeable amount of hardware and achieving more than 95% must be out of reach for a modest internal QA team.  Added to that, it’s a constantly moving target as new products enter the market and old ones are nudged out.

    Long tail of smartphone usage

    Device coverage for mobile testing

    Clearly, just getting the basics covered, particularly if you have an international component to your business, means the completion of testing timetables can spin far out into the future and present logistical challenges that a lean development team can’t realistically respond to.  Bottlenecks, delays and compromises in quality inevitably follow.

    The challenges of website and app testing coverage

    Keeping a reservoir of devices loaded with the right combinations of OS versions (both the latest and older iterations) to reflect the real fragmentation in the market place is a complex and time consuming task in its own right.

    At the same time, platforms can vary in their supported technologies. Some devices support face recognition, while others of the same platform family do not.  Your bank of testing equipment needs to include the right mix of devices and OS versions to cover all the required functionalities of the websites and app you’re testing.  And these, of course, will carry on expanding as your business grows, so you need to be constantly reinvesting in your test equipment.

    And how about covering the complexity of the conditions in which the devices are being used?  With flakey connections away from WiFi, relying on 3G or 4G, outdoors or while otherwise multitasking?  There are important real world use cases that you might want to test, that are time consuming and complex, but also require scale to be meaningful.

    What next?

    The pressure of time, scale and resource in website and app testing often thwart in-house testing teams as they race to get products to market.  IBM found that even when in house testing teams had the skills and equipment available to them, they often have more testing activities to do than they have time to complete. Nearly half of testers in their survey said that having enough time to test mobile products was one of the biggest blockers to their productivity.

    Outsourcing to a performance managed community, who have all the devices you need and the skilled, professionals available for manual testing when you need it most, is an increasingly popular solution for e-commerce brands.   

    It clearly takes the headache away of managing a bank of devices with the required diversity of OS, browsers and the like. But coupled with the right analytics packages it makes exactly the right volume and blend of these devices and their individual functionality instantly available to a digital team. Not only that, but the right testing platform will be able to return results from testing that are understandable for the entire business, supported with images and videos, that show the impact of a particular issue, in order for fixes to be found and implemented more quickly.  

    For a dynamic and effective website and app testing strategy, digital businesses need the breadth and volume of testers and devices to give confidence of coverage, but they also need specificity - the ability to hone in on particularly important device groups for powerful insight.  To do so in an infinitely flexible and scalable way can transform a business’s ability to deliver new functionality at speed and to the standard you and your customers require.

    It can liberate internal testers from repetitive and time consuming admin and maintenance tasks, helping them focus on interpreting results, devising new tests and overseeing the strategic push for ever improving quality.

    Want to improve your website testing process and test across thousands of different device and browser combinations? Talk to us now.

    You may also be interested in... Seeing A Decline In Your Mobile App Downloads? 

    Website testing blog CTA

    Amy Montague

    Amy Montague

    As one of the Marketing Executives for Digivante, Amy provides and reviews most of the copy and visual content for Digivante. Amy has a natural flair for the creative and introduces aspect into her marketing role.

    Accelerate your conversion rate in 24 hours-1

    Accelerate Your Conversation Rate In 24 Hours

    Make the impossible possible with in-depth website, app testing and usability insights.