The cost of mobile testing? Is it worth it?

    By Digivante, Thursday 30 April 2020

    The cost of mobile testing websites and apps can vary, depending on who does it, when and how it is managed. But the cost of not doing it properly - or not doing it all - can be astronomical.

    Mobile: We have finally reached the tipping point

    For the last twenty years, at least, mobile specialists have been claiming each year as the ‘year of the mobile’, but now with the volume of mobile traffic overtaking that of desktop and showing no signs of slowing down, it seems its significance for every website owner and digital retailer can no longer be exaggerated:

    Global mobile penetration_infographic

    Sources: GSMA, PixelUnion

    Yet, still mobile devices lag behind desktop when it comes to quality of user experience, conversion rate and basket abandon -

    Mobile testing_conversion rate optimisation_infographicSource: Baynard

    At the same time expectations are sky high:

    • 57% of internet users say they wouldn’t recommend a business with a poorly designed website on mobile

    • Google research shows that 73% of consumers will switch from a poorly designed mobile site to one that makes purchasing easier.

    • 59% of smartphone users feel more positive towards companies whose mobile sites or apps allow them to make purchases quickly.

    • 85% of adults think that a company’s website when viewed on a mobile device should be as good or better than its desktop website.

    But aren’t mobile and desktop fundamentally different? Shouldn’t we expect different results?

    Yes, browsing and buying behaviour is different on mobile devices, with evidence of more ‘window shopping’, ‘show-rooming’ and idle browsing leading to a greater proportion of basket abandons and lower conversion rates. This might be particularly the case in high value sectors, where large purchases may require more research or physical interaction with a product before a sale. Even so, mobile visits can be an integral part of that process and can’t be ignored.

    But the dramatic differences in conversion rates may be better attributed to poor design and the quality of subsequent testing than any inbuilt aversion to completing a transaction on a mobile or tablet.

    Build it, optimise it and they will buy

    Years ago, at the start of the mobile revolution, no one thought consumers would be at all interested in making large purchases on a ‘tiny device’. When Marks and Spencer launched their first mobile website in 2010, the press and the company themselves)were deeply impressed when someone bought a sofa using their iPhone. But the presence of the mobile didn’t create the demand for a mobile purchase of the product, it was just suddenly possible to do so quickly and easily - so customer’s did.

    Opportunities missed through poor quality mobile experiences

    There is opportunity for growth through mobile research and purchase in nearly every sector, it’s just the customer experience on the device has to give them confidence, match their expectations and actually work (in spite of their potential complexity). This article about the mobile and the travel industry makes this point really well.

    And in this competitive environment, if you’re not making the effort and optimising your experiences someone else will:

    According to research published in Forbes, 51% of mobile customers have made a purchase from a company other than the one they originally intended to

    All this points to one thing - there’s never been a more crucial time to get your website’s mobile experience right - and that really depends on adequate functional, performance and usability testing on a range of devices.

    Why is it so difficult to conduct meaningful mobile testing?

    Here’s the rub.

    There is huge complexity in the mobile landscape, that means it’s a real challenge to test all the possible combinations of device and browsers adequately,

    Right now more than 4 billion people are accessing the web using more than 9000 distinct devices, 21 different operating systems and 8 major engines.

    And that’s where revenue keeps falling through retailers’ fingers. You need specific skills and insights to understand mobile behaviour, and you need access to the actual physical devices to see what’s happening to consumers on the ground.

    With a fragmented device landscape the evidence of a problem in a purchase journey affecting specific browsers and devices combination can be so buried so deep inside your analytics you cannot detect it.

    Here are just a few of the ways mobile journeys can go wrong - with customer journeys splintered and distorted by different screen sizes, browsers or device types:

    • A purchase journey on a tablet that doesn’t work in portrait mode but only in landscape
    • Images and forms resizing in strange ways on devices and stopping customer journeys in their tracks
    • A mobile browser that sends you back to the top of an order page when an edit is made anywhere on the page
    • A convoluted payment journey with forms stretching way down a page or forms that are impossible to fill in on some browsers
    • Different demographics behaving differently on mobile - an in house QA team can fail to reflect those differences effectively in their testing

    And on top of these functionality and usability issues - you also may have a problem with site speed - as the following mobile page load stats demonstrate.

    Mobile page load and bounce stats

    Mobile testing_mobile page load times vs bounce rateSource: Pixelunion

    Not being able to see and test for site speed across various devices (with wifi, 3G and 4G connections) is the downfall of many developers as they try to optimise site performance for mobile.

    So, what’s the cost of doing mobile testing properly?

    Could you do it yourself?

    Most brands have given up on the idea of curating their own device labs. Some estimates say the cost of buying and maintaining a device lab of a minimum of 30 devices could start at £20,000 a year - rising to a six figure sum at the more elaborate end of the scale - and that’s before you start actually spending the time conducting any testing. Just searching for, maintaining, upgrading and buying new devices is a challenge in its own right. If you’ve got a global market then your ability to test on the ground and with the payment mechanisms common in those regions may also be severely curtailed.

    At the same time, emulators are famously unreliable and even if you choose to use a cloud based reservoir of devices you’re still not effectively reproducing a real world experience. They can’t successfully emulate the hard/soft keys combinations that characterise many peoples’ experiences, nor the actual physical restrictions of holding a device in your hand and swiping or typing with a thumb or finger. The cost of missing those subtleties in your testing could be countless functional and UX bugs that you just can’t detect and fix.

    The resource challenge still looms large

    And the huge time and human resource challenge of doing it all properly still looms large. Particularly, these days when there is so much complexity in the user journey to test across so many devices. The list of mobile innovation (and the potential for error) goes on - personalisation, AR, new payment channels - all these need to be tested thoroughly if a big release isn’t going to fall flat.

    IBM found that even when in-house testing teams had the skills and equipment available to them, they often had more testing activities to do than they had time to complete. Nearly half of testers in their survey said that having enough time to test mobile products was one of the biggest blockers to their productivity.

    Even if your team has access to the devices (or a proxy for them), then how do you translate the results of any tests they undertake into a format digestible for the whole business, arranged into actionable and prioritised fixes?

    The costs that in-house mobile testing represents in time, resource and effort - may really challenge your ability to deliver required results in anything like a cost effective way.

    But the dilemma remains - if inadequate mobile testing and optimisation can ruin your revenue and reputation, how can you make your approach to mobile QA achievable, fit for purpose and fit your wallet at the same time?

    Website Testing 2020 full width CTA

    You may also be interested in... Seeing A Decline In Your Mobile App Downloads? 



    Accelerate your conversion rate in 24 hours-1

    Accelerate Your Conversion Rate In 24 Hours

    Make the impossible possible with in-depth website, app testing and usability insights.