Systematic, ongoing website testing can help you trap, identify and eradicate the bugs, usability issues and other digital performance problems which harm your bottom line. But should it be an automated or manual website testing strategy that you deploy for maximum effectiveness?
The need for speed
Automated website testing appeals to our desire for speed, consistency and repeatability in the way we check our applications. Why use flawed humans, the argument runs, when they have only limited capacity, can become distracted easily and are bound to miss, sooner or later, some vital issue or other? Automation beats manual testing for cost-effectiveness, ability to scale and reliability every time. Or so they say.
The human touch
On the other hand, manual testing brings the reassurance of the human touch, the unique insight of real-world testers working on real devices. And what it lacks in its ability to scale rapidly and cheaply, it makes up for in the depth and quality of the insight it brings. For example, the ability to intimately understand on a human level where and why a particular feature of a customer journey is failing, or where a customer journey on a specific browser is leaking revenue.
But manual testing can be a lengthy and time-consuming process and some would argue this makes it unfit for an agile world of continuous deployment and regression testing.
So what are the strengths and weaknesses of each approach?
Strengths of an automated testing strategy
- Automation tests offer the ultimate in consistency and repeatability. Running an automated test procedure allows you to replicate and execute the same kind of testing and repetitive tasks multiple times
- You know you are always testing for the same thing against the same criteria, time and time again, so you can be sure when something has ceased to work in the way it should
- Once test scripts are configured, test automation requires less time and resource to execute compared to manual testing
- A computer can follow a sequence of steps more quickly than a person, and it can run the tests overnight to present the results in the morning
Weaknesses of an automated testing strategy
- All automated testing can do is alert you to changes of patterns, alert you to the fact that the script you ran has passed or failed, you still have to have the manual testing resource available to replicate the test and identify the specific issue
- The software required to run automated tests can be expensive
- Automated testing scripts may not always be most reliable at spotting real issues. They rely on recording sequences of mouse movements and keystrokes, then playing them back to see if the user interface responds, in the same way, every time. These recordings may not work properly when a button is simply moved or relabeled in a new release. A test may fail even though a program is functional
- Automation testing can’t change course in the middle of a test run, to examine something that has not previously been considered
The strengths of manual testing
- The breadth and scope of manual testing options - Manual testing offers different kinds of testing, promising levels of detail and quality that can’t be easily delivered by machines. Accessibility, security testing, localisation testing, functionality testing. The list goes on. Humans are obviously more versatile and intelligent in the type of tests they can run and the results they can deliver
- Manual testing is better for functionality and interface testing. A machine can’t tell you when a final user experience is muddled and illogical, a machine can’t conduct exploratory testing if it thinks there may be an issue lurking beyond a scripted scenario
- Manual testing simulates real user experiences in the way machines just can’t
- Manual testers can handle more complicated use-case scenarios than machines
Weaknesses of a manual testing strategy
- Writing test scripts to be followed by manual testers and collating responses for further action can be time-consuming
- Supporting a full-time QA team or bringing in testing contractors to work on specific projects can be expensive
- Humans are fallible, humans can miss things
- It is difficult for a small test or development function within an individual company to have access to the range of devices and browser types, as well as the time and resource available to thoroughly test big releases
- The amount of manual testing resource a single company has at its disposal against the speed of new releases and the fragmented world of device and browser penetration means it is practically impossible for a traditional team to test adequately. “Delay - or push and pray’ have become default settings for some, presenting further risk to revenue and reputation
For those experiencing the squeeze on their time and resource, automation testing might seem a way to get some peace of mind that things are working as they should be.
Experts know that humans win over machines every time when it comes to the creativity and understanding they bring to a testing scenario. As even automation testing guru Angie Jones admits:
“Automated scripts are checking known paths for expected results. That’s not truly testing. Testers discover the unknown, and this skill is still very much needed, whether there’s automation or not. Without human intuitive exploration, a team may be blind to their most expensive bugs.”
And that’s just one of the ways in which ‘community testing ’ can come in.
The wisdom of crowd testing
Crowd testing or ‘community testing’ as it is also known, applies the wisdom of global experts to the business of website testing. With a huge community of professional testers available, the volume of responses you can achieve for each testing scenario will not only quickly identify bugs and issues, but give you real, human insight into how a majority or segment of customers are likely to behave when confronted with your user experience.
Human insight at speed and scale
The best agencies with tens of thousands of highly professional testers spread across continents and countries ready to respond 24/7, 365 days a year, really give the lie to the idea that scale and speed is only the province of the automated testing world.
And add to that the human ability to judge, deal with complexity and behave in their tests exactly as your customers would, then manual community testing starts looking like a more cost-effective and reliable mechanism than many others.
Real devices and real people reacting in real-time within real customer journeys - that’s the promise of a community testing strategy done right.
And the right manual solution can run the gamut of testing practice. Localisation testing allows you to tap quickly into resource in new territories in a practical and cost-effective way. Community testing can offer a quick and easy way to segment testing against ‘real world’ audiences based on gender and age requirements. It can offer functionality testing against prewritten scripts as well as a more agile, exploratory testing services.
The sophistication of automated testing is increasing all the time and is often a requirement for effective bulk performance and stress testing. But as we move towards a more agile world of eCommerce development and deployment, with increasingly innovative and complex software offerings, community testing can offer the speed, scale, range and quality of tests that are really equal to this challenge.