Crowdtesting is a QA practice which first sprang up from the Open Source movement in software engineering. It has since become a commercial product and industry in its own right, bringing mass website testing to the mass market. But not all of these solutions are created equal.
What is Crowdtesting?
Crowdtesting once simply described a way of accessing a large volume of ‘volunteers’ - a pool of people willing to evaluate various aspects of digital products in return for financial or other rewards. Crowdtesting now describes the offerings of vendors in which large groups of individual testers from around the world, can be deployed in a flexible way to deliver app, website testing services at speed and scale.
The perception of the value of crowdtesting may have suffered by its association with some of other ‘crowdsourcing’ practices - the improvisational spirit of ‘open source’, the harnessing of free (or nearly free) non-professional resource and brainpower to solve creative and other problems.
And there’s no doubt that some of the first commercial website and app crowd testing solutions, could have a patchy reputation for delivering quality. Designed to offer a quick and cheap way for companies to conduct tests at scale with ‘real users - the users themselves and the reporting tools that clients were given to see their results, would offer varying degrees of value to businesses.
The reports submitted to companies may have picked up small or more trivial bugs, while ignoring graver, more complex issues. The way crowd testers were paid or incentivised would have obvious implications for the quality of their output. These payment practices can still have a significant impact on the standards and thoroughness applied to the crowd testing of websites.
As crowdtesting enterprises continued to evolve into more mature companies, some brands have still shied away from using these kinds of solutions because of:
- A perceived lack of control over the quality of work undertaken without their supervision
- A suspicion that output will not result in real, actionable insight
- That the output from tests will not be communicated effectively
- That they can’t work in tandem with an internal test function
- Lack of security over IP and other issues
Some outsourced solutions have come of age
But many outsourced solutions that use communities of testers have really come of age - as mature and trusted partners to brands around the world.
The best crowdtesting communities may now be understood as vast curated resources of highly skilled professional testers representing a wide demographic of users, that can be deployed rapidly and at scale to provide a huge array of testing services.
They can offer testing services that are impossible to deliver in any other way, and at cost, that is accessible to growing.
Outsourced testing is a vital part of the digital landscape
The logistical challenge of testing is itself enough to convince brands not to try to conduct all their testing on their own and to outsourced solutions.
Digital and development teams are getting leaner and smaller. Release cycles are getting shorter. There is more complexity to test on a greater variety of platforms, on an almost infinite variety of browser and device combinations.
In a globalised marketplace, the opportunity and temptation for smaller brands to launch globally is intense, but the challenge of providing QA without a real-world testing presence on the ground is huge.
Pulling in extra contractor support is an expensive way to bring more testing support into a business and is not necessarily possible or practical to support new overseas ventures.
Outsourced, a professional testing resource can make this kind of large scale testing power available to even the leanest, most agile, challenger brands. Thousands of testers can be deployed quickly and efficiently to test with real-world devices, and with the mentality of humans not machines.
Not all solutions are created equal - what provider is right for you?
But, of course, not all of these solutions are created equal. Before you consider using a company offering ‘crowdtesting’ services you should consider the following:
- What are the range of services they offer? Functional and non-functional tests at scale, human performance and security testing? Can the full range of devices and devices be deployed to test your site or app? What countries do they cover? Can they offer larger-scale testing solutions, with ‘real world’ user testing (reflecting the device and user habits of specific demographic groups). But can they also deliver smaller scale, agile teams capable of sophisticated, 24/7 exploratory testing to match the pace of businesses in a state of continual deployment?
- What are the analytics that underpin their offerings? Does real-world data insight from your website feed into their recommended testing strategies, highlighting and isolating potential quick wins and immediate fixes with specific revenue opportunities attached to them?
- How are their testers paid and incentivised? What kind of confidentiality clause govern their relationships with them? Are they enforced through robust, three-way contractual protections between the customer, the testing organisation and the testers themselves? Some organisations do not just utilise freelance testers, but also have large teams of remote testers that are permanent employees and can, therefore, provide full professional indemnity insurance protection.
- What will the output of this testing look like? Will it include screen-shots and video of issues and bugs as testers encountered them? Are the results, corrective recommendations and insights understandable by non-technical teams and truly actionable for a business as a whole?
There’s nothing wrong with many traditional crowdtesting solutions, but they’re not enough for some companies who are now looking to outsource all or part of their testing function to match the demands of a hypercompetitive marketplace. More and more businesses need truly agile, strategic solutions that cover a deeper range of testing services, but in a cost-effective way only possible through the ‘crowd.’
This, of course, means usability and performance testing at speed and scale, but also ongoing exploratory testing conducted by expert, professional test teams around the world - 24/7, 365 days a year. These testing solutions can be flexible, collaborative and bespoke when needed, while always supply highly consumable and actionable insight that fix issues and grow revenue.