Brands often want to automate regression tests to save their business time, effort and money. And this makes sense, since these tests are often repetitive and routine in their nature, eating up resource that could be better expended elsewhere.
But a truly automated regression testing strategy, where, at the touch of a button, tests are run with prioritised, actionable results flowing instantly through to a digital team is often an illusory hope. The reality can be much more messy, expensive and time-consuming than this
When automation is a drag on commercial decision making
Automation testing often starts with a developer coming to the process from a ‘unit test’ perspective.
As developers build out a website, they’ll create their own tests to check their syntax and code is standing up to the new changes they’re making. When a site is released, it’s these kinds of regression tests that can simply end up being automated - spitting out technical results that the commercial side of a business then need to act upon. Read the Website Testing 2020 guide.
In these cases, results from automated tests have to be interpreted for general consumption within an organisation, as the e-commerce teams work out what fixes it wants to prioritise. All this can eat up any gains in time that have been made from automating the tests in the first place.
Lost in translation between technical teams, business analysts and digital decision-makers - unmediated regression test results can require extra effort to explain and understand.
When agile out-sprints the pace of automation
The reality is, it’s a complex and time-consuming process to automate tests. And the number and type of possible regressions you could be experiencing just keeps expanding as you release and add new functionality to your site.
If a typical sprint cycle for an agile company is now two weeks this presents a new kind of pressure for those automating and running regression tests. After testing has been done manually at the end of the first sprint, by the time those tests have been automated they won’t be relevant to the whole site anymore. In other words, you could be breaking things in sprint two and three, that you haven't regression tested.
As the scale of regression testing grows with each sprint, test cases need to be maintained and updated to stop them becoming obsolete, unmanageable or otherwise ineffective.
And the challenge of maintenance obviously becomes more intense when regression testing is automated. In many ways updating, running and evolving regression testing manually can be a more flexible, agile and responsive method than trying to automate.
When pass/fail results don’t give you the whole picture
Automated testing is valuable for testing against scenarios on your website or app that are well established and typically unchanging, or those that are particularly prone to erroring. In these cases, automated tests can very quickly give you the pass/fail result that you know you can rely on so that you can conduct further investigations and direct engineering resource to implement necessary fixes more rapidly. But the more complex the scenario an automated test is expected to check, the more unreliable and flakey the results of that test might start to become.
What’s more, automated tests can’t necessarily give you the whole picture of what’s happening on a site that is continually changing. A test can be run to tell you if a check out button is present on a page, but it can’t note that the check out button is now in the top left-hand corner when it’s meant to be on the right.
That’s something a human can spot straight away, a regression that manual testing would be able to note and draw to the attention of a business, without necessarily halting a test altogether.
When humans make the difference
Automation often exists as a testing strategy because it seems the only alternative where a manual function is too expensive or too slow to provide the coverage and speed you need to check for regressions.
But manual testing agencies like Digivante can now provide the speed and scale you really need for regression testing results that you can really rely on. That means reducing the time taken to perform regression testing manually down from days to hours. Of course, there may be elements of automation involved in these solutions, carefully managed and maintained tests that bring more efficiencies, underpin insight or the effective management of manual resources.