Problem – How can you develop a robust regression testing strategy?
Solution – We walk you through every step along the agile testing strategy path, outlining objectives, risk and customer experiences.
Read Time – 7 Minutes
If Agile development requires a change in organisational mindset to deliver faster, better quality websites and apps, then it requires a change in your testing mindset, too.
An agile regression testing strategy needs to be an integral part of your agile development process to deliver on all its promised benefits. It should be part of an overall testing approach that is as flexible and responsive as possible. Adapted to delivering fast, actionable, customer-focused insights, and in the words of the Agile Testing Manifesto - ‘satisfying the customer through the early and continuous delivery of high-quality software.
What does an Agile testing strategy look like?
Agile testing should be a collaborative practice that occurs continuously, from project inception to software release and beyond, supporting frequent delivery of value for customers. Testing activities focus on building quality into products, using fast feedback loops to validate understanding of bugs and issues. These practices strengthen and support the idea a team is collectively responsible for quality.
Agile objectives - faster iterations - better customer experiences
The ‘Agile Manifesto’ argued for the constant iteration and frequent delivery of a working product based on customer feedback, to match the pace of market expectation and respond rapidly to changing user needs. Working in this kind of highly flexible and responsive way reduces the risk and associated costs of product failure.
Consequently, most digital businesses now deploy these development techniques. In fact, the 2019 State of Agile report states that 97% of businesses they surveyed are practicing agile in one form or another, while 35% say their developers are now working in cycles of ‘continuous deployment’.
Agile DevOps and a policy of continuous deployment is particularly suited to brands in disrupted sectors, who need to constantly shift and alter their digital plans to respond as rapidly to new commercial threats and opportunities. For some bigger brands, this has resulted in the embrace of ever-shortening sprints, and, increasingly, an environment where hundreds of smaller releases of new features are happening every day.
For many, this long term shift from a ‘Waterfall’ world of meticulous planning, where every last piece of functionality is specified and delivered precisely against the specification along with a whole gamut of major changes, to a world where teams of developers concentrate on incrementally delivering individual pieces of functionality presents a particular testing challenge.
The risk of constant change in web applications
Now changes are constantly being made to website and app code in swift succession, there is a greater risk that developers will unwittingly, introduce bugs elsewhere in an application. For this reason in agile, frequent regression testing is necessary to ensure your site’s baseline structure is constantly stable, regardless of the updates or changes that are being constantly being made.
But what is the best way of delivering regression testing as part of an agile strategy?
Should you automate regression testing?
Regression testing is notoriously hard to automate since scripts are necessarily long to cover the functioning of an entire site. And machines obviously cannot respond to nuance. If a human is testing an application and finds it behaves slightly differently from what is expected in the script, they can note that down and continue the test. But in these circumstances, an automated test could fail and create further unnecessary delays and investigations.
Breaking down a long, automated test into a series of shorter automated tests is one option for containing the potential of these kind problems, but that can just end up creating more work for an organisation. In regression tests humans can find workarounds, machines will just stop working. In the spirit of Agile, which seeks to prioritise the rapid release of working software, humans can exercise skilled judgement about whether a website or app is operating as it should be, whereas a machine simply can’t.
Manual regression testing - the pros and cons
Manual testing options can be more responsive and nuanced than machine regression testing and be better suited to the demands of an agile release cycle.
But the challenge for smaller teams with large digital real estate to test for regressions is how to deliver these tests with the necessary speed and scale. And if releases are happening on 365/24/7 basis this problem and the level of risk of failure involved is further exacerbated.
A robust smoke and sanity testing process should be a part of an agile testing strategy.
To conduct smoke testing you need a subset of test cases which can quickly test primary and core product workflows in staging. This will give you an immediate view of whether a release is so flawed you should not proceed or waste time on conducting deeper, more thorough regression testing.
Being able to regression test thoroughly in pre-release is ideal. But this requires an environment that exactly mirrors the conditions of live. However, not every brand or testing agency has the facility to do this.
It’s an expensive mistake to release updates that introduce bugs elsewhere into your website or app, but the more rapidly you iterate your offerings the more likely this is to happen. Undetected bugs and issues cost you money and reputation, and they famously cost more to fix the longer they go undetected.
In agile environments, ongoing testing is vital to ensuring the continuing stability of a website and app. The regression testing function, therefore, should be dedicated in the first place to identifying the issues that are making your application unusable - with a way of quickly and automatically feeding back priorities for fixes and resolution.
Professional, manual testers can be bought into to provide the speed and scale of response you need to ensure this happens, with the range of devices you need to provide useful and targeted feedback.
But true to the spirit of agile, they need the right set of skills and tools to feed effectively into a cycle of continual, bug-free release. Automated tests can be part of a regression testing strategy, but manual testers can bring the nuance, judgement and collaborative spirit that makes the process much more flexible and responsive.