What is functional testing in software?
Functional software testing is the discipline of enumerating application requirements into a set of test cases which can be executed to verify an application functions as a user expects it to. Functional test cases can be considered documentation of an application’s uses.
Test case priority should be established by business users early on. The level of priority normally represents the order that work is done. Lower priority work is started after the highest priority work is completed.
User acceptance tests (UATs) are a subset of high priority functional tests executed by end-users. The User Acceptance phase is the pinnacle of functional testing efforts. This optional phase is useful when handing off an application to another company or group. After development work is complete, the UAT phase validates the application before deploying to production. The product’s quality has been evaluated by previous functional testing and high priority issues should already be fixed. This phase is an opportunity to submit last-minute user experience bugs.
Regression tests are created from the set of functional tests. Regression is when a bug is introduced inadvertently by code changes such as patching or refactoring. A test run is executed on existing functional tests in order to verify no new bugs were created. Considering that software updates are more ubiquitous, it is becoming harder and harder to avoid some regression bugs being introduced on accident. Mitigating this eventuality requires constantly testing the product.
Functional tests are not the same as unit tests. Unit tests are created to test how one unit of code behaves during development whereas functional tests how an application behaves during use. They represent different perspectives and should be considered separately.
Unit testing, accessibility, penetration, performance, integration, globalization, localization, and fuzzing are all examples of non-functional testing because they are not based on users interacting functionally with the application. They are all important for different reasons, though, and test engineers should know the difference between them.
Role of business users
Functional testing increases confidence that a product is relatively free of defects that directly impact its expected behaviour. The original source of information for functional tests should be the end-user requirements as defined by business users plus error conditions and boundary cases. These can be found in documentation or overheard during design meetings.
For a business stakeholder, functional testing is important because it correlates business needs to application tasks or workflows. Business users often define the application in a specification or a set of Use Cases, for example. Test cases can be extrapolated from these artefacts. Also, a tester may interview business stakeholders or developers about the application.
When to perform functional testing?
- Engineer analyses story requirements to create test case statements. They relate only to the features being worked on in the current sprint
- Test case statements can be reviewed with a business user such as a Program Manager to verify test coverage meets their expectations. They should provide input about priority in case the engineer does not have enough time to test everything
- Test cases created from statements and organized according to category, priority, and logical flow. Test steps can be stubbed out at this point
Once feature work is deployed to a test environment and considered ready for test:
- Exact tests steps are enumerated and manually executed on the new features. This allows for quick identification of defects
- Close resolved bugs after verifying fixes
- Complete test automation until all automated tests pass
Agile software teams are expected to output a fully tested and deployable release at the end of every sprint. Passing functional tests can demonstrate the new features to stakeholders at the end of the sprint.
Another method known as Test-Driven Development is possible, but rarely used. Instead of creating the feature first, a failing test is created and feature work is done until the test passes in automation. This automation-first mentality is difficult to accomplish and developers usually resist it. If done correctly, though, a continuous development pipeline almost builds itself.
Software development life cycle: Waterfall
Other methodologies like SDLC waterfall will begin with smaller test efforts, then ramp up once a release candidate is created and deployed to an environment. Finally, it will taper off after release with a small team left to address in-flight maintenance bug fixes. Unfortunately, the phase to perform most functional tests tends to be right before release when the greatest pressure is on. More defects tend to make their way to production if the quality is impacted by a major release deadline. Also, bugs tend to overlap and hide each other if left unnoticed until the correct phase begins.
Being able to continually develop, deploy, and test is the peak of efficiency for software development teams. Automated functional tests can be run post-deployment to act as a quality gateway during code changes. A set of robust tests are chosen and added to the software pipeline once deployment to a test environment is automatic. This software development pipeline is similar to Henry Ford’s Moving Assembly Line. The more quality control added to the line, the more consistent the product will be.
The output of a test pass is valuable to many people. Failures need to be evaluated to determine if it is based on code or a broken test. Failures based on code should generate a defect report or bug. Defect reports are best tracked in some kind of database. At the very least, they should have a succinct title, exact steps for reproducing the issue, applicable logs and screenshots, and the expected behaviour vs the actual behaviour. Reports should be free of any unprofessional language or bias. The user story and test case can be linked to a defect in some systems. The data can be aggregated and visualized using reporting tools.
Notifications from a database are usually filtered and possibly overlooked in email. On some teams, an email from a tester with a link to the report is sent as a courtesy heads-up to the developer. This personal touch helps to facilitate a quicker fix by smoothing the interaction between a tester and a developer. The developer will know who to ask for more information and the tester has brought the issue to the developer’s attention in a socially acceptable way.
Why use community-sourced functional testing?
There are many ways crowd-sourcing functional tests can help improve software. Consider how important third-party verification is for the production of automobiles, household appliances, or erecting a building. Also, consider the complexity of devices and desktops that an app can serve and how crowd testing can fill the gaps in a complex test matrix. Community testers represent a diverse approach to quality assurance.
Community testers are motivated to discover bugs for other reasons than a cash reward. They achieve status in the community and awarded with a coveted position on a leaderboard. They operate within a framework that already handles reporting, tracking, and resolving issues while respecting intellectual property rights and non-disclosure. Furthermore, they represent the best of all of us: experts willing to help a stranger with a problem.
Tips & best practices
Here are some tips and best practices related to functional testing:
- Communicate effectively: The best testers will speak up about something that does not seem right. Identifying issues requires engineers to be willing and able to communicate effectively with diverse audiences. Managers should be willing to display an open, honest attitude about issues.
- Test early: Create functional tests as soon as the requirements are discovered. This is useful on Agile teams when features are created rapidly during the Sprint.
- Test often: Test during development. Automate repetitive tasks so that tests can be run on a schedule, on-demand, or on integration. Automate deployment so tests can be executed sooner.
- Organize: A test suite, or collection of tests, can grow quickly as the product matures. Organize test cases in a central repository that provides templates for test cases, defect reports, features, etc.
- Be frugal with time: Consideration should be made to which tests to run, in which order, and how long each test takes. Execute the least amount of steps to accomplish the goal.
- Maximize test efficacy: Choosing the least amount of test inputs to gain the greatest test coverage is an art. Do not create redundant tests covered by unit testing. Do not attempt to test every value or boundary case; instead, choose one or two inputs for each category.
- Create a regression test for unexpected bugs: If an unexpected bug is encountered and it lacks one, create a specific test case for it. Add it to the set of regression tests.
- Avoid UI Automation: It is not recommended to automate functional tests using UI automation. UI automation is too easily broken because browsers and systems are always being updated. Use community testers to fully test the UI, instead.
- Tester tip: Keep a test journal in a document updated with screenshots and notes during testing. This information often proves useful.
In conclusion, functional testing is the cornerstone of software quality assurance. Without functional testing, there is little trust an application behaves as expected. Crowd-sourced testers can ensure that it is fully tested. The online tester community has already changed the industry forever by allowing greater peace of mind for any business-wise enough to take advantage of it.
(This article was written by one of our key community members, Christopher Canova. Christopher is an accomplished IT professional, with over a decade of experience as an enterprise software development engineer in test (SDET) in the Pacific Northwest. He also enjoys crowd-testing or hunting bug bounties online as a security researcher.)