Testing is an often under-appreciated part of the software life cycle. Timelines will shift, there will be delays with the development and the testing phase is the part the suffers. Here we have compiled a list of tips to help increase the productivity of testing and to help avoid releasing bugs into production and keep a high level of quality in the software released.
The ten tips have been divided into the following categories:
- Technical Tests
- Configuration and Deployment
- Other
See also the post on Ten Tips for Testable Websites for Software Developers, aimed at developers.
1. Technical: Load test early
Load testing should be performed as early as possible during development. This will ensure there will be time to fix any errors uncovered from the testing. Aim for an early load test at half-way through the development plan; you might need to completely re-work your design if the early tests show that the original architecture is not up to scratch.
See here for details of how to calculate HTTP server performance under load.
2. Technical: Test performance on Production-like environments early on
If the load testing is not performed on a production like environment it is extremely hard to extrapolate the results to production. If (as is often the case) there are financial restrictions on a decent pre-production environment, consider using the actual production servers before go-live, or out of hours
3. Technical: Be thorough with soak testing
Soak tests are usually written for the most common user journey; however, this doesn’t that into account the other various issues that can be encountered. For example, is your code releasing resources under error conditions? During testing on a recent project we discovered an awkward bug in some code we inherited from another company whereby if the application was forced to return an HTTP 500 code after 20 requests, the database connection pool would be totally consumed and not allow any other access to the site until the web server was restarted, due to database connections not being released correctly.
Make sure you test for foul-weather conditions, not just fair-weather.
4. Technical: Test on Production-like environments
The Production servers are running Operating System [ABC], web framework [STU] and widget [XYZ], so we don’t need to test on Production because we’re running all of those components in our Dev and Test environments, right? Wrong! What about subtle but crucial issues such as CPU architecture, default paths, configuration values, redirect behaviour and cross-machine interaction behind firewalls?
As a heuristic, allow at least one quarter of your total testing time for resolving issues specifically related to the Production infrastructure. This means that if issues are discovered when deploying to production a large number of potential causes for issues can be ruled out.
5. Technical: Test for concurrency
Any resource or object that can be access by two users at once needs to be tested for concurrency. This should be done both at a code level and also using stress testing. When resources are scarce, certain classes of bugs are more likely to manifest themselves.
6. Config: Verify where settings are coming from
Developers have been known to hard code config values when debugging which can cause major issues when deploying to Production. It should not be assumed that entries in the config files are actually being used by the application, and therefore need to be tested and verified.
7. Config: Deployment process and verification
The process of deploying code to production needs to be well understood. Who does what where? What services need to be running or changes need to be made to config? It doesn’t matter how well the code is written or tested, if it’s not configured correctly it won’t work.
DevOps tools such as Puppet, Chef and Cucumber can help with the deployment and confirming the application is running correctly, but you don’t need to automate deployments fully to bring them under control. In particular, instigate post-deployment checks using bespoke tools or scripts which can validate the config files and initial state of the deployed application, database, etc.
8. Other: Check image transparencies
Incorrect image transparencies are a pain to test for as they must generally be tested by a human (by eye). A quick way to do this is to change the background colour of your browser. This will highlight any images that have white (rather than transparent) backgrounds.
9. Other: Test in different locales
Issues with numbers and dates can occasionally only be apparent when different locales are used, and as such these need to be tested, especially if the server hosting the code will be running in a different locale. Best practice suggests you would run the server in UTC and calculate timezone offsets from there, but that is not always possible.
10. Other: Automate wherever possible
This might be obvious, but automate whenever you encounter a frequent, repetitive activity. This includes builds, deployments, tests and anything else that is a frequent repetitive process. This means you will not need to spend time doing the same task on each and every release and can spend more time trying to break the new functionality which will lead to more robust code.
Some activities are fine to keep manual, especially if they are infrequent and complex to automate, but where possible, use tools such as Team City, Selenium and JMeter to help.
The Tester Mindset
And finally, here are three general rules that all thorough testers that we have worked with seem to follow:
- Don’t trust anyone, and be sceptical of everything.
- Treat all code as broken until proven otherwise.
- Because someone has told you they have validated all input doesn’t mean they have or have done it to an acceptable level. Make sure to prove that input is correctly validated and processed.