Introduction

The world of software development is full of different names for different types of server-side environments used throughout the development lifecycle. “Dev”, “Test” and “Production/Prod” are the most common triplet in use but we also have “UAT”, “Staging/Stage”, “Pre-Production/Pre-Prod”, “Production”, “Live” and so on. The terms are poorly defined and inconsistently used across the industry which leads to ongoing confusion and debate, particularly at the start of a project or when working with a new team, about what environments are needed, what their purpose is, and how each environment fits within the broader software delivery process.

Some time ago we tackled this problem at Priocept by standardising on our terminology in this area and creating formal definitions for each term. In this blog article we are publishing the pattern that we use when it comes to software environment naming and usage, to help others achieve similar standardisation and clarity.

Dev, Test, Prod

As this is the most common base level of terminology, let’s start here.

1. Development (“Dev”) Environment – As the name suggests, this is where initial software development takes place. However, there is often confusion or ambiguity about where this environment, or environments, are located. Are we talking about local development environments, running on individual software developers’ machines? Or are we talking about some kind of centralised development infrastructure that is used by all developers, prior to promoting software to Test?
2. Test Environment – As the name suggests, this is where the software is tested. But who is it being tested by at this stage? The development team? A project/product manager? Or the end customer?
3. Production Environment – Hopefully not too much debate with this one. This is the final environment that we are targeting, where the software will be used “for real”. So this is the environment at the end of the chain of environments, and where everything must work properly, must perform adequately, must be reliable, and must have suitable protections in terms of security, backup and change control.

This is a good start but already we have ambiguity around what exactly we mean by Development Environment, and the scope of the testing that should be performed on the Test Environment is unclear.

Local vs Centralised Development Environments

To address the first problem, Priocept introduced an explicit concept and definition for Local Development Environments:

Local Development Environments are used by individual developers, in complete isolation from other members of the team.

These environments will typically run directly on an individual developer’s local machine, but this need not be the case. They are “local” to an individual developer’s own working environment, but may not be physically located on their laptop or desktop machine, although this is usually the best approach for performance reasons.

The key attributes of Local Development Environments are as follows:

1. They are isolated from other team-members’ work.
2. They are the initial point at which software can be run, tested and debugged.
3. They are independent of external systems and should be usable without any dependencies or connectivity to the outside world. In practice, this often means that the development framework will need to include mocking of external services. Ideally it should be possible to do productive development on this environment even if on a plane, a train, or camping out in the woods with no internet connection.

If we formally define the development environments used by individual developers as Local Development Environments, then we then can use the term Central Development Environment to identify a singular, centralised development environment that might have otherwise just been known as “Dev” or “Development”.

The Central Development Environment is the second point at which software can be run, tested and debugged, after the Local Development Environments, and the first point at which it can be tested in an integrated fashion by team-members other than the developers.

The key attributes of the Central Development Environment are as follows:

1. It is shared by multiple developers within a team.
2. It runs on server-side (centralised) infrastructure such that it is permanently available and independent of individual developers’ local machines. Therefore it is usable even if one or more developers are on vacation or in a meeting.
3. It introduces system dependencies that may have been removed or mocked within Local Development Environments. Since the Central Development Environment is running on server-side infrastructure, whether an on-premise datacenter or cloud based infrastructure, it should be able to provide network connectivity to downstream services, such as web service endpoints, that may not be available or desirable to use within a Local Development Environment.

As a result of these characteristics, the Central Development Environment is also a form of “integration” environment, as it allows for the testing of both the integration between different team-members’ inter-related code, and the integration with external systems that the software under development relies on.

Note that like the Local Development Environments, the Central Development Environment is used only by the software development team. It should never be used directly by business users or end customers, and ideally should not even be accessible to them. However, this environment may be used by technical members of the team beyond the software developers, such as project managers or testers who need early access to the latest software to review or test it prior to making it visible to business users.

Although the Central Development Environment is hosted in a central location so that it is always available, independently of individual developers’ own machines, no guarantees will be made about the availability or reliability of this environment. The environment exists primarily to support the developers in their work, and as such as they should be free to deploy to, and if necessary break, this environment whenever they wish. Obviously this should be done in coordination with other members of the development team, but should not require prior agreement from a project manager or business user.

Test Environments

The Test Environment follows on logically from the Central Development Environment. It is centralised as per the Central Development Environment, in that it runs on server-side infrastructure that is available to all team members, but since there is no need for the concept of a “local test environment”, the distinction can be dropped from this point forward.

The Test Environment is the first environment to which business users are exposed, and on which they can test and provide feedback on the software under development.

Key attributes of the Test Environment are as follows:

1. It has all the attributes of the Central Development Environment.
2. It is owned by and is the responsibility of the project manager, product manager, delivery manager, or project team leader – not the developers.
3. It is used by business people or other individuals outside of the software delivery team, for the purpose of reviewing progress, testing and providing feedback.
4. It provides some level of availability guarantee, so that the business people are able to make their contribution without the frustration of working with a “flaky” environment.
5. It introduces a process for change approval. Typically, this means that the project manager (or equivalent) needs to approve the current state of the software on the Central Development Environment, before it can be deployed to the Test Environment. This isolates business users from unpredictable, untested, or inappropriate changes which cause frustration and lack of confidence when they are asked to participate in the testing process.

Note that the approval process introduced above does not need to be a heavyweight change request process! A simple “looks OK to deploy to the Test Environment” instruction via email, or even better, a facility for the project manager to trigger deployments to the Test Environment themselves via a single click within Continuous Integration/Continuous Delivery infrastructure, is all that is required.  The key point is that the project/product manager should act as a gatekeeper for the environment and there should be no surprises or frustration for business users caused by over-enthusiastic developers making unplanned or unexpected deployments to this environment.

“Test” vs “UAT”

The term UAT or “User Acceptance Testing” Environment is commonly used in the software industry as a label for the main Test Environment. We do not use the term UAT at Priocept for a few reasons:

1. Acronyms are best avoided.
2. It’s old school.
3. “User Acceptance Testing” specifically implies that the testing is to be performed by an end-user, and is to be performed with the specific objective of them formally “accepting” the software. Formal software acceptance is a different concept to general software testing and the two should be kept separate. We want to encourage business users or customers to participate in software testing and feedback, without making them feel like they have to formally accept or “sign-off” something in the process.
4. Many systems need testing by someone other than the end user. The system may be an “end-user-less” system which exists only to integrate other systems, and needs to be tested by specialist test engineers rather than by end users. Or we may simply want to perform functional testing ourselves internally on the Test Environment, without involving end users until we have a higher level of confidence ourselves in the state of the software.

If necessary, multiple “sub” Test Environments can be introduced into your set of environments. So you may decide that you specifically need a User Acceptance Test Environment for formal acceptance and sign-off of the software. Or you may need a Performance Test Environment which exists only for the purpose of performance testing the system. These become additional environments alongside the main Test Environment. But the generic term should simply be “Test Environment”, not “UAT” or anything else that is unnecessarily narrow.

Pre-Production

Once sufficient testing has been performed to develop confidence in the software that is being developed, we need to progress it towards the eventual goal of running it in production.

The Pre-Production Environment is the last environment on which the software should be deployed, evaluated and tested, prior to being used “for real” in production.

The Pre-Production Environment therefore has the following attributes:

1. It should be identical to the Production Environment at the infrastructure level as well as at the software platform level. This means that it should use the same hardware, operating system, networking and storage infrastructure as used on the Production Environment.
2. It is used by the software delivery team to achieve final confidence in the system being delivered, and the related deployment processes, before it is deployed to the Production Environment. It is generally not used by business users, who should have completed their participation using the Test Environment.
3. The majority of the time it should be running the same software release (version) as the Production Environment. The only time this is not the case is when a deployment to the Production Environment is in progress, when the software version running on Pre-Production will be temporarily ahead of Production. The Pre-Production Environment should therefore never be more than one software version behind the Production Environment.

These attributes mean that Pre-Production can also be used to reproduce and diagnose issues that occur in Production. Except during a short window when a deployment to Production is in progress, Pre-Production should always be “ready to go” as a very close mirror of Production. On the other hand, if your Production Environment is reliable and trouble-free, and if your deployment processes are reliable, repeatable and trouble-free, the Pre-Production Environment may be the least actively used of all of your environments. Either way, do not allow “bit-rot” on your Pre-Production Environment – make sure it is not neglected and is always a very close replica of the current state of production.

Above we stated that Pre-Production should be identical to Production at the infrastructure level. If you are running on modern software-defined cloud infrastructure  such as AWS or Google Cloud Platform, then it is generally acceptable for Pre-Production to use a scaled-down version of Production. For example, you may use smaller compute instance sizes, and a smaller baseline number of instances in your auto-scaling configuration. This is acceptable provided that your infrastructure-as-code solution is parameterised so that you are using a single code base to define both the Pre-Production Environment and Production Environment. But if, for example, you are using different AWS CloudFormation templates for Pre-Production and Production, then the entire purpose of the Pre-Production Environment has been defeated, as you are not using it to test the process and tooling that you will also use on the Production Environment.

In some organisations it is common to use the Pre-Production Environment for performance testing, as well as (or instead of) the other purposes described above.  However, this approach is mostly a hangover from the days when it was expensive to establish additional infrastructure that was a full replica of the production infrastructure.  The Pre-Production Environment was historically a very expensive environment and so it made sense to maximise its utilization by also using it for performance testing duties.  In the modern era of cloud infrastructure, it is very easy and cheap to create an additional temporary environment solely to support performance testing.  This also means that performance testing can be performed earlier in the workflow, without having to wait until you are ready to deploy to the Pre-Production Environment.  Earlier testing is always good. For these reasons there should be no need to use the Pre-Production Environment for performance testing unless you are using on-premise infrastructure and are limited by the number of environments that you can establish.

Staging

So what about staging environments? Are these the same thing as pre-production? If not, then how are they different? Like “Test” and “UAT”, we find that “Pre-Production” and “Staging” are often used interchangeably within an organisation, despite having slightly different meanings for different people.

At Priocept we have standardised on the term “Pre-Production” instead of “Staging”. However, there are some scenarios where an additional environment is needed, which is similar but subtlety different to Pre-Production.

Many software projects have a workstream or phase where it is necessary for business users to perform data entry, configuration, content authoring or other work that contributes to the final end product and may need to be carried out in parallel to the software development effort. In this scenario, the Test Environment is not an appropriate environment for the business users to use, because they are not simply testing the software, they are actually contributing to the end product.

Furthermore, if these business users are spending many hours or days working on this data entry work, then the impact and cost of losing this work would be considerable, and so the environment on which they are working should make appropriate protections against data loss or corruption – in other words it needs to be treated as a production grade system. However, the main Production Environment may also not be a suitable environment on which to perform this work, since it may either not yet have the required software deployed to it, or may be being used in parallel for existing “business as usual” activities.

In these scenarios, an additional Staging Environment is required.

The Staging Environment exists solely for the purpose of allowing business users to perform data entry tasks or make other tangible contributions to the project that are distinct from the core software development and testing processes.

The Staging Environment has the following attributes:

1. It runs the most recent stable version of the software under development – this is required to facilitate business users in their data entry, configuration or content authoring activities.
2. It is secured, protected and backed up in the same way as a full production system, to prevent the possibility of losing work produced by the business users.
3. It provides a mechanism for “downstream replication” of data or content to the Production Environment. Typically, the Production Environment is the definitive source of truth for business data, and this data is then “upstream replicated” back to Pre-Production, Test and Development environments, to keep them up-to-date as required for ongoing development. But in the case of a Staging Environment, work is being produced that will at some point need to be sent in the other direction, to the Production Environment.

This definition of the Staging Environment means that once you implement such an environment, you know that you are creating a replication task (and possibly a merge task) to perform in the future to move data from the Staging Environment to the Production Environment. If you do not have a Staging Environment, then you do not have this problem, and can assume that all data replication will be performed from the Production Environment, as the definitive source, back to the other upstream environments.

Environment Sequencing

The following diagram illustrates the flow of software development and associated artifacts, through each of the environments defined above:

Summary of Environment Characteristics

The following table lists all the environments in order of software progression from initial development through to production use, and compares the characteristics of each environment:

Other Considerations

Configuration Management

It obviously makes sense for all environments to be configured to be as similar as possible. Automated configuration management and infrastructure-as-code tooling should be in use to enable and enforce this, although these techniques are beyond the scope of this article.

However, some level of “drift” in configuration between environments is acceptable, provided that all environments are sufficiently similar to ensure that:

1. Testing of the software on a given environment provides confidence that it will then also work on each subsequent downstream environment.
2. An issue that occurs on the Production Environment can be easily reproduced, and can therefore be efficiently debugged and fixed, using whichever upstream environment is most appropriate for a given type of issue.

In practice, this usually means that certain aspects of the architecture should be identical between all environments, including the operating system (distribution, version and patch level), the software stack (products and versions), and file system configuration (path names and permissions).

Other aspects of the environments, such as the physical infrastructure, virtualization/containerization technology, and levels of redundancy, can usually vary between environments while still meeting the requirements above (except in the case of Pre-Production and Production as explained above).

Note that if individual developers are using Windows or Mac based machines to develop software that will then be deployed to Linux infrastructure downstream (or vice-versa), and if they only run and test their code on their Windows or Mac environment before deploying it to the Central Development Environment, then the requirements above have not been met! Each individual developer should be using an appropriate virtual machine (or Docker container or similar) on their local machine, to verify that the code work will work when deployed to the target Central Development Environment, and they should have this facility available at all times ready for when they need to quickly reproduce and fix an issue that occurs on the Production Environment.

Client-Side Software Applications

Where the software under development is a client-side application, such as a native mobile application, the concept of various server-side environments to mirror an eventual production environment may not be relevant.  In this scenario the sequence of environments described above can still be used, with appropriate tooling, to allow project team-members to test various versions of the application at various stages of progression.  Instead of deploying the software directly to a server-side environment, you can use remotely-accessible “test farm” infrastructure such as AWS Device Farm to allow your testers to interact with a known version of the application (or Firebase Test Lab for automated tests), running on a known platform specification.  For development of client-side Windows applications, the same result can be achieved with a sequence of remote desktop servers that mirror the environment sequence described above and have non-persistent disk configurations so that they revert to a known state once a tester completes their testing.

Continuous Delivery

While a carefully designed pipeline of environments from the Local Development Environment through to the Production Environment can dramatically improve the software delivery process, a high level of automation is also required to ensure that software can be efficiently progressed through all the environments and to provide the capability for swift roll backs. Automation creates a high degree of confidence in the process and minimises risk.

In a future article we will discuss how Priocept uses its preferred Continuous Integration and Continuous Delivery tools to achieve effective software delivery across the whole range of environments described above.