Introduction

If you are not using either CloudFormation or a third party tool like Terraform to create your AWS infrastructure, and instead are using manual point-and-click in the AWS web console, you should move to an infrastructure-as-code approach as soon as possible. This will ensure that all of your infrastructure creation is repeatable, automated, versioned and audited.

Most introductions to cloud infrastructure platforms like AWS or Google Cloud Platform start by talking about all the exciting stuff like unlimited scalability and big data capabilities. Infrastructure-as-code is less sexy, and is rarely mentioned in introductions to cloud infrastructure technologies, but is absolutely critical when building modern infrastructure and when developing DevOps practices.

But AWS CloudFormation can be a tricky beast to work with.

Given that infrastructure-as-code is such a critical part of cloud infrastructure, and given that CloudFormation is the native infrastructure-as-code offering from AWS, it’s surprising that it hasn’t been given more love and attention by Amazon. Out-of-the-box, it’s really not that much fun to work with. In this blog article we explore some techniques for taming CloudFormation and making it a less unwieldy tool to use.

CloudFormation Overview

CloudFormation provides a declarative method of defining a set of AWS infrastructure, in source code format. This means that AWS infrastructure can be defined, versioned, tested, automatically deployed, and delivered to third parties, in exactly the same way as any other type of software deliverable.

When you write and deploy CloudFormation code, it generates a stack in AWS.  A stack is a collection of interrelated resources which can be created, updated or deleted as a single logical entity using either the AWS command line (CLI) tools or the AWS web console.

CloudFormation is generally known as being a very powerful, but not very “developer friendly” tool.

CloudFormation Alternatives

Before we go deep into CloudFormation, a quick note on some alternative technologies. Terraform from HashiCorp, and the open source Troposphere tool are the two best known alternatives to CloudFormation. Terraform replaces CloudFormation entirely, while Troposhere provides an intermediary Python library that can be used to generate CloudFormation code, without writing it directly.

A comparison of the merits of CloudFormation versus Terraform or Troposphere is a subject for its own dedicated blog article, another time.

For now, let’s just assume that you have decided to use CloudFormation. Perhaps you need to deliver some software defined infrastructure for a customer that already uses CloudFormation. Or perhaps you are building some AWS infrastructure for someone in your DevOps team that has specifically requested that you use native AWS tooling. Or perhaps you want to use CloudFormation to take advantage of some features that Terraform does not offer.

Not-Cool CloudFormation Features

CloudFormation comes in for regular criticism in the follow areas and, generally, this criticism is entirely valid.

JSON

JSON was designed as machine readable format, not a human-readable format. It works perfectly for passing data across web services, and between different bits of code that can effortlessly parse a mess of curly and square brackets. But JSON is not a programming language, and it was never designed to be used in this way, making it decidedly developer-unfriendly.

Worst of all, JSON does not support commenting! Any language that does not support commenting in any form is a really, really bad choice of language for anything that needs to be a) humanly readable, and/or b) self-documenting.

Unfortunately, the AWS platform seems to have a bit of a JSON fetish in general, with IAM policies, ElasticBeanstalk, ECS and various other AWS products all enjoying a liberal sprinkling of JSON configuration files.

Even XML might have been a better choice than JSON. Perhaps. At least it supports commenting.

Lack of Modularity

CloudFormation has poor support for modular code. It has limited support for including existing libraries of reusable code from secondary files.  The AWS::Include transform does allow inclusion of secondary CloudFormation files inside a primary file, but only if the secondary file lives in an S3 bucket, making it very cumbersome to build large sets of infrastructure defined across dozens or separate files. Why should we have to upload a changed file to S3 every time we want to retest it?

Limited Conditional Logic

CloudFormation has fairly basic support for conditional code.  We can apply conditionals to individual resources or resource properties, but not to larger sections of CloudFormation code.

Verbose Code

A quick search online for example CloudFormation code to build some AWS infrastructure will typically return an intimidating CloudFormation file that is several hundred lines long. This is not so much because the CloudFormation syntax itself is verbose, but because CloudFormation has poor support for code reuse or Don’t Repeat Yourself (DRY) principles. As a result, CloudFormation files are often subject to a copy-and-paste form of “reuse” and quickly become bloated and impossible to follow. Any reasonably complex AWS infrastructure that is defined using CloudFormation is likely to run to 1000+ lines of code and having this code in a one large file is unmanageable.

S3 Binding

Finally, CloudFormation likes to work with the S3 storage service. If you have a tidy piece of CloudFormation code that is ready to go on your local machine, shouldn’t you be able to run a single command line to push it up to AWS and instantiate your infrastructure without having to go via S3 first? You can in fact do this (using the –template-body option in place of –template-url when running aws cloud formation create-stack) but with a limit of 51,200 characters for your code.

We like to talk about how cloud infrastructure provides “virtually unlimited scalability and data storage”, but then for some unknown reason, we end up with miserly limitations like this to contend with. For any reasonably complex AWS infrastructure, you will come up against this limit. You will then have to either split your infrastructure-as-code into “nested stacks”, or first upload your code to S3, and then tell CloudFormation to go read it from there. A one step process needlessly becomes two steps.

Streamlining CloudFormation with YAML

So let’s say that despite its faults, we want to go ahead and use CloudFormation. It is after all the “official” and idiomatic way of doing things on AWS. What can we do to address the not-cool issues above?

YAML to the Rescue

Since September 2016, CloudFormation supports YAML as well as JSON. Until this announcement, we tried to limit the use of CloudFormation at Priocept. Writing code in a language like JSON, with no ability to document the code with comments, was an issue for us. But YAML support is a game changer. If you are not familiar with YAML, it takes some getting used to. A YAML-aware editor (IDE) is a big help. But once you are comfortable with YAML there is no question that it leads to far more readable code than JSON.

Here is a quick example of the horror of JSON, compared to the relative beauty of the equivalent YAML code.

First the JSON version:

"EC2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {

"InstanceType" : "t2.micro",

"SecurityGroups" : [ {
"Ref" : "SecurityGroupSSH"
} ],

"KeyName" : { "Ref" : "KeyPairName" },

"ImageId" : { "Fn::FindInMap" :
[ "AmazonLinuxAMIMap", { "Ref" : "AWS::Region" }, "AMI"] }

}
}

Curlies all over the place, tokens such as “Properties” being defined as strings with the need for double quotes. And no comments anywhere, which means the code is not self-documenting, and before you know it, you will need a text file or wiki page somewhere to help other team members make sense of it.

Now the YAML version:

# create AWS resource
Resources:

# example EC2 instance
EC2Instance:
Type: "AWS::EC2::Instance"
Properties:

# t2.micro is fine as this is just a demo
InstanceType: "t2.micro"

# assign security group to allow SSH access
SecurityGroups:
- Ref: "SecurityGroupSSH"

# SSH key pair
KeyName:
Ref: "KeyPairName"

# image ID, defined based on region
ImageId:
!FindInMap
- "AmazonLinuxAMIMap"
- !Ref: "AWS::Region"
- "AMI"

Nice and clean, well formatted, and clearly commented/documented. Most editors can colour code YAML nicely.

A Helping Hand from Jinja2

The equivalent to CloudFormation on Google Cloud Platform (GCP) is Deployment Manager. Unlike CloudFormation, Deployment Manager includes a templating framework that provides for pre-processing of infrastructure code, before it is executed. The templating framework in GCP Deployment Manager is based on Python’s Jinja2 templating engine, and provides some key functionality not present in CloudFormation, including:

• Include statements to insert code from separate files, inline in the main code file.
• Looping statements, to allow creation of multiple repeating resources, without writing out similar code multiple times.
• Conditionals, to allow logic that includes, excludes, or alters resources based on previous configuration parameters.

Ansible also uses the Jinja2 templating framework for file creation, so for consistency with both Ansible and GCP, Jinja2 is the obvious choice for templating of CloudFormation files.

It is actually quite straightforward to implement your own Jinja2 templating feature for CloudFormation. The J2CLI tool can be used on the command line, to run CloudFormation files through the Jinja2 templating engine before executing them. Once we have installed JC2CLI, we can simply run a command such as this:

j2 stack.yml > stack.templated.yml

This will take the stack.yml file, which is a CloudFormation stack definition with additional Jinja2 templating included as necessary, and generate stack.templated.yml, which is a “pure” CloudFormation stack definition that can be deployed to AWS using the standard aws cloudformation create-stack CLI command. At Priocept we have a standardized deploy shell script that standardises and automates the process of installing J2CLI if required, running the templating process, and then creating the stack using the AWS CLI . The abbreviated version looks something like this:

#!/bin/bash

CF_FILENAME="stack.yml"
CF_TEMPLATED_SUFFIX=".templated.yml"
CF_TEMPLATED_FILENAME="$CF_FILENAME$CF_TEMPLATED_SUFFIX"

# install and verify j2cli Jinja2 templating
command -v pip >/dev/null 2>&1 || { echo >&2 'pip command not installed.'; exit 1; }
pip install j2cli >/dev/null
command -v j2 >/dev/null 2>&1 || { echo >&2 'j2cli not installed.'; exit 2; }

# run template file through Jinja2 tempting
j2 "$CF_FILENAME" > "$CF_TEMPLATED_FILENAME"
if [ $? -ne 0 ]; then
echo "Error processing template."
exit 3
fi

# other code omitted...

# create stack
echo "Creating CloudFormation stack..."
aws --region "$region" \
cloudformation create-stack \
--stack-name "$stackName" \
--template-body "file://$CF_TEMPLATED_FILENAME"

Note that we use the –template-body option to run create-stack against the local CloudFormation YAML file. This avoids the hassle of having to upload the file to an S3 bucket first. We will cover this in more detail in Part 2.

Where your Jinja2 templating logic requires access to variable values, these can either be defined as environment variables before running j2, or in a separate variable file that you pass in to J2CLI via a command line parameter.

YAML Indenting Challenges

The YAML syntax is whitespace sensitive and requires correct indentation structure at all times. This creates a problem with Jinja2 include statements, because included files are not indented to match the indentation of the include statement. They need to be explicitly indented. This can be achieved with the following Jinja2 templating trick:

# define include macro
{% macro include(file) %}{% include(file) %}{% endmacro %}

# include sub YAML file, with required indentation:
Resources:
{{ include('resources.yml')|indent(2) }}

The include statement above will insert the contents of resource.yml with an indentation of two spaces, thereby maintaining the correct YAML indentation. The macro line defines a macro called include which is just a wrapper around the include function. This appears redundant but is required because the indent Jinja2 filter is used to apply the indent and Jinja2 filters can be applied only to the output of macros, not functions. The macro only needs to be defined once at the stop of your primary CloudFormation file.

As an aside, if you use Ansible to orchestrate your CloudFormation deployments, instead of the AWS CLI (a subject for a different blog post), you might be wondering if you can use the Ansible template module to implement your Jinja2 templating, instead of using J2CLI. Unfortunately, this will not work as the implementation of macro in Ansible will not import variable definitions to the included files. So it works best to just call j2 using the Ansible shell module.

Some examples follow where Jinja2 include functionality makes a big difference to the readability and maintainability of a CloudFormation stack.

Example 1 – UserData Includes

First let’s look at an example where we define a “UserData” shell script for an EC2 instance. This is a very common approach to any kind of AWS/EC2 automation:

# create AWS resources
Resources:

# create example EC2 instance
EC2Instance:
Type: "AWS::EC2::Instance"
Properties:

# define user data (startup) script
UserData: !Base64 |
#/bin/bash
# EC2 user data script to auto configure instance
echo 'Bootstrapping EC2 instance...'
# more shell script stuff here
echo 'Startup complete.'> /tmp/startup.txt

Why is this a bad idea?

Firstly, while we have a trivial bash script example above, this script is likely to be much more complex for any kind of real-world application. Suddenly you might find yourself with dozens or even hundreds of lines of shell script embedded inside your CloudFormation template.

Secondly, we have two different languages, YAML and Bash, in one file. We would like to have separate stack.yml and userdata.sh files, if only so that we can use the correct file extensions and allow our IDE to syntax highlight them correctly.

Third, what happens if we have two or more EC2 instances with the same (or similar) UserData script? We want to apply the DRY principal, and write the shell script once, then use in many places.

And finally, do we really want to have to spin up an EC2 instance just to test a simple shell script? The UserData script above is a fully portable Bash script. Why not test this initially on a local development VM or perhaps a Docker container? We shouldn’t need to go through the whole process of launching (and paying for) an EC2 instance, just to find out that we have a shell script error. To avoid this, we need to separate the shell script source code out into a separate .sh file, that we can edit, run and test, completely independently of AWS.

Using Jinja2 templating, we can change the above code to:

# create AWS resources
Resources:

# create example EC2 instance
EC2Instance:
Type: "AWS::EC2::Instance"
Properties:

# define user data (startup) configuration
UserData: !Base64 |
{{ include('user-data.sh')|indent(6) }}

Now we have made the CloudFormation template more compact and more readable, and we have established a separation of concerns between the CloudFormation infrastructure-as-code and the EC2 instance level bootstrapping. The Bash scripting expert in your team can potentially go ahead and develop and test the user-data.sh script without knowing anything about AWS or CloudFormation.

Example 2 – AMI ID Region Mappings

In AWS, an AMI ID uniquely references a particular base EC2 image, such as Amazon Linux, Ubuntu, or Microsoft Windows. An AMI image ID is needed in CloudFormation to define what image you would like a given EC2 instance to be built from.

Unfortunately AMI image IDs differ between AWS regions. This means that even if you have standardized on, say, Amazon Linux for all your AWS infrastructure, you will need to use different AMI image IDs for different parts of your infrastructure that are running on different regions.

The standard solution for this is to use the CloudFormation mappings feature, to build a lookup of all the different AMI image IDs, for a given image type, for all the different regions that you are using. For example:

Mappings:
# AMIs for various regions, for Amazon Linux AMI 2017.03
# released 2017-04-04, HVM (SSD) EBS-Backed 64-bit
AmazonLinuxAMIMap:
# US East N. Virginia
us-east-1:
AMI: "ami-c58c1dd3"
# US East Ohio
us-east-2:
AMI: "ami-4191b524"
# US West N. California
us-west-1:
AMI: "ami-7a85a01a"
# ten or more further mappings potentially required here...

Resources:
EC2Instance1:
Type: "AWS::EC2::Instance"
Properties:
ImageId: !FindInMap [AmazonLinuxAMIMap, !Ref "AWS::Region", 32]
InstanceType: t2.micro

Here we build a mapping from each AWS region, to the corresponding AMI image ID. Aside from being verbose and causing bloat in your CloudFormation file, the mappings are likely to be exactly the same across all your infrastructure, and will be duplicated across all your CloudFormation files.

Applying Jinja2 templating to this problem, the above code now becomes:

Mappings:
# include mappings from AWS regions to AMIs
{{ include('ec2-amazon-linux-amis.yml')|indent(2) }}

Resources:
EC2Instance1:
Type: "AWS::EC2::Instance"
Properties:
ImageId: !FindInMap [AmazonLinuxAMIMap, !Ref "AWS::Region", 32]
InstanceType: t2.micro

Here we replace 20-30 lines of mapping entries (not all shown above), with a single include line. We can include the AMI mappings from a central library file, which we can probably standardize in a single file across all our projects.

Example 3 – AWS Lambda

Here is one more example. We have a Lambda function, written in JavaScript, and we can move it to a separate lambda.js file, to provide separation of concerns, and Javascript syntax highlighting in our IDE, similar to the UserData example above:

Before:

# example Lambda function with embedded Node.js code
LambdaExampleFunction:
Type: "AWS::Lambda::Function"
Properties:
Code:
ZipFile: !Sub |
"use strict";

exports.lambdaFunction = function(event, context, callback) {

console.info(“Doing some fancy Lambda stuff here. “);

// potentially dozens of lines of Javascript follow...
}
FunctionName: "example-lambda-function"
}

After:

# example Lambda function with Node.js code in separate .js file
LambdaExampleFunction:
Type: "AWS::Lambda::Function"
Properties:
Code:
ZipFile: !Sub |
{{ include('lambda-function.js')|indent(8) }}
FunctionName: "example-lambda-function"

Summary

By writing CloudFormation in YAML format, we can avoid the monstrosity of JSON files and enjoy the “luxury” of being able to comment our infrastructure-as-code, making it self-documenting.

Furthermore, by adding Jinja2 templating into our CloudFormation toolbox, we can make our AWS infrastructure-as-code more efficient to develop and easier to maintain, and we can also separate concerns so that code such as UserData scripts and Lambda functions are testable outside of AWS.

In Part 2 of this blog post, we will look at some more advanced Jinja2 templating features, and other tricks to make CloudFormation development a more pleasant experience.