Patterns for extensible Terraform modules - AMI IDs

Terraform is my preferred tool for automating cloud infrastructure deployments. It’s not without it’s problems but in balance I still prefer it over any other tool I’ve tried. When re-useable modules were added fairly early on in the development of Terraform (sometime before 0.5.0 in 2015) it became possible to package up Terraform resources and to build upon the work of others.

The Terraform Community Modules GitHub organization is a good example of building quality modules — the modules all have documentation, a sensible versioning strategy, and make things configurable via input variables. These modules have been a huge help to me in getting a new project up and running quickly at my latest job. I’d especially like to thank them for writing the tf_aws_vpc module as it let me create a VPC with all the right subnets super quickly.

However the problem when using other people’s modules is that it often won’t do quite what you need it to do. (This isn’t a dig at the tf_aws_vpc or any other specific module). Perhaps it has a provisioner that SSHs directly to an EC2 instance but in your deployment everything has to go via an SSH bastion/jump-host; or maybe it uses a stock Ubuntu AMI (Amazon Machine Image) with a single shared “ubuntu” user and a shared SSH keypair and you can’t have any shared credentials in your environment so need to configure users somehow. The “best” case is that the change to the module is small and the maintainer is still active so you can “easily” fork the module, add the feature you want and have it accepted upstream. Life rarely works out this nicely though.

In this series I’m going to cover some of the tricks that you, as a module author, can do to keep the scope of your module small, but provide suitable extension points so that users can tweak it to their hearts’ content without having to open pull requests for every site-specific feature.

(These tips will be written up using AWS resources as this is what I use most of the time. Some of them are AWS specific such as the aws_ami data source but the general principles should apply to other cloud providers.)

Lets start with a trivial Terraform module - one that just spins up an EC2 instance using an Ubuntu 16.04 LTS AMI:

resource "aws_instance" "mod" {
  ami           = "ami-d15a75c7"
  instance_type = "t2.micro"
}

(You can view all of the code samples from this blog post on my code-samples Github repo.)

Don’t hard-code AMI IDs¶

When terraforming instances for your own project hard-coding an AMI is fine - you know what image you want, and what region you run it in. This isn’t a great way to build reusable modules though. The main reason is that AMIs are specific to each region so hard-coding will make the module only work in the original region, and perhaps most importantly if a new version of the AMI is built to include security updates then the module source will need updating. If anyone has pinned the module to a specific version (which is probably sensible to do so that things don’t break/change underneath you) they will need to also update the module source stanza at the call-site.

(You do apply security updates to your AMIs and rebuild them. Don’t you?)

There are a few ways around this hard-coding:

1. Use a variable for ami¶

This lets the module caller easily change it without having to open a pull request. I would always recommend adding a default value to input variables where at all possible. In the module code it looks like:

variable "ami_id" {
  default     = "ami-d15a75c7"
  description = "AMI ID to use when launching our EC2 instance. Default is Ubuntu 16.04 LTS in us-east-1"
}

resource "aws_instance" "mod" {
  ami           = "${var.ami_id}"
  instance_type = "t2.micro"
}

And then called like this:

provider "aws" {
  region = "us-east-2"
}

module "mymod" {
  source = "..."
  # We're in a different region, so we need to specify a different AMI.
  ami_id = "ami-5e94b23b"
}

(In the specific case of different regions it is possible to use “map” variables to provide a lookup of region to AMI id.)

Having the ami_id input variable allows for customization by the end user but puts some onus on them to find the right AMI and to manually track updates. There’s a way to avoid this though.

2. Use the aws_ami data source¶

Instead of having to specify an AMI manually Terraform can use the AWS APIs to find an image matching certain criteria automatically. This example is shown in the terraform docs for the aws_instance resource.

In the module:

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "mod" {
  ami           = "${data.aws_ami.ubuntu.id}"
  instance_type = "t2.micro"
}

When you terraform this The aws_ami data source will use the filters to search for available images in the current region (available means the image is public, one in your AWS account, or one to which you’ve been granted permissions on).

Note: You can add tags to any AMIs you build, and also filter on them, but searching by tag only works within the same AWS account. This is why the name in the above example is so long.

When using most_recent = true it will find the most recently published AMI every time Terraform is run which can be a good thing or a bad thing depending on your point of view. Having a terraform destroy and re-create an EC2 instance to update to the latest AMI might not be what every one wants, or they might want to use a different AMI “release” all together which they can’t do with this mechanism alone.

It is possibly to combine both of these first two tips to give us a bonus tip:

3. Automatically find AMI IDs but allow override.¶

The advantage of this mechanism is that it works with zero extra input in the typical case but gives users control if they need it. It looks like this in the module:

variable "ami_id" {
  default     = "" # Note this is empty.
  description = "Use this specific AMI ID for our EC2 instance. Default is Ubuntu 16.04 LTS in the current region"
}

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "mod" {
  ami           = "${var.ami_id != "" ? var.ami_id : data.aws_ami.ubuntu.id}"
  instance_type = "t2.micro"
}

This works by using the conditionals feature of the interpolation language introduced in Terraform version 0.8.0 (first released in December 2016): "${var.ami_id != "" ? var.ami_id : data.aws_ami.ubuntu.id}" — this will use the value from the ami_id variable if it is not empty and fall-back to the AMI the data source discovered.

The other advantage of this pattern is that it lets someone start with the module in “auto-discovery” mode, take the AMI that it found and feed it back in under the ami_id input to pin to a specific AMI – putting the user in control of when an upgrade happens.

Or they could use their own aws_ami data source to find and track an AMI of another distribution without us having to provide input variables for every constant used in our AMI filter.

(You can view all of the code samples for this post on my code-samples Github repo.)

If you found this first part of my series on Terraform useful check back (or subscribe to the feed) for more. In future posts I’ll cover:

• how you can make user data/cloud-init extendable
• applying user-supplied tags in combination with computed ones (i.e. a compute Name tag in addition to what ever else is given)
• conditionally disabling data sources (which is especially useful with private AMIs in a big organization)