Decluttering your Terraform code using for_each

Introduction

Terraform has become the go-to tool for infrastructure management, offering impressive features like for_each. This powerful meta-argument allows you to dynamically create multiple instances of resources based on maps or sets. Whether you’re an experienced Terraform user or just starting, this blog post will equip you to master the art of dynamic resource creation using for_each. Using for_each provides several benefits:

Dynamic creation of resources

With for_each, you can dynamically create multiple instances of a resource based on the elements of a map or set. This allows you to manage a variable number of resources easily, which is particularly useful when working with dynamic environments or configurations. Imagine a scenario where you need to spin up multiple instances of a resource - perhaps security groups, subnets, or virtual machines - each with its unique configurations. With Terraform’s for_each, this task becomes a breeze and DRY-er. With a simple map or set, you can dynamically create as many instances as you need, making it ideal for managing dynamic environments or configurations.

Predictable & stable resource identifiers

With infrastructure management, stability and predictability is key. Unlike its counterpart count, which relies on numerical values and can cause headaches when resources are reordered, for_each ensures stable and predictable resource identifiers. Each resource instance gets a unique key based on elements in the map or set, ensuring that their identities remain consistent, regardless of changes in the order. This leads to predictable and hassle-free resource management.

Eaaaasy removal without surprises

Change is inevitable in any infrastructure. When you remove an element from the for_each map or set, Terraform handles the corresponding resource removal without disrupting other existing resources. Say goodbye to the anxiety of unintentional resource destruction, as for_each ensures a smooth and non-destructive removal process.

Configuration is more data-driven

for_each works well with maps, which can be defined as variables or generated from data sources. This enables you to define resources based on data-driven configurations, making your Terraform code more flexible and maintainable, this opens up a whole new world of possibilities.

Configuration is more concise and explicit

Using for_each often leads to more concise and explicit configurations compared to count. It is especially useful when dealing with a collection of resources that have unique configurations but follow the same resource type and definition.

Granular control over resource lifecycle

Managing resource lifecycles with precision is a core requirement for Us infrastructure engineers, with for_each, you can have granular control over the lifecycle of individual resources. By managing the elements of the map or set, you can selectively create, update, or delete resources based on specific requirements. Say goodbye to resource declaration sprawl within your codebase.

Show and tell

Now let’s deep-dive and have a look at Terraform’s for_each in action! To better showcase for_each’s benefits, I will show you a static Terraform code and will walk you through the painpoints and how we can address those painpoints by adding for_each. Below is an example main.tf file that statically creates a vpc and a subnet.

variable "vpc_name" {
    type    = string
    default = null
}

variable "vpc_cidr_block" {
    type    = string
    default = null
}

variable "snet_name" {
    type    = string
    default = null
}

variable "snet_cidr_block" {
    type    = string
    default = null
}

resource "aws_vpc" "this" {
    cidr_block = var.vpc_cidr_block

    tags = {
        Name = var.vpc_name
    }
}

resource "aws_subnet" "this" {
  vpc_id     = aws_vpc.this.id
  cidr_block = var.snet_cidr_block

  tags = {
    Name = var.snet_name
  }
}

And here is an example terraform.tfvars to configure the resources declared in main.tf.

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"
snet_name       = "my-demo-snet"
snet_cidr_block = "10.0.1.0/24"

This is how you would normally declare resources in Terraform, by adding resource blocks for the resource you need. This is well and good but what happens when we suddenly add more resources? With that in mind Let’s try adding x2 additional subnets to the main.tf file.

variable "vpc_name" {
    type    = string
    default = null
}

variable "vpc_cidr_block" {
    type    = string
    default = null
}

variable "snet1_name" {
    type    = string
    default = null
}

variable "snet1_cidr_block" {
    type    = string
    default = null
}

variable "snet2_name" {
    type    = string
    default = null
}

variable "snet2_cidr_block" {
    type    = string
    default = null
}

variable "snet3_name" {
    type    = string
    default = null
}

variable "snet3_cidr_block" {
    type    = string
    default = null
}

resource "aws_vpc" "this" {
    cidr_block = var.vpc_cidr_block

    tags = {
        Name = var.vpc_name
    }
}

resource "aws_subnet" "this_1" {
  vpc_id     = aws_vpc.this.id
  cidr_block = var.snet1_cidr_block

  tags = {
    Name = var.snet1_name
  }
}

resource "aws_subnet" "this_2" {
  vpc_id     = aws_vpc.this.id
  cidr_block = var.snet2_cidr_block

  tags = {
    Name = var.snet2_name
  }
}

resource "aws_subnet" "this_3" {
  vpc_id     = aws_vpc.this.id
  cidr_block = var.snet3_cidr_block

  tags = {
    Name = var.snet3_name
  }
}

As you can see adding additional subnets means that the instances of the variable and resource block declaration scales up as the number of resources increases, now, imagine this happening in a large project, you will have to declare resources repetitively and managing the resources becomes more challenging thus making even the smallest code change carries elevated risks. Its the same case with modules, the more you reuse a module the more you repeat yourself and your variable decalration scales with it - and because your variable declaration scales your inputs scale as well, let’s go into our terraform.tfvars and see how that would look like.

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"
snet1_name       = "my-demo1-snet"
snet1_cidr_block = "10.0.1.0/24"
snet2_name       = "my-demo2-snet"
snet2_cidr_block = "10.0.2.0/24"
snet3_name       = "my-demo3-snet"
snet3_cidr_block = "10.0.3.0/24"

As expected, the amount of inputs also increased as we increased the amount of subnets in our terraform code. A few things stand out here for me. Naming repeating resources, variables and inputs as this_1, this_2, this_3, snet1_, snet2_, and snet3 doesn’t feel very DRY, unecessarily verbose, and complex. The same goes for modules, next thing you know, you are declaring s3_bucket_1, s3_bucket_2, s3_bucket_3 modules and declaring variables specific to those modules just to configure 3 individual s3 buckets.

Wouldn’t it be nice if the only thing that scales is your inputs/configurations but not your actual terraform code? That’s where for_each comes into play. While doing it this way is functional, it can become challenging to manage and maintain as the number of resources increases. Let’s refactor the main.tf file to use for_each.

variable "vpc_name" {
    type    = string
    default = null
}

variable "vpc_cidr_block" {
    type    = string
    default = null
}

variable "snets_configuration" {
    type    = map(object({
        cidr_block = string
    }))
    default = {}
}

resource "aws_vpc" "this" {
    cidr_block = var.vpc_cidr_block

    tags = {
        Name = var.vpc_name
    }
}

resource "aws_subnet" "this" {
    for_each = var.snets_configuration
    vpc_id     = aws_vpc.this.id
    cidr_block = each.value.cidr_block

    tags = {
        Name = each.key
    }
}

Phew! that wasn’t too hard! Notice how there’s significantly less code? Let’s have a look at the changes. We’ve just declared var.snets_configuration, allowing us to supply a map of objects to configure the underlying resource instead of repeatedly declaring variables. Notice that we’ve gotten rid of the multiple static subnet declaration and replaced it with just aws_subnet.this, this resource, using the for_each meta-argument, will then iterate through the map of objects declared inside the var.snets_configuration to create multiple instances of this resource. Here’s a breakdown of what each line inside the aws_subnet.this resource does:

• for_each = var.snets_configuration - This tells Terraform to create one aws_subnet resource for each element in the var.snets_configuration map. The var.snets_configuration map is expected to contain key-value pairs where the key is the name of the subnet and the value is an object containing configuration details for the subnet, such as its CIDR block.

• vpc_id = aws_vpc.this.id - This sets the VPC ID for each subnet to the ID of the VPC resource named this. This means all the subnets will be created within this VPC.

• cidr_block = each.value.cidr_block - This sets the CIDR block for each subnet. The CIDR block is retrieved from the value object of each element in the var.snets_configuration map.

• tags = { Name = each.key } - This assigns a tag to each subnet. The tag’s key is "Name" and its value is the key of the current element in the var.snets_configuration map. This effectively names each subnet after its corresponding key in the map.

In summary, this refactored code block creates multiple AWS subnets within a specified VPC, with each subnet’s CIDR block and name determined by the var.snets_configuration map. The for_each meta-argument allows for efficient creation of multiple similar resources in Terraform. Now, let’s have a look at our refactored terraform.tfvars and see how it has changed.

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"

snets_configuration = {
  my-demo1-snet = {
    cidr_block = "10.0.1.0/24"
  }
}

The newly refactored terraform.tfvars as we can see above is defining a map of objects as input to a variable named snets_configuration which we declared earlier in our main.tf file. Notice how its less verbose, less repetive and more structured compared to the previous one? In this example we have a single object with the key my-demo1-snet declared inside the snets_configuration map.

Because the code is now dynamic thanks to for_each,we can just add more subnets by modifying the terraform.tfvars.

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"

snets_configuration = {
  my-demo1-snet = {
    cidr_block = "10.0.1.0/24"
  }
  my-demo2-snet = {
    cidr_block = "10.0.2.0/24"
  }
  my-demo3-snet = {
    cidr_block = "10.0.3.0/24"
  }
}

Notice how much less code was written, and yet it’s highly configurable and simpler to manage and maintain for day-to-day use? Essentially, we don’t have to write another resource to create a subnet; you just go to your tfvars and add a new subnet section under the configuration block.

We can also dynamically remove all subnets just by leaving the snets_configuration block empty.

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"

snets_configuration = {}

Deleting or undeclaring the snets_configuration from terraform.tfvars also has the same effect. Let’s test this by removing the snet_configuration variable completely!

vpc_name        = "my-demo-vpc"
vpc_cidr_block  = "10.0.0.0/16"

Conclusion

Terraform’s for_each offers unparalleled flexibility and efficiency in resource management. Whether you’re provisioning resources for a small, or large project, embracing the simplicity and power of for_each to streamline your Terraform deployments will certainly improve your and your team’s quality of life.

Congratulations on venturing beyond Infrastructure as Code and stepping into the realm of Infrastructure as Data. In the next segment, we will delve into the implementation of the optional() object type attribute, which empowers you to define optional arguments in your Terraform code. Additionally, we’ll discover how optional() can supercharge your for_each code, enabling you to create multiple resource instances with diverse configurations based on maps or sets of strings. The synergy of these features bestows upon you unparalleled flexibility in effectively managing your infrastructure configurations! Happy provisioning!