Gruntwork Newsletter, November 2021

Published in

Gruntwork

16 min readNov 9, 2021

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we have updated Terragrunt to support multiple include blocks (which allows you to make your code more DRY), updated Terratest with native support for continuously enforcing policies on your terraform code using Open Policy Agent (OPA), updated our infrastructure code to be compliant out-of-the-box with the new recently released version 1.4.0 of the CIS AWS Foundations Benchmark, started updating all our modules with support for using IMDSv2, and made many other bug fixes and improvements. Finally, a reminder that, as usual, we’ll be taking a holiday break for two weeks in December, so please plan accordingly!

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates
DevOps News

Gruntwork Updates

DRY Terragrunt — multiple include blocks

Motivation: For a long time, Terragrunt only supported one level of include blocks. include is a feature of Terragrunt that allows you to import common configurations so that you can share them across your project. With only one level of include, you could only define one set of configurations to share. This was commonly used to share the definitions for project level configurations, like remote state (terraform.backend) and provider blocks. However, oftentimes you had multiple levels of shared configurations. For example, the configurations to use for a single component like vpc is oftentimes very similar across all your environments (qa, stage, prod), with only a few things differing like the CIDR blocks to use. The single level include was very limiting in this regard, as you could not promote those common settings to a single configuration that gets imported.

Solution: We introduced the ability to include multiple files in a single configuration starting with v0.32.0 of Terragrunt! This addresses the pain points of single level include while working around the technical limitations of multiple levels. Now, you can define your component level common configurations in a separate file that gets imported and merged with the project level common configurations.

For example, consider the following folder structure:

└── live
    ├── terragrunt.hcl
    ├── _env
    │   └── vpc.hcl
    ├── prod
    │   └── vpc
    │       └── terragrunt.hcl
    ├── qa
    │   └── vpc
    │       └── terragrunt.hcl
    └── stage
        └── vpc
            └── terragrunt.hcl

In this structure, the root live/terragrunt.hcl configuration contains the project level configurations of remote state and provider blocks, while the _env/vpc.hcl configuration contains the common inputs for setting up a VPC. This allows the child configurations in each env (qa, stage, prod) to be simplified to:

include "root" {
  path = find_in_parent_folders()
}

include "env" {
  path = "${get_terragrunt_dir()}/../../_env/vpc.hcl"
}

inputs = {
  cidr_block = "10.0.0.0/16"
}

You can learn more about this feature in the following links:

Terragrunt documentation describing multiple include blocks, exposed includes, and deep merge and how they can be used to DRY your project.
Example project that leverages multiple include blocks.

What to do about it: Download the latest terragrunt version and give multiple include blocks a try!

OPA with Terratest

Motivation: Many organizations have business and legal requirements that must be enforced on their infrastructure. These requirements most often stem from various compliance frameworks like HIPAA and PCI. Traditionally, many of these requirements were enforced through manual review, but this tends not to be a repeatable, consistent process. How many requirements does the organization have? How many people in your team know all of them by heart? How many are trained to spot non-compliance in code snippets?

Solution: We added support in Terratest v0.38.1 for enforcing policies on your Terraform code using Open Policy Agent (OPA, pronounced “oh-pa”)! OPA is an open source, general-purpose policy engine that unifies policy enforcement across the stack. Terratest adds an interface that seamlessly integrates OPA policies with Terraform (OPA does not have native support for HCL and Terraform). You can call a single function (test_structure.OPAEvalAllTerraformModules) in Terratest to apply an OPA policy against all your Terraform modules!

You can read more about OPA, how it works, and how Terratest helps automate OPA checks for Terraform in our blog post Automatically Enforce Policies on Your Terraform Modules using OPA and Terratest. You can also take a look at the OPA example in the Terratest website.

What to do about it: Checkout the blog post, write some OPA policies for your Terraform code, and add automated OPA checks to your modules using the latest version of Terratest!

CIS v1.4.0

Motivation: In May, 2021, version 1.4.0 of the CIS AWS Foundations Benchmark was released, which introduced changes and new requirements that our customers wanted reflected in their infrastructure code.

Solution: We have updated our Gruntwork Compliance modules to be compliant with version 1.4.0 of the CIS Benchmark out of the box!

What to do about it: For more information, see the dedicated blog post, which explains what changed and how to upgrade to the new version.

Instance Metadata Service Version 2 Updates

Motivation: A while back, AWS released Instance Metadata Service version 2 (IMDSv2), which helps protect against several attack vectors, including misconfigured website application firewalls, misconfigured reverse proxies, unpatched SSRF vulnerabilities, and misconfigured layer-3 firewalls and NATs. This is better from a defense in depth perspective, but using IMDSv2 is a bit trickier than IMDSv1, so customers were asking for help.

Solution: We have updated a number of our repos with support for using IMDSv2:

First, we updated bash-commons with support for using either IMDSv1 or IMDSv2. For now, until we’ve updated all our modules to work with v2, we still default to v1, but you can set the environment variable GRUNTWORK_BASH_COMMONS_IMDS_VERSION to 2 to use IMDSv2 instead.
Next, we updated terraform-aws-server, terratest, terraform-aws-openvpn, and terraform-aws-eks with support for using IMDSv2. We are actively working on updating the rest of our repos too.
Finally, we added two new modules for working with IMDSv2. One is require-instance-metadata-service-version, which can be used to disable IMDSv1, and ensure only v2 is used. The other is disable-instance-metadata, which can be used to disable Instance Metadata entirely, which is useful on EC2 instances that don’t need it, or those that only need it during the initial boot, and can shut it down after.

What to do about it: Starting using the new code in the previous section and let us know how it works for you. For an intro to IMDSv2, see this blog post.

Winter break, 2021

Motivation: At Gruntwork, we are a human-friendly company, and we believe employees should be able to take time off to spend time with their friends and families, away from work.

Solution: The entire Gruntwork team will be on vacation December 20th — January 2nd. During this time, there may not be anyone around to respond to support inquiries, so please plan accordingly.

What to do about it: We hope you’re able to relax and enjoy some time off as well. Happy holidays!

Service Catalog Updates

terraform-aws-service-catalog

v0.61.0: Replaced ECS cluster autoscaling machinery from CPU based to Capacity Provider based autoscaling. Capacity provider based autoscaling is a more superior form of ECS cluster autoscaling that bases decisions on ECS task scheduling and cluster availability. Refer to the AWS blog post on ECS cluster autoscaling for more information on how this works. This is a backward incompatible change. Refer to the migration guide for information on how to adapt your ECS cluster to this new form of autoscaling.
v0.61.1: In the ecs-deploy-runner, we now pass through docker image builder hardcoded options and args. See the release notes in terraform-aws-ci
v0.61.2: Update dependency gruntwork-io/terraform-aws-vpc to v0.17.5 Extend Elasticsearch to support Multi AZ & Master Accounts. Expose security_group_tags for App VPCs.
v0.62.0: Update several dependencies. Refer to the release notes for more details.
v0.62.1: Makes load_balancing_algorithm_type configurable in the ecs-service module; Updates for-production examples for architecture catalog v0.0.21; Adds README for the ecs-deploy-runner module; Optional retention period for fluent-bit CloudWatch log group.
v0.62.2: Pass through routing_rules and viewer_protocol_policy through to the underlying modules in the public-static-website service.
v0.62.3: Bump terraform-aws-security to v0.53.7.
v0.62.4: Exposed ability to manually configure the CIDR blocks for the subnets on the VPC.
v0.62.5: Added ability to manage non-alias subdomain records on public zones in the route53 module. Use the newly added subdomains field on the public_zones input variable to configure the records.
v0.62.6: Added the ability to modify the VPC flowlogs CloudWatch IAM role and name of the CloudWatch Log Group.
v0.63.0: Updated various dependencies. Note that services/public-static-website has a backward incompatible change. Refer to the release notes for more information.
v0.63.1: Updated dependencies. Refer to the release notes for more details.
v0.63.2: Exposed the ability to pass arbitrary args to bootstrap.sh script on EKS worker nodes.
v0.63.3: Switches the python-based sleep null_resource to use the native time_sleep resource to wait for account creation.
v0.63.4: Fixed source of perpetual diff in eks-cluster module; Exposed permissions_boundary field on IAM roles for EKS cluster; Exposed tag field on Security Groups for EKS workers.
v0.64.0: Exposed the ability to set custom tags on Jenkins; Updated ecs-cluster to allow disallowing certain Availability Zones for the worker pool; Updated variable description for container definitions in ecs-service to be more accurate with what is expected; Updated dependencies (see the release notes for more details).
v0.64.1: Update dependency gruntwork-io/terraform-aws-static-assets to v0.12.2; Update kubernetes provider version to workaround bug.
v0.65.0: Exposed feature flags to shut off kubergrunt features when deploying an EKS cluster with the eks-cluster module; Exposed ability to set terminationGracePeriodSeconds on pods deployed with the k8s-service module; Updated dependency gruntwork-io/terraform-aws-eks to v0.46.0 — this is a backward incompatible update! A naive update will replace your self managed worker pool. Refer to the migration guide for more information.
v0.65.3: Fix a bug in the route53 module in how it was reading the zone_id parameter. You can now configure the ecs-cluster module to use a public IP using the new cluster_instance_associate_public_ip_address input variable.

terraform-aws-cis-service-catalog

v0.27.5: Adjusts the minimum version of the Terraform AWS provider in the aws-securityhub module. Updates dependency gruntwork-io/terraform-aws-security to v0.55.3.
v0.27.6: vpc module improvements: NACLs creation will no longer be attempted for the subnets that are not created; Subnet CIDR blocks are now configurable. Also, gruntwork-io/terraform-aws-service-catalog is updated to v0.62.4.

Open Source Updates

Terragrunt

v0.31.11: Terragrunt will now honor the log level set using the environment variable TERRAGRUNT_LOG_LEVEL. This log level will also be used on the global fallback logger, which will log out stack traces on error.
v0.32.0: Added support for using multiple include blocks in a single configuration. Note that with this change, usage of a bare include block with no labels (e.g., include {}) is deprecated. It is recommended to update your configuration starting this release to attach a label to all include blocks (e.g., include "root" {}). You can learn more about multiple include blocks in the updated documentation.
v0.32.1: Fixed bug where mock_outputs_merge_with_state ignored mock_outputs_allowed_terraform_commands.
v0.32.2: Fixed bug where Terragrunt would error out if no outputs were defined.
v0.32.3: Fixed bug where Terragrunt would break on Windows paths in dependencies
v0.32.4: Fixed bug where Terragrunt did not honor terraform_binary when fetching outputs for dependency.
v0.32.5: Fixed bug where Terragrunt did not correctly handle generated remote state backend configurations in run-all commands.
v0.32.6: You can now access later stage configuration elements in exposed includes (e.g., inputs) if certain conditions are met. Refer to the updated documentation for more info (specifically, section "Limitations on accessing exposed config").
v0.33.0: Running destroy on a single module will now check for dependency references that point to that module. If there are any modules that reference the module being destroyed, terragrunt will warn you with the list of modules that point to the module being destroyed and prompt for a confirmation that you are ok destroying the module. To avoid the prompt and restore previous behavior, you can pass in --terragrunt-non-interactive.
v0.33.1: run-all will now error with a more sensible error message if you are missing the Terraform command that Terragrunt should run on all modules.
v0.33.2: Terragrunt now has a new command render-json which can be used to render the json representation of the fully interpreted terragrunt.hcl config. This can be used for debugging purposes, as well as for enforcing policies using OPA. Reverted the change from v0.32.2, as it is intentional for terragrunt to error out on dependency blocks that have no outputs. If it is desired to allow empty outputs on dependency blocks, you should configure mock_outputs and mock_outputs_allowed_terraform_commands or skip_outputs.
v0.34.0: Fixed bug where the iam_role config attribute was ignored when creating the S3 bucket during Auto-Init. Terragrunt now has an additional parsing stage to parse iam_role attribute early, so that it can use it to resolve each of the get_aws_* functions. This means that the get_aws_* functions now return answers after the IAM role has been assumed, whereas before it was always based on the . Note that this additional parsing stage means that locals are parsed an additional time, which may cause side effects if you are using run_cmd in locals, and the args are dynamic so as to bust the cache.
v0.34.1: Fixed bug where the include related functions were not being correctly parsed when used in locals blocks.
v0.34.2: Updated error messages related to locals parsing.
v0.34.3: Fixed bug where iam_role configuration functionality broke for dependencies. Updated hclfmt to only log files that were updated by the command by default. You can get the original output if you set --terragrunt-log-level debug. Fixed bug where Terragrunt prompts for run-all plan were not shown to the console. Added new configuration option that can be used to set the assume role session name when iam_role is configured.
v0.35.0: Fixed bug where Terragrunt only took the last include block into consideration for the dependency run graph. Now all dependency blocks defined across all include configurations will be taken into consideration.
v0.35.1: Fixed numerous bugs related to assume IAM role features of Terragrunt. Refer to the Release Notes for more details.
v0.35.2: Fixed a regression bug introduced in v0.35.1 where the iam_role config was ignored when managing the s3 state bucket in auto init.
v0.35.3: Fixed bug where Terragrunt panicked with a nil pointer error for certain configurations.
v0.35.4: Terragrunt will now log the order in which modules are deployed when using run-all, instead of all the modules in the graph including those that are excluded. You can get the old format logs if you use --terragrunt-log-level debug. Refer to the Release Notes for sample output.
v0.35.5: Terragrunt will now log parsing errors during the dependency detection phase for destroy at the debug level.
v0.35.6: Warning logs about partial parsing of included configurations have been converted to debug level.
v0.35.7: Fixed bug where tfr source did not handle registries that returned absolute URLs. Fixed regression bug where auto-init no longer handled updates to the source attribute of the terraform block.

Terratest

v0.37.9: Updated IP sanity check script to use Instance Metadata Service Version 2. Add SSM CommandDoc support.
v0.37.10: Fixed bug where IsJobSucceeded did not check for completions, and thus it could pass if no Pods had started yet.
v0.37.11: Added helper functions for interacting with Azure ServiceBus. See servicebus.go for the list of supported functions.
v0.37.12: Added GetTagsForVpc for looking up tags on a VPC. Also added Tags attribute on VPC struct for easy lookup. Fixed bug in one of the Azure tests. Various updates to documentation.
v0.37.13: Dependencies related to azure module has been updated. Added new helper functions GetS3BucketTags and GetS3BucketTagsE for retrieving tags on an S3 bucket.
v0.38.0: Several core cloud packages (azure-sdk-for-go, aws-sdk-go, go-containerregistry, and related) has been updated. While we don't anticipate downstream code changes being required to accommodate the update, we are marking this as backward incompatible to signal that main dependencies were updated across several backward incompatible versions.
v0.38.1: Added initial support for Open Policy Agent with Terraform. Refer to the new terraform-opa-example and associated test (terraform_opa_example_test.go) for an example of what you can accomplish with the new functions.
v0.38.2: Fixed bug where opa eval output was intermingled when multiple files were being evaled. Added feature to automatically rerun opa eval with data result query on failure so that all evaluated expressions are logged. This is useful for debugging failed checks.
v0.38.3: Added GetTagsForSubnet() (and Tags property to Subnet struct) into aws module.

Cloud-nuke

v0.5.2: Added support for nuking DynamoDB resources. The usage of cloud-nuke is exactly the same — nuke all resources given the resource-type , nuke based on a given region , or a given age filter using the flag older-than. For specific usage examples, please refer to the official cloud-nuke Readme.

bash-commons

v0.1.8: Introduced support for AWS Instance Metadata Service (IMDS) Version 2. These changes are fully backward compatible, and bash-commons continues to default to Version 1 of the Instance Metadata Service.
v0.1.9: Added convenience function for echo-ing to stderr.

helm-kubernetes-services

v0.2.5: Added ability to configure emptyDirs that are backed by the worker node disk, rather than tmpfs.
v0.2.6: Added ability to configure terminationGracePeriodSeconds.

Other updates

terraform-aws-ci

v0.38.11: Updated the jenkins-server module to propagate custom tags to more resources.
v0.38.12: Exposed ability to store stdout and stderr from ECS Deploy Runner runs in S3 to programmatically interact with command outputs. Refer to the updated docs for more info.
v0.38.13: Updated Java version to 11 in the Jenkins installation script.
v0.38.14: Bumped default kaniko version installed to v1.5.2.
v0.39.0: Bumped the default versions of various tools in ECS Deploy Runner. Refer to the Release Notes for more details.
v0.39.1: Exposed ability to set the version of packer that gets installed by build-packer-artifact when it is not available.
v0.39.2: Bumped underlying kaniko image version to 1.7.0.
v0.39.3: Fixed bug where branch refs passed as ref to infrastructure-deploy-script was not being handled correctly; Bumped default module_ci_tag in ECS Deploy Runner docker files to this release tag.

terraform-aws-security

v0.55.2: Adds a depends_on between the bucket object ownership and the policy in private-s3-bucket. This resolves an issue where we expected an implicit dependency between the resources (formed by a resource reference) but there was none, resulting in conflicting conditional operation errors.
v0.55.3: Exposed ability to set role-session-name for the assume role session created by aws-auth .
v0.55.4: The iam-users module can now store the access keys for an IAM user in AWS Secrets Manager (if you set store_access_keys_in_secrets_mgr to true) in addition to encrypting the access keys with PGP. This is primarily useful for machine users, where you want the access keys stored somewhere multiple team members can access them (whereas with PGP, typically only one person has the private key).

terraform-aws-ecs

v0.31.5: Exposed ability to configure tags on ecs-daemon-service; Exposed ability to customize the IAM role name of the ECS cluster; Exposed ability to configure permissions boundary on ecs-cluster instance IAM role.
v0.31.6: Updated roll-out-ecs-cluster-update.py script to increase the max size of the ASG if there is not enough capacity to replace all the nodes.
v0.31.7: Exposed ability to configure deployment_controller on ECS services.

terraform-aws-eks

v0.45.1: Updated the kubergrunt version that gets installed to latest.
v0.45.2: Expose permissions_boundary field for cluster IAM role
v0.46.0: Refactored resource naming to use name_prefix instead of name to support create before destroy lifecycle rules.
v0.46.1: Updated map_ec2_tags_to_node_labels.py to use IMDSv2.
v0.46.2: Updated to the latest chart and app versions of AWS LB Controller.

terraform-aws-server

v0.13.5: Added ability to bring your own IAM role to single-server. There is now a new input variable create_iam_role (default true), and when it is false, the module will lookup the IAM role using the iam_role_name variable instead of creating a new one.
v0.13.6: Updated example persistent-ebs-volume to use Instance Metadata Service Version 2. Added new module require-instance-metadata-version which allows you to easily require a specific version of Instance Metadata Service be used by your EC2 instances.
v0.13.7: Update unmount-ebs-volume script to use Instance Metadata Service Version 2.

repo-copier

v0.0.18: repo-copier can now copy code to GitLab.com!
v0.0.19: repo-copier now supports GitHub.com and GitHub Enterprise as destinations!

terraform-aws-vpc

v0.17.6: Added policy variable for the VPC Gateway Endpoint resources and added SES to the vpc-interface-endpoint module.

DevOps News

HashiCorp S1

What happened: HashiCorp has filed an S1 with the SEC, indicating their plans to go public.

Why it matters: An IPO can bring many benefits to a company: more money, more public awareness and brand recognition, more stability, etc.

What to do about it: We wish congratulate our friends at HashiCorp and wish them success in their future IPO!

Interesting Terraform 1.1.0 feature: moved blocks

What happened: Terraform 1.1.0 is not out yet, but in the CHANGELOG, we can see an interesting upcoming feature: moved blocks for “refactoring within modules: module authors can now record in module source code whenever they’ve changed the address of a resource or resource instance, and then during planning Terraform will automatically migrate existing objects in the state to new addresses.”

Why it matters: Previously, if you refactored your Terraform code: e.g., changed the ID on a resource, say, from:

resource "aws_db_instance" "foo" { ... }

To:

resource "aws_db_instance" "bar" { ... }

Terraform would see this as you asking to delete the old database and create a totally new one, which is probably not what you want! In the past, the only solution was to manually run terraform state mv commands to update Terraform’s state so that it new that aws_db_instance.foo is now known as aws_db_instance.bar. With moved blocks, it sounds like we’ll be able to capture these sorts of moves as code, and Terraform will take care of the state updates automatically, making refactoring much easier and safer.

What to do about it: As soon as Terraform 1.1.0 is out, we’ll start using moved blocks in our modules whenever we do refactoring, making it easier and safer for customers to upgrade to new versions of those modules.