Introducing Terragrunt ‘catalog’ and ‘scaffold’

Standardize and speed up how you browse your module catalog and configure your modules for deployment

In this blog post, I’m going to walk you through two powerful new features we’ve added to Terragrunt:

1. terragrunt catalog: browse your module catalog.
2. terragrunt scaffold: scaffold out files for configuring a module for deployment.

We believe these two features will help you both (a) standardize how you manage and deploy your modules and (b) make it easier and faster for developers to configure modules for deployment.

To see how, let’s see these features in action!

These features are available in Terragrunt v0.55.15 and above.

Terragrunt Catalog

Let’s say you created a bunch of OpenTofu/Terraform modules and you worked hard to ensure these modules meet your company’s needs in terms of security, compliance, scalability, and so on. How can you help all the developers at your company find these modules and make use of them?

The first step is to put these modules into one or more Git repos. Here’s an example of just such a repo: https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example. Note how this repo defines all of the modules in the modules folder:

With a repo like this, developers on your team can use the terragrunt catalog command to browse the modules available to them:

terragrunt catalog \
  https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example

This command will pop up an interactive terminal app that lets you browse your modules:

The catalog command shows all the modules you have in a list, allowing you to use the up and down arrow keys to select a module and hit ENTER to see that module’s documentation. You can also quickly find a module by hitting / to enable typeahead search:

Configuring your entire module catalog

Having to know the URL of a Git repo with modules is a bit annoying—especially if you have modules across multiple Git repos. Therefore, the canonical way to use terragrunt catalog is to create a terragrunt.hcl file with a catalog { } configuration that specifies one or more Git repos to search for modules:

# terragrunt.hcl

# Configure what repos show up when you run 'terragrunt catalog'
catalog {
  urls = [
    "https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example",
    "https://github.com/gruntwork-io/terraform-aws-utilities",
    "https://github.com/gruntwork-io/terraform-kubernetes-namespace"
  ]
}

The code above configures 3 Git repos (these are all open source, so feel free to try it out locally!), each with a set of modules in their modules folders, to use for the catalog command. You would typically put this code into your infrastructure-live repo, which is where you configure your modules for deployment into live environments. So now, whenever you’re in that repo, or any subfolder, you can run the catalog command with no arguments:

cd infrastructure-live/non-prod/us-east-1/stage
terragrunt catalog

Note: the catalog command will git clone the repos locally, so the first run may take a little while if you have lots of repos, but it should be much faster on all re-runs.

The terminal UI will open up and show you the available modules across all these repos:

In short, terragrunt catalog allows you to treat all the modules you use across a variety of Git repos as a single, unified, searchable, module catalog.

Terragrunt Scaffold

Being able to see your entire module catalog in a single place is great, but what’s even better is being able to use that catalog right away. Terragrunt now has scaffolding built in, which allows you to generate the code to use a module automatically.

When browsing your catalog, you can select a module, and hit S to scaffold it out:

In the .gif above, I picked the s3-bucket module, and hit S to scaffold it. This created a terragrunt.hcl file in the current working directory (note: I ran catalog in a sub-folder of the infrastructure-live repo and not the root to ensure the generated terragrunt.hcl didn’t overwrite my root terragrunt.hcl) with the following contents:

# This is a Terragrunt module generated by boilerplate.
terraform {
  source = "git::https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/s3-bucket?ref=v0.8.0"
}

include "root" {
  path = find_in_parent_folders()
}

inputs = {
  # --------------------------------------------------------------------------------------------------------------------
  # Required input variables
  # --------------------------------------------------------------------------------------------------------------------

  # Description: The name of the S3 bucket
  # Type: string
  name = "" # TODO: fill in value


  # --------------------------------------------------------------------------------------------------------------------
  # Optional input variables
  # Uncomment the ones you wish to set
  # --------------------------------------------------------------------------------------------------------------------

  # Description: If set to true, block all public access on this bucket.
  # Type: bool
  # block_public_access = true

  # Description: If set to true, delete all the contents of the bucket when running 'destroy' on this resource. Should typically only be enabled for automated testing.
  # Type: bool
  # force_destroy = false
}

The scaffold command does the following automatically:

• Figure out the module URL and the latest version (tag) available, and fill that into the source URL.
• Parse all of the module’s input variables and generate placeholders in the inputs = { } block to make it easy to fill those variables in.

Developers on your team can now run catalog to see what modules are available, use / to find the module they need, hit S to scaffold it out, and then all they need to do is fill in the well-documented variables to get their infrastructure deployed!

You can also use the scaffolding functionality as a standalone feature by running the scaffold command and passing it the URL (any valid module source URL) of a module to scaffold.

terragrunt scaffold <MODULE_URL>

For example, here is how you can scaffold out the same s3-bucket module with a single command:

terragrunt scaffold \
  github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/s3-bucket

Customizing scaffolding

Terragrunt scaffolding has one more trick up its sleeve: you can fully customize how scaffolding works by using Boilerplate, our open source, cross-platform code generator tool (see Introducing Boilerplate for a quick walkthrough).

The terragrunt.hcl file you’ve seen generated by the scaffold command so far is created from a default Boilerplate template built into Terragrunt. However, whenever you run the scaffold command on a module, Terragrunt looks into the directory where your module is defined, and if it finds a .boilerplate folder in there, it’ll assume that’s a custom Boilerplate template, and use that for scaffolding instead.

For example, the ecs-fargate-service module has a .boilerplate folder which contains an example of a custom Boilerplate template. That folder includes the following boilerplate.yml file:

variables:
  - name: TeamName
    description: The name of the team using this module
    type: string

This file defines a single input variable for this template; as you’ll see shortly, Terragrunt will interactively prompt the user for this input variable during the scaffolding process. This allows you to create custom scaffolding experiences for every one of your modules.

Inside that .boilerplate folder you’ll also find a README.md file that uses Go templating syntax to render that input variable:

(... text omitted ...)

- You can define input variables and other configuration in 
  `boilerplate.yml`. E.g., This example defines a variable
  called `TeamName`, which you've set to: `{{ .TeamName }}`.

(... text omitted ...)

Note the use of {{ .TeamName }} to render the value of the input variable.

Finally, the .boilerplate folder also contains the following terragrunt.hcl:

terraform {
  source = "{{ .sourceUrl }}"
}

include "root" {
  path = find_in_parent_folders()
}

inputs = {
  # --------------------------------------------------------------------------------------------------------------------
  # Required input variables
  # --------------------------------------------------------------------------------------------------------------------
  {{ range .requiredVariables }}
  # Description: {{ .Description }}
  # Type: {{ .Type }}
  {{ .Name }} = {{ .DefaultValuePlaceholder }}  # TODO: fill in value
  {{ end }}

  # --------------------------------------------------------------------------------------------------------------------
  # Optional input variables
  # Uncomment the ones you wish to set
  # --------------------------------------------------------------------------------------------------------------------
  {{ range .optionalVariables }}
  # Description: {{ .Description }}
  # Type: {{ .Type }}
  # {{ .Name }} = {{ .DefaultValue }}
  {{ end }}
}

This is using more Go templating syntax to:

• Render the module source URL (sourceUrl), which is something Terragrunt scaffolding automatically passes into your Boilerplate template.
• Render the required and optional input variables. Terragrunt scaffolding automatically parses whichever module you’re using to discover these input variables, and passes all the information about these variables to your templates, so you can render that information however you want. See the custom templates for scaffolding docs for full details on what data is available.

When you scaffold out this module, Terragrunt will automatically use this custom Boilerplate template, prompt you for input variables, and render the README.md and terragrunt.hcl file (instead of just the default terragrunt.hcl):

It’s also worth mentioning that you can use custom Boilerplate templates with any module by passing the URL of the template as the second argument to the scaffold command:

terragrunt scaffold <MODULE_URL> <BOILERPLATE_TEMPLATE_URL>

For example, here is how you can use the s3-bucket module with the custom Boilerplate template from the ecs-service

terragrunt scaffold \
  github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/s3-bucket \
  github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/ecs-fargate-service/.boilerplate

Conclusion

As you’ve seen, the catalog command makes it easier to browse your catalog, find the modules you need, and hit S to kick off scaffolding. The scaffold command generates code for you, either from Terragrunt’s default template, or a custom Boilerplate template of your own, making it easier to configure modules for deployment, and to standardize your coding patterns.

Put these two together, and you’ll be able to accelerate adoption of Infrastructure as Code (IaC) and developer self-service patterns in your organization. This is one of the pieces of functionality that you now get out-of-the-box as part of Gruntwork’s DevOps Foundations.

Go install the latest version of Terragrunt now, try these new features out, and let us know what you think!