New Terragrunt features: graph, structured logging, telemetry

Just a few days ago, we announced two new Terragrunt features: terragrunt scaffold and terragrunt catalog. Today, you get three more features! In this blog post, you’ll learn about:

1. Terragrunt graph: Run a command against the graph of dependencies.
2. Structured logging: Output all logs in JSON format.
3. Telemetry: Output traces and metrics in OpenTelemetry format.

Let’s go through these one at a time.

Terragrunt graph

For a number of years, Terragrunt has supported terragrunt run-all <COMMAND> as a way to run the Terraform/OpenTofu command COMMAND against all modules in the current folder and its subfolders. Today, we’re announcing terragrunt graph <COMMAND>, which will run the Terraform/OpenTofu command COMMAND against the graph of dependencies for the module in the current working directory, where this graph consists of all modules that depend on the module in the current working directory via a depends_on or dependencies block, plus all the modules that depend on those modules, and all the modules that depend on those modules, and so on, recursively up the tree.

This gives you a way to automatically run a command against all dependent modules, no matter where they are in your folder structure. It also allows you to deploy modules separately (which helps limit blast radius, speed up performance, etc), while still keeping all the modules up to date as if they were all defined in a single giant module.

For example, consider the following dependency graph:

Each time you run apply on the eks module, its behavior and output variables may change, so you may want to run apply on all the modules that depend on it as well. To do that, you can run terragrunt graph apply in the eks folder, and Terragrunt will execute the following steps:

1. Run apply on the eks module.
2. Run apply on eks-service-1 and eks-service-2, concurrently.
3. Run apply on eks-service-2-v2, eks-service-3, and eks-service-5, concurrently.
4. Run apply on eks-service-3-v2 and eks-service-4, concurrently.
5. Run apply on eks-service-3-v3.

(Note: since none of the lambda modules depend in any way on eks, they are not affected.)

Structured logging

You can now enable structured logging with Terragrunt using the --terragrunt-json-log and --terragrunt-tf-logs-to-json flags. This will result in every log line being written in JSON format, which has several major benefits:

1. It makes it easier to parse logs programmatically. You can use this for scripting, or to gather metrics, or to set up alerts on your logs.
2. It makes it easier to read the logs manually, too, as each log line tells you which module is logging, and includes a timestamp. This is especially useful when using terragrunt run-all or terragrunt graph, where many modules are logging concurrently, as the JSON fields let you clearly see which module is doing what.

For example, if you run:

terragrunt run-all apply \
  --terragrunt-json-log \
  --terragrunt-tf-logs-to-json \
  --terragrunt-non-interactive

You’ll see log output that looks like this:

{"level":"info","msg":"The stack at /projects/gruntwork/terragrunt-tests/run-all will be processed in the following order for command apply:\nGroup 1\n- Module /projects/gruntwork/terragrunt-tests/run-all/app1\n- Module /projects/gruntwork/terragrunt-tests/run-all/app2\n- Module /projects/gruntwork/terragrunt-tests/run-all/app3\n\n","time":"2024-01-11T21:31:46Z"}
{"level":"info","msg":"OpenTof v1.6.1","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"on linux_amd64","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"OpenTofu v1.6.1","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"on linux_amd64","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"OpenTofu v1.6.1","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"on linux_amd64","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"Initializing the backend...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"Initializing provider plugins...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"- Finding latest version of hashicorp/local...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"Initializing the backend...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"Initializing provider plugins...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"- Finding latest version of hashicorp/local...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"Initializing the backend...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"Initializing provider plugins...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"- Finding latest version of hashicorp/local...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"- Installing hashicorp/local v2.4.1...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"- Installing hashicorp/local v2.4.1...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app1"}
{"level":"info","msg":"- Installing hashicorp/local v2.4.1...","time":"2024-01-11T21:31:46Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app3"}
{"level":"info","msg":"- Installed hashicorp/local v2.4.1 (signed by HashiCorp)","time":"2024-01-11T21:31:47Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}
{"level":"info","msg":"","time":"2024-01-11T21:31:47Z","workingDir":"/projects/gruntwork/terragrunt-tests/run-all/app2"}

Notice how each line of log output includes a workingDir, which tells you which module is doing the logging.

Telemetry

You can now enable telemetry in Terragrunt so that it emits metrics and traces in OpenTelemetry format. This allows you to send data about what’s happening in Terragrunt to tools such as Prometheus, Jaeger, and Grafana Tempo. You can use these metrics and traces to track performance, debug issues, etc.

For example, you could fire up Jaeger on localhost as follows:

docker run \
  --rm \
  --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:1.54.0

And then configure Terragrunt to report traces to Jaeger on localhost as follows:

export TERRAGRUNT_TELEMETRY_TRACE_EXPORTER=http
export TERRAGRUNT_TELEMERTY_TRACE_EXPORTER_HTTP_ENDPOINT=localhost:4318
export TERRAGRUNT_TELEMERTY_TRACE_EXPORTER_INSECURE_ENDPOINT=true

Now, when you run any Terragrunt commands, you’ll see traces in the Jaeger UI:

Check out the Terragrunt Telemetry docs for details on how to integrate Terragrunt with other systems.

Conclusion

As we’ve announced in this blog post and in Monday’s blog post, there are a lot of powerful new features in Terragrunt:

1. terragrunt catalog: browse your module catalog.
2. terragrunt scaffold: scaffold out files for configuring a module.
3. terragrunt graph: Run a command against the graph of dependencies.
4. Structured logging: Output all logs in JSON format.
5. Telemetry: Output traces and metrics in OpenTelemetry format.

Install the latest version of Terragrunt, take these new features for a spin, and let us know how they work for you!