Gruntwork Newsletter, March 2018

Published in

Gruntwork

11 min readMar 5, 2018

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

We had a busy month! We developed a new module for building serverless apps with API Gateway and Lambda that works in dev, prod, and integrates with Swagger and Terraform; we’ve added support for CircleCI 2.0 and a reusable module for running Jenkins; we’ve made an important fix to the OpenVPN module; we’ve updated our Redis module with support for encryption and sharding; we updated many of our modules to work with CentOS; we’ve added support to our ECS modules for a health check grace period; lots of bug fixes; and a whole lot more.

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates
DevOps News

Gruntwork Updates

API Gateway + Lambda Module

Motivation: One of our customers asked us to develop modules so they could use API Gateway with Lambda to build serverless web apps. We wanted a solution that met the following requirements:

Use API Gateway
Use Lambda Functions
Provide a way to easily test API Gateway and Lambda Functions locally
Provide a method deploy to production, while still managing everything as code
Integrate with existing Terraform-managed infrastructure (for example reading database URLs or VPC ids from Terraform).

After evaluating AWS SAM Local, the Serverless Framework, and Terraform’s API Gateway support, we realized that there wasn’t a solution that met the above requirements without requiring a massive amount of verbosity, copy/paste, and duplication.

Solution: We created package-sam, including the CLI tool gruntsam. The basic idea is that you to define your APIs using Swagger, which allows you to take advantage of all the tools in the Swagger ecosystem, such as automatic generation of documentation and client and server bindings and documentation:

swagger: '2.0'
paths:
  /example:
    get:
      summary: example endpoint
      produces:
      - application/json
      responses:
        200:
          description: ok
        500:
          description: internal server error
      x-gruntwork-apigateway-integration:
        integrationType: aws_proxy
        integrationHttpMethod: post
        localSourcePath: ./js
        handler: test1.get
        runtime: nodejs6.10
        authorization: none
        apiKeyRequired: false

Note that the Swagger code includes some simple Gruntwork extensions for specifying what Lambda function to run. This allows you to use the gruntsam CLI tool to test your code completely locally (under the hood, we generate template.yml for SAM):

$ gruntsam local start-apiLoading Swagger document "swagger.yml"
Auto-generating SAM "./template.yml" from "swagger.yml"
Launching sam local with [local start-api -t ./template.yml]
Connected to Docker 1.32Mounting example.get at http://127.0.0.1:3000/example [GET]You can now browse to the above endpoints to invoke your functions.

gruntsam also allows you to generate the code for a Terraform module from your Swagger definition to create an identical API Gateway configuration for production deployment:

$ gruntsam deploy -output-path=./api-gateway-moduleLoading Swagger document "swagger.yml"
Auto-generating "gen/main.tf" from "swagger.yml"
Auto-generating "gen/vars.tf" from "swagger.yml"
Auto-generating "gen/outputs.tf" from "swagger.yml"
Running "terraform fmt" on generated terraform code

Now you can use the generated module with the rest of your Terraform code, including Lambda functions defined using package-lambda!

module "example_lambda" {
 source = 
  "git::git@github.com:gruntwork-io/package-lambda.git//modules/lambda?ref=v0.2.0"  name        = "example"  source_path = "${path.module}/js"
  runtime     = "nodejs6.10"
  handler     = "example.get"  timeout     = 30
  memory_size = 128
}module "api_gateway" {
  source = "./api-gateway-module"  name            = "api-gateway-example"
  example_get_uri = "${module.example_lambda.invoke_arn}"
}

With package-sam, you can run your code in dev and prod, keep it DRY, and integrate it with the rest of your Terraform infrastructure.

What to do about it: Check out the new package-sam.

CircleCI 2.0 support and Circle CI 1.0 sunsetting

Motivation: CircleCI released version 2.0 at the end of last year. This is a major new release that adds:

Support for the latest version of Docker.
Support for Docker layer caching (faster builds).
The ability to run the build in a custom Docker image that has all your dependencies already installed (even faster builds).
More powerful caching options (even more faster builds).
Workflows, such as triggering builds on a schedule (e.g., nightly) or with manual approvals.

This month, they also announced that they will be sunsetting 1.0 in August, so everyone needs to migrate. The sample apps in the Gruntwork Reference Architecture were still using 1.0, and a number of customers asked us for help on upgrading.

Solution: We’ve updated the sample apps in the Reference Architecture to CircleCI 2.0. As promised, builds are now faster, and more powerful, but a fair warning: CircleCI 2.0’s YAML syntax is clunky, verbose, and unintuitive. It took a while for us to figure it out, and while you get much more power, you should expect to spend a fair amount of time figuring out how to use your newfound powers.

What to do about it: Check out our instructions for Upgrading to CircleCI 2.0, which include example code you can copy.

Jenkins Module

Motivation: We’ve had Jenkins as a “usage pattern” in our Reference Architecture for a long time, but never as a reusable, tested module. We needed to add CentOS support for Jenkins, so it was time to extract that code.

Solution: We’ve added two new modules to module-ci:

install-jenkins: A bash script to install and configure Jenkins on Ubuntu or CentOS.
jenkins-server: A Terraform module to run Jenkins on top of an ASG, with an EBS Volume and ALB attached.

What to do about it: Check out this example code to see how to put these new modules together to run Jenkins.

OpenVPN Fix

Motivation: By default OpenSSL, which package-openvpn relies on, sets its Certificate Revocation List (CRL) to expire every thirty days. This meant if a you hadn’t revoked a certificate at least every 30 days, then your CRL would expire, and your OpenVPN installation would stop working.

Solution: We updated the init-openvpn module of package-openvpn to set the expiration of the CRL to 10 years, rather than 30 days.

What to do about it: Upgrade to the latest version (v0.5.3) of package-openvpn.

Automatic, scheduled EC2 and EBS Backup

Motivation: We needed a way to make scheduled backups of stateful services such as Jenkins and WordPress that run on EC2 Instances and use EBS Volumes.

Solution: We created a new module called ec2-backup, which is a Lambda function that runs on a scheduled basis to take snapshots of your EC2 Instances and EBS Volumes (under the hood, it uses ec2-snapper). It can also delete old snapshots while preserving certain constraints (e.g., delete snapshots older than 7 days, but ensure there are always at least 5 snapshots available) so AWS doesn’t charge you too much. Finally, the module updates a metric in CloudWatch every time it runs, making it easy to use the scheduled-job-alarm module to configure a CloudWatch alarm that goes off if the backup job ever fails.

What to do about it: Check out the ec2-backup module for documentation and the Jenkins example to see how we use ec2-backup to make scheduled backups of Jenkins.

CentOS support for Gruntwork modules

Motivation: One of our customers wanted to use a number of our modules on CentOS, but we only supported Ubuntu and Amazon Linux.

Solution: We’ve updated a number of our modules with CentOS support:

What to do about it: Check out the release notes of each module for details.

Encryption for Redis

Motivation: AWS recently added support for enabling in-transit and at-rest encryption for Redis on ElastiCache, but the Gruntwork redis module did not expose that new functionality.

Solution: We’ve updated the redis module so you can enable encryption at rest and in transit using the new parameters enable_at_rest_encryption and enable_transit_encryption.

What to do about it: Update to module-cache, v0.3.3 and enable encryption if you need it. Note that if you already have Redis deployed without encryption, you cannot enable encryption for it; you must create a new Redis cluster that has encryption enabled and migrate to that.

ECS Health Check Grace Period

Motivation: ECS Services can automatically register with a load balancer and use it for health checks, but for a long time, there was no support for a “grace period,” so if your ECS Service took a long time to boot up, the load balancer would mark it as unhealthy, and ECS would replace it before it even finished booting, starting the whole cycle over again.

Solution: ECS has finally added support for a health check grace period and we’ve updated the ecs-service and ecs-service-with-alb modules in module-ecs with support for setting this grace period in your services!

What to do: Update to module-ecs v0.6.2 and use the health_check_grace_period_seconds parameter to set a grace period for your services.

New Terraform Utility Modules

Motivation: Occasionally, when Terraform is missing some feature we need, we call out to a shell script or python script. Unfortunately, the way we build the paths to those scripts only worked on *nix systems and did not work on Windows.

Solution: We added two new modules to package-terraform-utilities:

operating-system: This can be used to detect the operating system on which Terraform is currently running.
join-path: This can be used to join multiple path parts (folders, files) into a single path, using the proper separator for the current OS.

What to do about it: Check out the new modules in package-terraform-utilities.

Windows fixes: package-kafka, package-zookeeper, server-group

Motivation: The way we called the rolling deployment script used in the server-group module, which we use under the hood for our ZooKeeper and Kafka modules, did not work correctly on Windows.

Solution: We fixed the way we call the rolling deployment script in server-group and updated the Kafka and ZooKeeper to the latest version.

What to do about it: Update to package-zookeeper, v0.3.1 and package-kafka, v0.2.1 (NOTE: the latter is a pre-release; see package-kafka, v0.2.0 for details). If you’re using server-group directly, update to module-asg, v0.6.9.

Lambda@Edge

Motivation: One of our customers wanted to use Lambda@Edge, but due to some limitations on Lambda@Edge, our existing lambda module did not work.

Solution: We’ve added a new lambda-edge module that should work with Lambda@Edge!

What to do about it: Give the new lambda-edge module a shot. Note that we have not yet updated the CloudFront code to add triggers automatically, so you’ll have to do that manually for now.

Checksums in fetch and gruntwork-install

Motivation: We wanted a more secure way to verify that the modules we were installing using fetch and the Gruntwork Installer couldn’t be hijacked by an attacker and replaced with malicious code.

Solution: fetch, v0.2.0 and gruntwork-install, v0.0.21 now support the ability to validate a Release Asset (binary) downloaded from GitHub with either a SHA256 or SHA512 checksum.

What to do about it: Update to the latest versions of fetch and gruntwork-install and you can start using the --binary-sha256-checksum and --binary-sha512-checksum params to validate checksums of release assets (binaries). For example:

gruntwork-install \
  --repo https://github.com/gruntwork-io/gruntkms \
  --binary-name gruntkms \
  --tag <VERSION> \
  --binary-sha256-checksum <CHECKSUM>

For now, you’ll have to calculate the checksums manually, but we’ll update our builds in the future to publish checksums. We’ll also add support for verifying modules (i.e., those installed via --module-name param) using sha1 commit IDs.

Open source modules

We’ve had a number of fixes contributed by the community to our open source projects Consul, Nomad, Vault, and Terragrunt:

terraform-aws-vault, v0.2.0: You can now enable S3 as a storage backend again (while still using Consul as the HA backend)!
terraform-aws-vault, v0.2.1: You can now install Vault from a custom URL. This is especially useful for Vault enterprise.
terraform-aws-vault, v0.3.0: The vault-elb module no longer exposes an availability_zones parameter. Please use subnet_ids instead.
terraform-aws-nomad, v0.1.1: You can now specify a custom name for the Nomad ASG using the asg_name parameter.
terraform-aws-nomad, v0.2.0: To be more consistent with the Consul module, the nomad-cluster module now takes as input a cluster_tag_value parameter that will be used to tag the instances for auto-discovery.
terraform-aws-consul, v0.1.2: You can now specify custom tags for the Consul cluster using the tags parameter.
terraform-aws-consul, v0.2.0: The Consul ASG is now namespaced using the cluster_name variable, just like all the other resources. Terraform treats renaming an ASG as deleting the old one and creating a new one, so be careful when updating!
terraform-aws-consul, v0.2.1: You can now use Spot Instances in your Consul cluster by setting the spot_price parameter. Also, updated the run-consul script to retry its search for EC2 Tags if none are found, as Tags can sometimes take a while to propagate.
terraform-aws-consul, v0.2.2: You can now specify a custom data center configuration for Consul by setting the --datacenter option when calling run-consul.
Terragrunt, v0.14.1: Fix out of bounds slice issue when Terragrunt is called with a command that requires multiple arguments, but some of those arguments are missing.
Terragrunt, v0.14.2: If you specify a custom endpoint property in your S3 backend configuration, Terragrunt will now use that endpoint too.

Other fixes

module-security, v0.7.1: Use a more reliable way to look up the path to the ssh-iam binary during the install process.
module-server, v0.4.0: Fixed a number of minor bugs in the persistent-ebs-volume module. The aws_security_group in now uses name_prefix instead of name and sets create_before_destroy to true. This should fix the DependencyViolation: resource sg-XXX has a dependent object error described in terraform-providers/terraform-provider-aws#1671.
module-server, v0.4.1: Update the attach-eni script to retry if 0 ENIs are found. Previously, the script would simply exit.
aws-auth: We now have instructions for how to use aws-auth with LastPass: LastPass with AWS. These make it easier to authenticate to your AWS account on the CLI.
module-asg, v0.6.7, v0.6.8, v0.6.9: Fix rolling deployment script path on Windows for the server-group module.

DevOps News

Inter-region VPC peering now available in more regions

What happened: A couple months ago, AWS finally launched the ability to peer VPCs that are in different regions. Now this functionality is available in 9 additional regions: AWS EU (London), EU (Ireland), EU (Paris), Asia Pacific (Mumbai), Asia Pacific (Sydney), Asia Pacific (Singapore), Asia Pacific (Tokyo), Canada (Central) and South America (São Paulo) Regions in addition to AWS US East (Northern Virgina), US East (Ohio), US West (Northern California), US West (Oregon) Regions.

Why it matters: If you are using multiple AWS regions, this makes it much easier to connect them all together so resources in one region can access resources in another.

What to do about it: Check out the VPC peering documentation.

PostgreSQL 10 now available in RDS

What happened: PostgreSQL version 10 is now available in RDS.

Why it matters: Version 10 adds a number of new features, including native table partitioning, support for improved parallelism in query execution, ICU collation support, column group statistics, and enhanced postgres_fdw extension.

What to do about it: Simply specify version 10 next time you deploy RDS (see the rds module).

AWS Instance Scheduler

What happened: AWS has released a set of patterns and CloudFormation Stacks for shutting down and restarting EC2 Instances and RDS databases on a scheduled basis.

Why it matters: You can use this approach to shut down non-production instances when you’re not using them (e.g., at night and on weekends) to save a lot of money.

What to do about it: Check out the AWS Instance Scheduler docs. If you’re interested in having us build a Terraform version of this, email us at support@gruntwork.io.