A guide to automating HashiCorp Vault #1: Auto-unsealing

Vault is an open source tool created by HashiCorp for securely storing secrets, such as database passwords, API keys, and TLS certs.

Authenticating to Vault as a normal user is easy, you just need to remember a secret such as a username/password or token. But there are two problems that used to be fairly hard:

1. How to unseal Vault. In the past, every time a Vault node went down, or you added or updated a node, human operators would have to manually unseal that node, which was cumbersome.
2. How to have servers authenticate to Vault. Authenticating manually is easy: you just need a memorized secret such as a username/password or token. But how do you authenticate automatically? That is, how do you get your app servers to authenticate to your Vault servers without storing any secrets in plaintext on the app servers (which would defeat the purpose of Vault)?

We will walk through how to solve both these problems with Vault in this three-part blog post series:

1. Auto-unsealing (this blog post)
2. Authenticating with instance metadata
3. Authenticating with an IAM user or role

We have also implemented all these patterns in the open source repos for AWS and GCP, so the you can check them out and try the working examples.

Vault Auto-unseal

When Vault first boots, it does not yet have the master key in memory, and therefore it cannot decrypt its own data. While Vault is at this state, we say it is “sealed.” Vault uses Shamir’s Secret Sharing, which splits the master key into multiple pieces, each of which must be provided manually by a human operator and recombined in order to regenerate the master key and unseal Vault. This unseal process is manual and tedious. Luckily, Vault offers a feature that allows automatic unsealing.

The auto-unseal feature delegates the unsealing process to a Key Management Service such as AWS KMS or GCP KMS. Vault uses the KMS key as a seal-wrap mechanism: it encrypts and decrypts Vault’s master key, and it does so with the whole key, replacing the Shamir’s Secret Sharing method.

Gruntwork worked closely with HashiCorp to build production-grade modules for getting Vault up and running with multiple examples on how to best leverage many of its features. In the following examples I’ll go into details on AWS, but the principle is the same for GCP. You can see the GCP-specific example here.

To fire up a Vault cluster quickly in AWS, you can use the following Terraform code:

module "vault_cluster" {
  source = "github.com/hashicorp/terraform-aws-vault.git/modules/vault-cluster?ref=v0.11.3"

  cluster_name  = "my-vault-cluster"
  cluster_size  = 3
  instance_type = "t2.micro"

  ami_id    = "${var.ami_id}"
  user_data = "${data.template_file.user_data_vault_cluster.rendered}"

  vpc_id     = "${data.aws_vpc.default.id}"
  subnet_ids = "${data.aws_subnet_ids.default.ids}"
}

data "template_file" "user_data_vault_cluster" {
  template = "${file("${path.module}/user-data-vault.sh")}"

  vars {
    aws_region = "${data.aws_region.current.name}"
  }
}

# Using default VPC just to keep this example short
data "aws_vpc" "default" {
  default = "true"
}

data "aws_subnet_ids" "default" {
  vpc_id = "${data.aws_vpc.default.id}"
}

data "aws_region" "current" {}

This code executes a User Data script during boot to start up Vault:

#!/bin/bash

readonly VAULT_TLS_CERT_FILE="/opt/vault/tls/vault.crt.pem"                       readonly VAULT_TLS_KEY_FILE="/opt/vault/tls/vault.key.pem"      
                                      
/opt/vault/bin/run-vault \
  --tls-cert-file "$VAULT_TLS_CERT_FILE" \
  --tls-key-file "$VAULT_TLS_KEY_FILE"

Make sure the filepaths for Vault and the TLS certs are correct or, preferably, use Packer to build an image using our template available here, which already does lots of configuration for you.

Now let’s move on to how we can modify the previous code to support auto-unseal! First, create an AWS KMS key* in your desired region and take note of the key alias. Vault will need an awskms stanza in Vault’s configuration file (usually default.hcl) with the key information.

*Please note that AWS KMS keys have a cost per month per key, as well as an API usage cost.

seal "awskms" {
  kms_key_id = "alias/KEY_ALIAS"
  region     = "KEY_REGION"
}

Gruntwork’s run-vault script creates best-practices configuration for you and can add this stanza if you specify the following arguments:

run-vault \
  --enable-auto-unseal \
  --auto-unseal-kms-key-id "$kms_key_id" \
  --auto-unseal-kms-key-region "$aws_region"

Finally, set the enable_auto_unsealflag and pass the KMS key ARN:

data "aws_kms_alias" "vault-example" {
  name = "alias/${var.auto_unseal_kms_key_alias}"
}
module "vault_cluster" {
  source = "github.com/hashicorp/terraform-aws-vault.git/modules/vault-cluster?ref=v0.11.3"
  enable_auto_unseal      = true
  auto_unseal_kms_key_arn = "${data.aws_kms_alias.vault-example.target_key_arn}"
  # ... other vars
}

Now, when you deploy your Vault cluster-and every time a node is added or replaced-it will unseal automatically, without any human intervention!

Next steps

The Vault Auto-unseal feature was originally only available in Vault Enterprise but, recently, while we were working to add an example to our modules, it was added to the open source package from version 1.0 onwards. You might also want to use HashiCorp Consul as a storage backend and service discovery mechanism for Vault. Check our complete examples of spinning up Consul and Vault clusters with auto-unsealing on AWS and GCP.

In the next post, we’ll talk about how servers can automatically authenticate to Vault using instance metadata.

Your entire infrastructure. Defined as code. In about a day. Gruntwork.io.