Vault HA cluster with integrated storage on AWS
Challenge
Vault supports many storage providers to persist its encrypted data (e.g. Consul, MySQL, DynamoDB, etc.). These providers require:
- Their own administration; increasing complexity and total administration.
- Provider configuration to allow Vault as a client.
- Vault configuration to connect to the provider as a client.
Solution
Use Vault's Integrated Storage to persist the encrypted data. The integrated storage has the following benefits:
- Integrated into Vault (reducing total administration)
- All configuration within Vault
- Supports failover and multi-cluster replication
- Eliminates additional network requests
- Lowers complexity when diagnosing issues (leading to faster time to recovery)
![Reference Architecture](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault-raft-0.png%26width%3D960%26height%3D305&w=1920&q=75)
HashiCorp Cloud Platform (HCP) Vault clusters use Integrated Storage. To learn more about the managed Vault clusters, refer to the Getting Started with HCP Vault Dedicated tutorials. If you are a Kubernetes user, visit the Vault Installation to Minikube via Helm with Integrated Storage tutorial.
Table of contents
This tutorial walkthrough the following:
- Prerequisites
- Setup
- Create an HA cluster
- Join nodes to the cluster
- Data snapshots for recovery
- Resign from active duty
- Remove a cluster member
- Recovery mode for troubleshooting
- Clean up
- Help and reference
Prerequisites
This tutorial requires an AWS account, Terraform, and additional configuration to create the cluster.
- Create an AWS account with AWS credentials and a EC2 key pair
- Install Terraform
Setup
The Terraform files start four instances each running Vault. Here's a diagram:
![Scenario](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault-raft-6.png%26width%3D989%26height%3D591&w=2048&q=75)
- vault_1 is initialized and unsealed. The root token creates a transit key that enables the other Vaults auto-unseal. This Vault does not join the cluster.
- vault_2 is initialized and unsealed. This Vault starts as the cluster leader. An example K/V-V2 secret is created.
- vault_3 is only started. You will join it to the cluster.
- vault_4 is only started. You will join it to the cluster.
Demonstration only
The cluster created by these terraform files is solely for demonstration and should not be run in production.
Retrieve the configuration by cloning the
hashicorp/learn-vault-raft
repository from GitHub.Change the working directory to
learn-vault-raft/raft-storage/aws
.Set your AWS credentials as environment variables.
Tip
The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.
Copy
terraform.tfvars.example
and rename toterraform.tfvars
Edit
terraform.tfvars
to override the default settings that describe your environment.terraform.tfvars1 2 3 4 5 6 7 8 9 1011121314151617
Initialize Terraform.
Run
terraform apply
and review the planned actions. Your terminal output should indicate the plan is running and what resources will be created.Enter
yes
to confirm and resume.When the
apply
command completes, the Terraform output will display the IP addresses of the provisioned Vault nodes.Example:
Additional setup time
While Terraform's work is done, these
instances need time to complete their own installation and configuration.
Progress is reported within the log file /var/log/tf-user-data.log
and
reports Complete
when the instance is ready.
Create an HA cluster
Currently vault_2 is initialized, unsealed, has HA enabled, and is the only cluster member. Nodes, vault_3 and vault_4, have not yet joined the cluster.
Examine the leader
Let's discover more about the configuration of vault_2, and how it describes the current state of the cluster.
Open a new terminal and SSH into vault_2.
Examine the vault_2 server configuration file (
/etc/vault.d/vault.hcl
).To use the Integrated Storage, the
storage
stanza is set toraft
. Thepath
specifies the path where Vault data will be stored (/vault/vault_2
)./etc/vault.d/vault.hcl1 2 3 4 5 6 7 8 9 10111213141516171819202122232425
Warning
Although the listener stanza disables TLS for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.
Configure the
vault
CLI to use the Vault server.Configure the
vault
CLI to use the root token for requests.View the cluster information.
The cluster reports that vault_2 is the only node and is currently the leader.
Join nodes to the cluster
Add vault_3 to the cluster using the vault operator raft join
command.
Open a new terminal and SSH into vault_3.
Terraform output
You can display the output again by running
terraform output
from within the same directory where you ranterraform apply
.Configure the
vault
CLI to use the Vault server.Join vault_3 to the vault_2 cluster.
When a node joins the cluster it receives a challenge from the leader node and must unseal itself to answer this challenge. This node unseals itself through vault_1, via the transit secrets engine unseal method, and then correctly answers the challenge.
Server aliases
The vault_2 server address,
http://10.0.101.22
, is aliased tovault_2
in the/etc/hosts
file.Configure the
vault
CLI to use vault_2 root token for requests which is stored in the~/root_token
file on the vault_2 host. (Be sure to retrieve the~/root_token
on the vault_2 host.)Examine the current raft peer set.
Now, vault_3 is listed as a follower node.
Verify that you can read the secret at
kv/apikey
.This node has access to the secrets defined within the cluster of which it is a member.
Retry join
Similarly, you can use the vault operator raft join
command to join
vault_4 to the cluster. However, if the connection details of all the nodes
are known beforehand, you can configure the retry_join
stanza in the server
configuration file to automatically join the cluster.
Open a new terminal and SSH into vault_4.
Stop vault_4.
Open the server configuration file,
/etc/vault.d/vault.hcl
in a text editor of your choice.Example:
Add the
retry_join
block inside thestorage
stanza as follows. Be sure to set the correctleader_api_addr
value for vault_2 and vault_3 using their public IP addresses.The resulting
/etc/vault.d/vault.hcl
file should look like:/etc/vault.d/vault.hcl1 2 3 4 5 6 7 8 9 101112
Since the address of vault_2 and vault_3 are known, you can predefine the possible cluster leader addresses in the
retry_join
block.Alternatively, you can configure
auto_join
instead ofleader_api_addr
.The
auto_join
was introduced in Vault 1.6.0. Read the Cloud auto join section to learn more.Start vault_4.
Configure the
vault
CLI to use vault_2 root token for requests which is stored in the~/root_token
file on the vault_2 host. (Be sure to retrieve the~/root_token
on the vault_2 host.)List the peers and notice that vault_4 is listed as a follower node.
Patch the secret at
kv/apikey
.The secret has updated for all nodes.
To verify, return to the terminal connected to vault_3 and get the same secret again.
Cloud auto join
Note
The cloud auto join feature requires Vault 1.6.0 or later.
While theleader_api_addr
requires the knowledge of IP addresses, you can use
auto_join
instead which takes cloud provider specific configurations as input.
When auto_join
is configured, Vault will automatically attempt to discover and
resolve potential leader address.
For Amazon EC2:
provider
(required) - the name of the provider ("aws" in this case).tag_key
(required) - the key of the tag to auto-join on.tag_value
(required) - the value of the tag to auto-join on.region
(optional) - the AWS region to authenticate in.addr_type
(optional) - the type of address to discover: private_v4, public_v4, public_v6.access_key_id
(optional) - the AWS access key for authentication.secret_access_key
(optional) - the AWS secret access key for authentication.
This parameter value is very similar to how Consul's cloud auto-join parameter works.
In the server log, you should find entries similar as below:
Example output:
You can specify retry_join
blocks as follows:
1 2 3 4 5 6 7 8 9 10111213141516171819
Important to note that you can specify either leader_api_addr
or auto_join
within a single retry_join
block and not both.
Refer to the Integrated Storage (Raft) Backend documentation as well as the operator raft command documentation for additional information.
Data snapshots for recovery
Raft provides an interface to take snapshots of its data. These snapshots can be used later to restore data if ever becomes necessary.
Take a snapshot
Execute the following command to take a snapshot of the data.
You can take a snapshot from any of the cluster nodes (vault_2, vault_3, or vault_4); however, since the leader node is the single source of truth, it makes more sense to execute the snapshot command on the leader node (vault_2).
Automated snapshots
Note
Automated snapshot require Vault Enterprise 1.6.0 or later; the scenario environment in this tutorial uses the Vault Community Edition.
Instead of taking a snapshot manually, you can schedule snapshots to be taken automatically at your desired interval. You can create multiple automatic snapshot configurations.
Create an automatic snapshot configuration named, daily
which takes a snapshot
every 24 hours. The snapshots are stored locally in a directory named,
raft-backup
and retain 5 snapshots before one can be deleted to make room
for the next snapshot. The local disk space available to store the snapshot is
1GB. This means that raft-backup
retains up to 5 snapshots or 1GB of data
whichever the condition meets first.
In absence of a specific file_prefix
value, the snapshot files will have a
prefix of vault-snapshot
.
Read and verify the automatic snapshot configuration.
Available snapshot storage types are: local
, aws-s3
, azure-blob
, and
google-gcs
. Depending on the target location, the configuration parameters
differ.
View the path help on the sys/storage/raft/snapshot-auto/config
endpoint.
Refer to the API documentation for more details.
Simulate loss of data
First, verify that a secrets exists at kv/apikey
.
Next, delete the secrets at kv/apikey
.
Finally, verify that the data has been deleted.
Restore data from a snapshot
First, recover the data by restoring the data found in demo.snapshot
.
(Optional) Examine the server log of the active node (vault_2).
Verify that the data has been recovered.
Resign from active duty
Currently, vault_2 is the active node. Experiment to see what happens if vault_2 steps down from its active node duty.
In the vault_2 terminal, execute the step-down
command.
Now, examine the current raft peer set.
Notice that vault_3 is now promoted to be the leader and vault_2 became a follower.
Remove a cluster member
It may become important to remove nodes from the cluster for maintenance, upgrades, or to preserve compute resources.
Remove vault_4 from the cluster.
Verify that vault_4 has been removed from the cluster by viewing the raft configuration.
Add vault_4 back to the cluster
If you wish to add vault_4 back to the HA cluster, return to the vault_4 SSH session terminal and stop the vault_4 server.
Delete the existing data in the vault/vault_4
directory.
Start the vault_4 server.
You can again examine the peer set to confirm that vault_4 successfully joined the cluster as a follower.
Recovery mode for troubleshooting
In the case of an outage caused by corrupt entries in the storage backend, an operator may need to start Vault in recovery mode. In this mode, Vault runs with minimal capabilities and exposes a subset of its API.
Simulate outage
Stop the Vault service on all remaining cluster members, vault_2 and vault_3, to simulate an outage.
Return to the terminal where you SSH'd into vault_2 and stop the Vault service.
Stop Vault on vault_2.
Return to the terminal where you SSH'd into vault_3 and stop the Vault service.
Start in recovery mode
Return to the terminal where you SSH'd into vault_3 and start Vault in recovery mode.
The content after the @
symbol is appended to the vault server command
executed by this service. This is equivalent to running the vault server -config /etc/vault.d -recovery
.
View the status of the vault@-recovery
service.
Create a recovery operational token
Generate a temporary one-time password (OTP).
Start the generation of the recovery token with the OTP.
Example output:
View the recovery key that was generated during the setup of vault_2. In the vault_2 SSH session terminal, read the recovery key value.
Note
Recovery key is used instead of unseal key since this cluster has Transit auto-unseal configured.
Create an encoded token.
Enter the recovery key when prompted. The output looks simiar to below.
Complete the creation of a recovery token with the Encoded Token value and OTP.
Example output:
Fix the issue in the storage backend
In recovery mode Vault launches with a minimal API enabled. In this mode you are able to interact with the raw system backend.
Use the recovery token to list the contents at sys/raw/sys
.
Imagine in your investigation you discover that a value at a particular path is
the cause of the outage. To simulate this, assume that the value found at the
path sys/raw/sys/counters
is the cause of the outage.
Delete the path at sys/raw/sys/counters
.
Resume normal operations
First, stop the vault@-recovery
service.
Next, restart Vault service for vault_2 and vault_3.
Cluster reset
When a node is brought up in recovery mode, it resets the list of cluster members. This means that when resuming normal operations, each node will need to rejoin the cluster.
Clean up
Return to the first terminal where you created the cluster and use Terraform to destroy the cluster.
Destroy the AWS resources provisioned by Terraform.
Delete the state file.
Help and reference
- Integrated Storage
- Recovery Mode
- Raft GitHub repository
- [High Availability with Consul]