Vault Agent Templates
The adoption of Vault is an incremental journey. First, you move your secrets into Vault so that they are securely encrypted and stored. The next step is to update your applications' behavior to use secrets from Vault.
Clients must authenticate with Vault and get a token before accessing authenticated endpoints. Vault Agent was introduced to reduce the credential lifecycle management burden for distributed applications.
![Vault Agent](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault%252Fdiagram-vault_agent-workflow.png%26width%3D850%26height%3D256&w=1920&q=75)
The following tutorials demonstrate the Vault Agent Auto-Auth method:
In addition, the Vault Agent Caching tutorial walked through the Caching feature introduced in Vault 1.1.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Challenge and solution
After the successful authentication, Vault clients can start interacting with the Vault. Many Vault users adopted the Consul Template tool to minimize the level of changes introduced to their existing applications.
![Vault and Consul Template](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault-agent-templates.png%26width%3D904%26height%3D293&w=1920&q=75)
The Direct Application Integration tutorial demonstrates the use of Consul Template for Vault users.
In this case, Consul Template is the client directly interacting with Vault; therefore, you had to configure Consul Template to read the token from the agent's sink location in order to interact with Vault. This requires you to operate two distinct tools to provide secrets to applications.
In Vault 1.3, Vault Agent introduced Vault Agent Templates allowing Vault secrets to be rendered to files using the Consul Template markup language. This significantly simplifies the workflow when you are integrating your applications with Vault.
Note
This tutorial focuses on the templates feature of Vault Agent assuming that you are already familiar with Vault Agent Auto-Auth. If you are new to Vault Agent, try the Vault Agent with AWS tutorial or the Vault Agent with Kubernetes tutorial before continuing with this tutorial.
Prerequisites
To complete this section of the tutorial, you will need:
- AWS account and associated credentials that allow for the creation of resources
- Vault version 1.3 or later
- Terraform installed
Provision the cloud resources
Clone the demo assets from the hashicorp-education/learn-vault-agent-templates GitHub repository to perform the steps described in this tutorial.
This repository contains supporting content for all of the tutorial.
Be sure to set your working directory to where the
terraform-aws
directory is located.The following assets are located under this directory:
Example output:
Note
The example Terraform configuration in this repository was designed just for demonstration purposes.
Set an
AWS_ACCESS_KEY_ID
environment variable to hold your AWS access key ID.Set an
AWS_SECRET_ACCESS_KEY
environment variable to hold your AWS secret access key.Tip
The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.
Create a file named
terraform.tfvars
and specify thekey_name
. (You can copy theterraform.tfvars.example
file and rename it asterraform.tfvars
.)Example
terraform.tfvars
:Tip
If you don't have an EC2 Key Pair, refer to the AWS documentation and create one.
Run
terraform init
to initialized the Terraform workspace, and download the necessary provider resources.Run
terraform plan
to verify your changes, and the resources that Terraform will create when the plan gets applied.If all looks good, run
terraform apply
to provision the resources; in this example, approval to apply the plan is skipped for convenience with the-auto-approve
flag.
Note
This step creates actual real-world resources in AWS. Be mindful of their presence going forward, and be sure to follow the suggestion clean up steps at the conclusion of this tutorial.
Example expected output:
The Terraform output will display the public IP address to SSH into your Vault server and client instances.
Configure AWS IAM auth method
Note
This step should be performed on the Vault server instance.
In this step, you configure Vault to allow AWS IAM authentication from specific IAM roles.
SSH into the Vault server instance.
When you are prompted, enter "yes" to continue.
Example expected output:
Note
If you get a
Permissions 0664 for '<key_name>.pem' are too open
error, be sure to appropriately set the file permission.Run the
vault operator init
command to initialize Vault. For the convenience of this tutorial, save the generated recovery keys and the initial root token in a file namedkey.txt
.The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.
Log into Vault using the generated initial root token save in the
key.txt
Examine the
/home/ubuntu/aws_auth.sh
script.Execute the
/home/ubuntu/aws_auth.sh
script.The script enabled key/value v2 secrets engine at
secret
and wrote some data atsecret/customers/acme
. It also created anapp-pol
policy, enabled and configuredaws
auth method, and created a role namedapp-role
.Read the
app-pol
policy.Get the secrets written at
secret/customers/acme
Start Vault Agent
Note
This step should be performed on the Vault Client instance.
In the client node, Vault Agent authenticates with Vault via aws
auth method
you configured in the configure AWS IAM auth
method step. Using the acquired token, the
agent pulls secrets from Vault defined by the template file.
![Scenario Diagram](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault-agent-templates-2.png%26width%3D1133%26height%3D371&w=3840&q=75)
SSH into the Vault Client instance.
When you are prompted, enter "yes" to continue.
Example output:
Explore the Vault Agent configuration file,
/home/ubuntu/vault-agent.hcl
.The contents of the file resembles this example:
The Vault Agent uses the
aws
auth method to authenticate with the Vault server running on the server instance asapp-role
. Also notice that there istemplate
block which sets/home/ubuntu/customer.tmpl
as the source template file and the output will be written to the destination,/home/ubuntu/customer.txt
.Execute the following command to start the Vault Agent with log level set to
debug
.You should get output similar to the following:
Open a second SSH terminal into the client machine.
Verify the client token stored in
/home/ubuntu/vault-token-via-agent
.Example output:
Check the details about the token (e.g. attached policies, TTL, etc.).
Example output:
The
app-pol
policy is attached to the token. Remember that theapp-pol
policy grants read operation against thesecret/data/*
path.Review the
customer.tmpl
file which is written using the Consul Templates markup language.Example output:
Notice that the secret path is set to
secret/data/customers/acme
.Examine the resulting
customer.txt
file.Based on the
customer.tmpl
file, Vault Agent Templates read the secrets atsecret/data/customers/acme
and then generated thecustomer.txt
file.
Challenge
What happens if the data at secret/data/customers/acme
was updated?
On the server instance terminal, update the
contact_email
tojenn@acme.com
.Verify updates to the data.
Return to the client instance SSH session where the Vault Agent is running. Wait for a few minutes (about 4 minutes). The agent pulls the secrets again.
When reading secrets from Key/Value v2 secrets engine, omitting the
?version
parameter reads the latest version of the secrets at the path. If you want to always pull a specific version of the secret, specify the desired version number. For example,{{ with secret "secret/data/customers/acme?version=3" }}
will always read the version 3 of thesecret/data/customers/acme
values.
Clean up
Execute the following commands to destroy cloud resources.
At this point, you can delete the Terraform state files from the directory.
Summary and reference
Vault Agent is a client daemon that solves the secret-zero problem by authenticating with Vault and manage the client tokens on behalf of the client applications. The Consul Template tool is widely adopted by the Vault users since it allowed applications to be "Vault-unaware".
Vault Agent Templates combines the best of the two tools to make the end-to-end workflow even simpler.
![Scenario Diagram](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault-agent-templates-2.png%26width%3D1133%26height%3D371&w=3840&q=75)
Help and reference
- Blog post: Why Use the Vault Agent for Secrets Management?
- Video: Streamline Secrets Management with Vault Agent and Vault 0.11
- Vault Agent Templates
- Consul Template - Templating Language
- Direct Application Integration