There are three telling security statistics in modern cloud and hybrid environments:<\/p>\n
Data breaches cost an average of $4.5 million per incident<\/a> These three stats compel security and platform teams to closely examine their approach to cloud infrastructure security workflows from Day-0 through Day-N. There are three primary challenges every security team must solve pertaining to remote infrastructure access and credential\/secrets management in the cloud:<\/p>\n How will we manage and secure human and machine access credentials to cloud accounts and resources? What if organizations could address those challenges within a single repeatable workflow? Better yet, what if they could wire up all the necessary configurations at provision-time, so every time a new cloud infrastructure resource is created, security teams can rest assured that the standard patterns of Security Lifecycle Management are applied by default.<\/p>\n These kinds of \u201cgolden\u201d approaches to infrastructure and security automation workflows lie at the core of The Infrastructure Cloud from HashiCorp<\/a>, which delivers Infrastructure and Security Lifecycle Management (ILM and SLM) via the HashiCorp Cloud Platform (HCP)<\/a>. This post illustrates using HCP services to build a secure provisioning workflow that automates solutions to the aforementioned challenges. <\/p>\n This diagram represents the \u201cgolden path\u201d workflow pattern modeled in this post:<\/p>\n While this example uses Microsoft Azure, the same workflow can be applied to any cloud provider, as well as to multi-cloud and hybrid cloud environments. You can follow along with building this golden path by using the full source code found in this GitHub repository<\/a>.<\/p>\n In this example scenario, an application development team needs to deploy their app to a Linux-based virtual machine (VM) running in Azure. The application needs to access a sensitive text string in order to execute (this could be an API token, database connection string, etc.) Also, despite a broad preference for immutable infrastructure<\/a>, developers and site reliability engineers may need SSH access to the VM to debug the application in case of an emergency.<\/p>\n The first step in creating the dev team\u2019s VM is to generate an admin SSH key-pair. The public key must be trusted by the VM and the private key must be stored securely with role-based access controls (RBAC) and auditing.<\/p>\n This golden path will use the core ILM component of The Infrastructure Cloud, HashiCorp Terraform<\/a>, to define and orchestrate all the pieces of this end-to-end secure provisioning workflow. Terraform is HashiCorp\u2019s infrastructure as code<\/a> solution and HCP Terraform<\/a> is a SaaS platform that manages Terraform provisioning runs<\/em> in a consistent and reliable environment. Key benefits of HCP Terraform include: <\/p>\n Secure management of shared state and secret data All the infrastructure created in this post will be codified with Terraform and committed to GitHub. This walkthrough uses a GitOps<\/a> workflow that integrates GitHub and HCP Terraform to trigger terraform plan and terraform apply as new commits are made. Configuring your HCP Terraform workspace <\/a>and VCS connection<\/a> is outside the scope of this blog.<\/p>\n Terraform lets you define resources<\/em> that represent infrastructure artifacts. One such resource is called tls_private_key<\/a>, which generates an SSH key pair:<\/p>\n #Create SSH keypair Next, you need to put your keys somewhere safe. HashiCorp Vault<\/a> is a centralized, API-driven secrets management system that handles the security lifecycle aspects of The Infrastructure Cloud. All the data in Vault is encrypted and protected with access-control list (ACL) policies. To avoid the burden of spinning up and managing your own Vault cluster, this post uses HCP Vault Dedicated<\/a>, the HashiCorp-managed version of Vault Enterprise<\/a>. <\/p>\n Terraform integrates with external systems via plugins known as providers<\/a>. Often a set of credentials are needed to access the external system APIs called by a provider. This introduces the complexity of securing those credentials, also called the “secure introduction” or “secret zero” problem<\/a>. Fortunately, HCP Terraform presents information about a Terraform workload to an external system \u2014 such as its workspace<\/a>, organization<\/a>, or whether it\u2019s a plan<\/em> or apply<\/em> \u2014 and allows other external systems to verify that the claims are genuine. This is called workload identity<\/a>. Each workload is issued a JSON Web Token (JWT) that is signed by HCP Terraform\u2019s private key, and expires at the end of the plan or apply timeout. <\/p>\n HCP Terraform\u2019s workload identity can be integrated with Vault to generate dynamic credentials for the Vault provider in your HCP Terraform runs. Dynamic provider credentials<\/a> improve your security posture by letting you provision new, temporary credentials for each run. This workflow eliminates the need to manually manage and rotate credentials across an organization.<\/p>\n To configure dynamic provider credentials for Vault, establish the trust between HCP Terraform and HCP Vault Dedicated<\/a>, set up a Vault role and policy, and configure your HCP Terraform workspace with the required variables<\/a> shown here:<\/p>\n #Upload the policy #Configure TF workload identity auth to Vault #Create the role You can now use Terraform\u2019s Vault provider<\/a> to create a new key-value secrets engine<\/a> path and stash the SSH keys there:<\/p>\n #Store All SSH keys in Vault KV Next, load the public key onto the machine. Your target cloud in this post is Microsoft Azure, so use Terraform\u2019s Azure provider<\/a> to create the VM and its associated networking components. The Azure provider\u2019s resource called azurerm_linux_virtual_machine<\/a> has an admin_ssh_key<\/a> block that defines the admin username and SSH public key. The VM resource configuration references the key-pair created earlier:<\/p>\n admin_ssh_key { When HCP Terraform applies the code, it will create SSH keys, store them in Vault, and provision the VM with the public key added to the SSH authorized_keys file \u2014 all in a single codified configuration.<\/p>\n The next step is to enable human operator access. This involves another product in the Infrastructure Cloud, HashiCorp Boundary<\/a>. Boundary is a modern privileged access management (PAM) solution, consisting of a control plane and workers that proxy sessions to infrastructure. You can use the HashiCorp-managed HCP Boundary<\/a> to run the control plane, register workers, and configure host endpoints with a default port and a protocol to establish a session (known as targets<\/em>). <\/p>\n The Boundary Terraform provider<\/a> orchestrates the steps required to enable human access to the VM, all within the provisioning workflow. Now register the new VM as an SSH target<\/a> in Boundary and configure a connection between Boundary and Vault<\/a> to let Boundary inject<\/a> the SSH private key into the user\u2019s session:<\/p>\n #Get the Boundary project resource “vault_token” “example” { #Create credential library for the SSH keys resource “boundary_host_set_static” “example” { The snippet above shows session recording<\/a> is enabled on the Boundary target. This is a critical enterprise feature that records every SSH session initiated to that host for later review by security teams and auditors.<\/p>\n By design, the Azure VM does not have a public IP, it’s on a private subnet. The users who need access to it reside outside the private network, so this is where Boundary\u2019s multi-hop<\/a> proxying capability comes in. If you place a Boundary worker<\/a> into the same Azure virtual network (VNet) as the VM, you can leverage it as a reverse proxy<\/a>. <\/p>\n The Boundary worker in Azure establishes a persistent outbound connection to an upstream worker that resides in HCP. The worker in Azure is configured as an egress worker<\/a> and given an azure tag and an egress worker filter is set on the Boundary target. The filter matches workers that have the tag azure. When users connect to the Boundary target, Boundary will route the connection only to the workers that match the tags set in the target filter. Here is a diagram to help you visualize the routing:<\/p>\n Boundary workers can run on VMs, bare-metal servers, and containers. In this example, the worker will run as a container. This is an efficient way to run Boundary workers since containers are lightweight, fast, and can scale as demand changes. Spin up your Boundary worker in an Azure Container Instance using Terraform:<\/p>\n resource “boundary_worker” “az_worker” { resource “azurerm_subnet” “cg” { service_delegation { resource “azurerm_container_group” “container” { container { ports { You now have a fully automated, codified, and secured access workflow for operator access to the VM. When a properly authorized<\/a> user logs into a Boundary client<\/a>, they will see the VM as an available SSH target. When they click the Connect button, Boundary pulls the SSH private key from the Vault key-value path where it’s stored and injects it into the session using protocol decryption<\/a>. The key is never exposed to the user. The user is then proxied and authenticated to the VM within a recorded and time-restricted session. Since Boundary exposes APIs, it can be easily integrated into service management solutions as part of an approval workflow. <\/p>\n Here you can see the target listed in the Boundary Desktop<\/a> UI:<\/p>\n Now that you\u2019ve solved two of the original challenges, it\u2019s time to address the final one: the workload running on the VM needs to retrieve a secret at runtime. <\/p>\n Having a central secrets management system is a core tenet of modern IT security, but it’s only as effective as the level of adoption by developers. Developers could implement application logic<\/a> to authenticate to Vault, retrieve secrets, and manage the lifecycle of the Vault token. However, this requires application refactoring to make it \u201cVault-aware\u201d, and developers often don\u2019t have the bandwidth or Vault knowledge needed to refactor. Security teams requiring developers to refactor hundreds (or thousands) of applications creates significant cost and time implications, establishing barriers to adoption of secrets management.<\/p>\n To help lower the barrier to adoption by providing a more scalable and simpler way for applications to integrate with Vault, HashiCorp created Vault Agent<\/a>. Vault Agent is a client daemon mode of the Vault binary. It has several uses, but one of its primary functions is to automatically authenticate to Vault, manage the lifecycle of the token, and fetch secrets from Vault, whether those are static secrets, dynamic credentials, or certificates. Your dev team\u2019s VM will leverage Vault Agent to render secrets for use by applications without the need for custom application logic.<\/p>\n To install Vault Agent on the VM, adhering to the principles of immutability and codification, you will use another component in The Infrastructure Cloud: HashiCorp Packer<\/a>. Packer will codify installation and configuration of the Vault Agent into the VM image, so it’s up and running immediately after provisioning the VM. The Packer CLI<\/a> builds the Azure VM image from an HCL template<\/a>, pushes the image artifact to Azure, and registers the image metadata to HCP Packer<\/a>. <\/p>\n HCP Packer is a hosted artifact registry that tracks the metadata of your golden images, with features including versioning, release channels, ancestry tracking, revocation, and Terraform integrations. This Terraform integration<\/a> is important because it lets developers reference a golden image<\/a> by querying HCP Packer rather than hard-coding the image identifier. This ensures that only fully tested, compliant, and patched images are used to provision VMs and containers.<\/p>\n Your VM will use an image built from a Packer HCL template using Azure\u2019s standard Canonical Ubuntu 22.04 LTS as the base. With the help of Packer\u2019s shell provisioner<\/a>, it will layer on the Vault Agent binary, its configuration file, a systemd<\/a> unit file, and a Consul template file<\/a> used to render the secret out to a text file:<\/p>\n #Stage Vault Keeping in mind the need to secure credentials across the workflow, you need to decide how Vault Agent should authenticate to Vault. There needs to be an initial credential\/identity to log in to Vault and get the token to use on subsequent requests for secrets. <\/p>\n One benefit of using a cloud platform to host infrastructure is that the platform creates identities for resources<\/a> and lets external systems validate those identities. Using Vault\u2019s Azure Authentication Method<\/a>, you can configure Vault Agent auto-auth<\/a> with the VM\u2019s Azure-managed identity JWT token and map the identity to a Vault role that has ACL policy access to the application secret. This resolves the \u201csecure introduction<\/a>\u201d problem by proving that the VM is a legitimate recipient for the secret and avoids managing yet another set of credentials. To configure the integration, log in to Vault, set up the Azure Authentication Method, and create a role for the agent:<\/p>\n #Upload the policy #Configure Azure auth method for Vault Agent #Create the role In the example configuration above, any resource from the Azure subscription can use the role “vault-agent-role” and access secrets based on its ACL policy “vault-agent”. You can, of course, define finer-grained bindings. <\/p>\n Here\u2019s the auto-auth snippet from the agent config file to include in the Packer image:<\/p>\n auto_auth { After building the image<\/a> and registering it in HCP Packer<\/a>, the next step is to configure Terraform to provision the VM from it. First you need to query the latest version from HCP Packer using the HCP provider<\/a>:<\/p>\n # Locate the Packer built image Then set an environment variable on the VM that tells Vault Agent the address of the Vault cluster to connect to. By using variables injected at provision time, you can use the same VM image across different environments such as dev, test, staging, and production. Use cloud-init<\/a> to do this:<\/p>\n #Injecting Vault cluster address to .env file used by systemd The code to provision the Linux VM resource references the image you looked up in HCP Packer<\/a> and inserts the cloud-init config into the VM\u2019s custom_data<\/a>. Notice the VM has a system-assigned identity:<\/p>\n resource “azurerm_linux_virtual_machine” “example” { All the pieces to solve the three security challenges are now codified with Terraform. Run an HCP Terraform plan in your workspace and review the output:<\/p>\n Now you have one provisioning workflow to create and store SSH keys, create a VM for an application from a golden image, access secrets, and register the VM into a remote access system.<\/p>\n Note that there is a Terraform run task<\/a> configured that validates the Packer image being used is valid and not revoked:<\/p>\n Apply the plan and provision everything. Log into the VM through Boundary to verify Vault Agent rendered the application secret to the file as expected:<\/p>\n While it might seem daunting to adopt all the components presented in this blog post, you can get started with an iterative approach. Signing up for the HashiCorp Cloud Platform gives you immediate access to free-tier versions<\/a> of all the products used in this tutorial.<\/p>\n It makes sense to start your Infrastructure Cloud journey<\/a> by getting your secrets under management with HCP Vault Dedicated, then move on to adopting infrastructure as code with HCP Terraform and scale up to codified image management with HCP Packer and identity-based access management with HCP Boundary. The investment to enable a secure provisioning workflow will be well worth it to strengthen and speed up your organization\u2019s ILM and SLMSecurity Lifecycle Management practices. <\/p>\n For more tutorials on using HashiCorp solutions, visit our HashiCorp Developer<\/a> education site and watch our demo video series<\/a>. If you\u2019d like to discuss your specific infrastructure and security transformation challenges, our sales and solutions engineers are here to listen<\/a>.<\/p>\n
\n74% of breaches<\/a> involve a human element
\n9 out of 10 web application breaches<\/a> result from stolen credentials<\/p>\n
\nHow will we enable humans to remotely access cloud resources with just-in-time credentials when the need arises (e.g. a \u201cbreak glass\u201d emergency situation or to debug an application)?
\nHow will we enable workloads running on the cloud to access secrets?<\/p>\nSecuring SSH access by default<\/h3>\n
\nAccess controls for approving changes to infrastructure
\nA private registry for sharing approved Terraform modules
\nDetailed policy controls for governing the contents of Terraform configurations
\nDay 2 visibility and optimization features like drift detection and a workspace explorer<\/p>\n
\nresource “tls_private_key” “ssh_key” {
\n algorithm = “RSA”
\n rsa_bits = 4096
\n}<\/p>\n
\nvault policy write tfc-workload tfc-workload-policy.hcl<\/p>\n
\nvault auth enable jwt
\nvault write auth\/jwt\/config
\n oidc_discovery_url=”https:\/\/app.terraform.io”
\n bound_issuer=”https:\/\/app.terraform.io”<\/p>\n
\nvault write auth\/jwt\/role\/tfc @vault-jwt-auth-role.json<\/p>\n
\nresource “vault_kv_secret_v2” “example” {
\n mount = “kv”
\n name = “ssh\/${var.vm_name}”
\n data_json = jsonencode(
\n {
\n public_key_openssh = tls_private_key.ssh_key.public_key_openssh,
\n private_key_openssh = tls_private_key.ssh_key.private_key_openssh,
\n public_key_pem = tls_private_key.ssh_key.public_key_pem,
\n private_key_pem = tls_private_key.ssh_key.private_key_pem
\n username = var.vm_admin
\n }
\n )
\n}<\/p>\n
\n username = var.vm_admin_username #adminuser
\n public_key = tls_private_key.ssh_key.public_key_openssh
\n }<\/p>\nHuman access to the virtual machine<\/h3>\n
\ndata “boundary_scope” “project” {
\n name = var.boundary_project_name
\n scope_id = var.boundary_scope_id
\n}<\/p>\n
\n policies = [“tfc-workload”]
\n no_parent = true
\n renewable = true
\n period = “24h”
\n metadata = {
\n “purpose” = “boundary credential store token”
\n }
\n}
\n#Create credential store
\nresource “boundary_credential_store_vault” “example” {
\n name = “HCP Vault”
\n description = “HCP Vault Credential Store”
\n address = var.vault_addr
\n namespace = var.vault_namespace
\n token = vault_token.example.client_token
\n scope_id = data.boundary_scope.project.id
\n}<\/p>\n
\nresource “boundary_credential_library_vault” “example” {
\n name = “${var.vm_name}-sshkeys”
\n description = “VM SSH keys”
\n credential_store_id = boundary_credential_store_vault.example.id
\n path = vault_kv_secret_v2.example.path
\n http_method = “GET”
\n credential_type = “ssh_private_key”
\n credential_mapping_overrides = {
\n private_key_attribute = “private_key_pem”
\n }
\n}
\n#Add the VM to Boundary
\nresource “boundary_host_catalog_static” “example” {
\n name = “azure-vm-catalog”
\n description = “Azure VM Host Catalog”
\n scope_id = data.boundary_scope.project.id
\n}
\nresource “boundary_host_static” “example” {
\n name = var.vm_name
\n host_catalog_id = boundary_host_catalog_static.example.id
\n address = azurerm_linux_virtual_machine.example.private_ip_address
\n}<\/p>\n
\n name = “azure-vm-host-set”
\n host_catalog_id = boundary_host_catalog_static.example.id
\n host_ids = [
\n boundary_host_static.example.id
\n ]
\n}
\nresource “boundary_target” “example” {
\n name = “${var.vm_name}-ssh”
\n description = “${var.vm_name}-SSH”
\n type = “ssh”
\n default_port = “22”
\n scope_id = data.boundary_scope.project.id
\n host_source_ids = [
\n boundary_host_set_static.example.id
\n ]
\n injected_application_credential_source_ids = [
\n boundary_credential_library_vault.example.id
\n ]
\n egress_worker_filter = var.boundary_egress_filter
\n enable_session_recording = true
\n storage_bucket_id = var.boundary_session_bucket
\n}
\nresource “boundary_alias_target” “example” {
\n name = “boundary_alias_target”
\n description = “Alias to target using host boundary_host_static.example”
\n scope_id = “global”
\n value = “ssh.${var.vm_name}.boundary”
\n destination_id = boundary_target.example.id
\n authorize_session_host_id = boundary_host_static.example.id
\n}<\/p>\n
\n scope_id = “global”
\n name = “azure boundary worker 1”
\n}<\/p>\n
\n name = “container-group-subnet”
\n resource_group_name = var.resource_group
\n virtual_network_name = azurerm_virtual_network.example.name
\n address_prefixes = [“10.0.3.0\/24”]
\n delegation {
\n name = “delegation”<\/p>\n
\n name = “Microsoft.ContainerInstance\/containerGroups”
\n actions = [“Microsoft.Network\/virtualNetworks\/subnets\/join\/action”, “Microsoft.Network\/virtualNetworks\/subnets\/prepareNetworkPolicies\/action”]
\n }
\n }
\n}<\/p>\n
\n name = “boundary-worker-group”
\n location = var.az_location
\n resource_group_name = var.resource_group
\n ip_address_type = “Private”
\n subnet_ids = [azurerm_subnet.cg.id]
\n os_type = “Linux”
\n restart_policy = “Never”<\/p>\n
\n name = “boundary-worker”
\n image = “hashicorp\/boundary-enterprise”
\n cpu = 1
\n memory = 2<\/p>\n
\n port = 9202
\n protocol = “TCP”
\n }
\n environment_variables = { “HCP_BOUNDARY_CLUSTER_ID” = var.hcp_boundary_cluster_id, “WORKER_ACTV_TOKEN” = boundary_worker.az_worker.controller_generated_activation_token }
\n volume {
\n name = “boundary-config”
\n mount_path = “\/boundary”
\n git_repo {
\n url = “https:\/\/github.com\/bfbarkhouse\/hashistack-secure-infra-workflow”
\n }
\n }
\n commands = [
\n “\/bin\/sh”, “-c”, “mv \/boundary\/hashistack-secure-infra-workflow\/azure\/boundary-worker-config.hcl \/boundary\/config.hcl; rm -rf \/boundary\/hashistack-secure-infra-workflow; \/usr\/local\/bin\/docker-entrypoint.sh server -config \/boundary\/config.hcl”
\n ]
\n }
\n}<\/p>\nAutomatic application secret retrieval<\/h3>\n
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [
\n “wget -O- https:\/\/apt.releases.hashicorp.com\/gpg | sudo gpg –dearmor -o \/usr\/share\/keyrings\/hashicorp-archive-keyring.gpg”,
\n “echo “deb [signed-by=\/usr\/share\/keyrings\/hashicorp-archive-keyring.gpg] https:\/\/apt.releases.hashicorp.com $(lsb_release -cs) main” | sudo tee \/etc\/apt\/sources.list.d\/hashicorp.list”,
\n “apt update && sudo apt install vault”
\n ]
\n inline_shebang = “\/bin\/sh -x”
\n }
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [“mkdir \/vault”]
\n }
\n provisioner “file” {
\n destination = “\/tmp\/agent-config.hcl”
\n source = “.\/agent-config.hcl”
\n }
\n provisioner “file” {
\n destination = “\/tmp\/app-secret.ctmpl”
\n source = “.\/app-secret.ctmpl”
\n }
\n provisioner “file” {
\n destination = “\/tmp\/vault.service”
\n source = “.\/vault.service”
\n }
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [“mv \/tmp\/agent-config.hcl \/etc\/vault.d”]
\n }
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [“mv \/tmp\/app-secret.ctmpl \/etc\/vault.d”]
\n }
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [“mv \/tmp\/vault.service \/usr\/lib\/systemd\/system”]
\n }
\n provisioner “shell” {
\n execute_command = “chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh ‘{{ .Path }}'”
\n inline = [“systemctl enable vault.service”]
\n }
\n}<\/p>\n
\nvault policy write vault-agent vault-agent-policy.hcl<\/p>\n
\nvault auth enable azure
\nvault write auth\/azure\/config
\n tenant_id=$YOUR_TENANT_ID
\n resource=https:\/\/management.azure.com\/
\n client_id=$YOUR_CLIENT_ID
\n client_secret=$YOUR_CLIENT_SECRET<\/p>\n
\nvault write auth\/azure\/role\/vault-agent-role
\n policies=”vault-agent”
\n bound_subscription_ids=$YOUR_SUBSCRIPTION_ID<\/p>\n
\n method {
\n type = “azure”
\n namespace = “admin”
\n config = {
\n authenticate_from_environment = true
\n role = “vault-agent-role”
\n resource = “https:\/\/management.azure.com\/”
\n }
\n }<\/p>\n
\ndata “hcp_packer_artifact” “secure-infra-workflow” {
\n bucket_name = var.packer_bucket_name
\n channel_name = var.packer_channel_name
\n platform = var.packer_platform
\n region = var.packer_region
\n}<\/p>\n
\ndata “template_cloudinit_config” “vault-config” {
\n gzip = true
\n base64_encode = true
\n part {
\n content_type = “text\/cloud-config”
\n content = “bootcmd: [echo VAULT_ADDR=${var.vault_addr} >> \/etc\/vault.d\/vault.env]”
\n }
\n}<\/p>\n
\n name = var.vm_name
\n resource_group_name = var.resource_group
\n location = var.az_location
\n size = var.vm_size
\n source_image_id = data.hcp_packer_artifact.secure-infra-workflow.external_identifier
\n admin_username = var.vm_admin
\n custom_data = data.template_cloudinit_config.vault-config.rendered
\n identity {
\n type = “SystemAssigned”
\n }<\/p>\nPutting it all together<\/h3>\n
Getting started with an iterative approach<\/h2>\n