Architecture

1. Vault Components

1.1 Storage Backends

• Configures location for storage of vault data
• Defined in main vault configuration file
• Data is encrypted in transit (TLS) and at rest using AES256
• Some storages are for high availability and others for better management and data protection
• Only one storage backend per vault cluster, all the nodes share the same db backend, one node is active and rest are standby

Note

• Replication can be done among different clusters but they never talk to each other’s storage backend, they talk to nodes. This allows replication cluster can use different backends irrespective of others backend

• Supported Storage backend documentation is available 🌐
• Docker setup of Vault 🌐

Tip

How to choose a vault storage backend?

graph TD
    %% Define Node Styles
    classDef decision fill:#395697,stroke:#fff,stroke-width:2px,color:#fff;
    classDef endpoint fill:#f9f977,stroke:#333,stroke-width:1px,color:#000;

    %% Define Nodes
    A{Production?}
    B[In-Memory]
    C{HA Support?}
    D{HashiCorp Supported?}
    E[Filesystem]
    F["Azure, 
Cassandra, 
CockroachDB, 
CouchDB, 
Manta, 
MSSQL, 
S3, 
Swift"]
    G{HashiCorp Supported?}
    H["Consul,
Raft"]
    I["DynamoDB, 
Etcd, 
FoundationDB, 
MySQL, 
Google Cloud Spanner, 
Google Cloud Storage, 
PostgreSQL, 
Zookeeper"]

    %% Define Edges
    A -- "No" --> B
    A -- "Yes" --> C

    C -- "No" --> D
    D -- "Yes" --> E
    D -- "No" --> F

    C -- "Yes" --> G
    G -- "Yes" --> H
    G -- "No" --> I

    %% Apply Styles to Nodes
    class A,C,D,G decision;
    class B,E,F,H,I endpoint;

1.2 Secrets Engines

• Manages secrets
• Secret engines can store, generate and encrypt data
• Many engines can connect to other services to generate dynamic credentials on demand
• There can be multiple instances of same type of secret engine
• Secret engines are mounted at path and all interactions are done with the path itself

1.3 Authentication methods

• Manages authentication and manages identities
• Responsible for assigning identity and policies to users
• Multiple authentication methods can be enabled
• There are 2 types of authentication methods classes: human and machine
• Once authenticated, vault issues a token to the client or user to make all subsequent vault requests until TTL
• Each token has a policy and a TTL
• Default authentication method for vault is also token based

1.4 Audit devices

• Keeps detailed log for all requests and responses to vault
• More than one audit devices can be enabled
• Audit log is formatted using json
• Sensitive info is hashed before logging
• There can be multiple audit devices enabled
• Vault required at least one audit device to write the log before completing the vault request - if enabled
• Prefers safety over availability
• Log files should be protected as a user with a permission can still check the values of those secrets via the /sts/audit-hash API and compare to the log file

1.4.1 Types of Audit Devices

1.4.1.1 File

• Writes to a local file
• Does not assist with log rotation
• Use fluent or similar tool to send to collector and a tool to rotate logs

1.4.1.2 SysLog

• Writes logs to a local syslog upstream sever
• Send those logs to a local agent and local agent send that upstream to desired location

1.4.1.3 Socket

• Writes to TCP, UDP or UNIX sockets
• Unreliable due to underlying protocol (if its UDP)
• Should be used where strong guarantees are required

Command	Effect
vault audit enable file file_path=FILE_PATH	Enables file based audit device

2 The Encryption Barrier

• More information can be seen at 🌐

2.1 Vault Paths

• Everything in vault is path based
• The path prefix tells vault which component a request should be routed
• Secret engines, authentication methods and audit devices are mounted at paths
• Paths available are dependent on features that are enabled in the vault
• System backend is the default backend in vault which is mounted at /sys endpoint
• Every vault component has its own default path

2.2 Vault Data Protection

• Master key protects encryption key
• Encryption key protects vault data
• Encryption key is stored in vault node memory

2.3 Master Key

• Used to encrypt and decrypt encryption key
• Created using initialization and re key operations
• Never written to storage when using traditional unseal mechanism
• Written to core/master (storage backend) when using auto unseal

2.4 Encryption Key

• Used to encrypt and decrypt the vault data written to storage backend
• Encrypted by the master key
• Stored alongside the data in a keyring on the storage backend
• Can be easily rotated

2.5 Seal and Unseal

• Vault starts in sealed state, meaning it knows where to access the data but can not decrypt it. Users can seal it manually as well
• Almost no operations are possible when the vault is in sealed state (only status check and unsealing are possible)
• Unsealing vault means that a node can reconstruct the master key in order to decrypt the encryption key, and ultimately read the data
• After unsealing, the encryption key is stored in memory
• Every node in the cluster should be unsealed to do any operations
• Sealing vault means it throws away the encryption key and requires another unseal to perform any further operations

Caution

When to seal vault?

• Key shards are exposed
• Detection of a breach or compromise of network
• Spyware/malware on the vault nodes

2.5.1 Seal and Unseal Options

2.5.1.1 Key Sharding (Shamir algorithm)

• The master key is broken into N parts and given to N people
• While unsealing the vault,it requires a threshold of keys, threshold can be set at vault initialization
• It is the default option
• No single person should have access to all those shards
• Shards should not be stored online

Tip

We can have PGP keys of people before initialization of vault so that even if keys given by vault, the keys are only readable by respective person only. This increases the security of the key. The minimum number of keys required is equal to number of threshold. Also, the root token can be encrypted as well

Command	Effect
vault status	See vault status
vault operator init	Initializes vault and gives root token, and key shards
vault operator unseal	Initiate vault unseal or progress it, do this until the threshold is reached
vault login ROOT_TOKEN	Login using root token
vault operator seal	Seal the vault, requires root privilages on login token

2.5.1.2 Cloud Auto Unseal (cloud key management like KMS)

• Use cloud based key management solution to encrypt and decrypt master key or a on prem HSM
• Master key is written to storage backend but is encrypted using cloud KMS
• Vault configuration file identifies the key use for decryption
• Cloud auto unseal automatically unseals the vault upon restart of node or service
• When using auto unseal, it provides recovery keys instead of unseal keys
• Vault is initialized in unsealed state as it reaches out to cloud KMS key and auto unseals itself
• We can disable the KMS key to seal the vault

Note

Seal configuration for AWS: config.hcl

seal "awskms" {
    region = "REGION"
    kms_key_id = "KEY_ID/KEY_ARN"
    # Other parameters are not required if using AWS IAM roles for key access
    # Else AWS keys need to be specified
    ...
}

# After this, restart vault and initialize

Similarly it can be configured fro other major cloud providers

2.5.1.3 Transit Auto Unseal

• Utilizes another vault’s transit secrets engine to seal and unseal itself
• Returns recovery keys instead of unseal keys
• The transit secret engine may be configured in a namespace
• Supports key rotation
• This minimizes the overhead of creation of KMS key for all vaults in an organization and have better centralized protection
• But all are dependent on some other vault except one, if the main one stops or is sealed, others stop working too (hierarchy)
• The core vault must be highly available (multi node, HA storage backend)
• Sets recovery seal type to transit

Note

Transit auto unseal can be configured like this

seal "transit" {
    # Address configuration
    address = "DIFFERENT_VAULT_ADDRESS"
    token = "TOKEN" # token to use to access other vault
    disable_renewal = "true/false"

    # Key configuration
    key_name = "KEY" # key to use to unseal self
    mount_path = "SECRET_ENGINE_PATH" # secret engine path on other vault
    namespace = "ns1" # see docs

    # TLS configuration
    tls_ca_cert = "PATH"
    tls_client_cert = "PATH"
    tls_client_key = "PATH"
    # tls_client_cert and tls_client_key enable mutual TLS (mTLS): the client presents a certificate and proves possession of its private key so the server can authenticate the client at the TLS layers
    # mTLS is required only if the transit Vault endpoint (or a fronting load balancer) is configured to request/require client certificates; otherwise these fields are unused
    tls_server_name = "NAME"
    # The tls_server_name sets the Server Name Indication (SNI) host used in the TLS handshake when connecting to the transit Vault endpoint
    # This is useful when connecting via IPs, internal service names, or load balancers whose backend cert common name/subjectAltName does not match the configured address field
    # It overrides the host used for TLS verification
    # Add it only if verification fails due to a name mismatch
    # Set it to the certificate’s expected DNS name if it differs from the address used, ensuring hostname verification succeeds
    #
    tls_skip_verify = "true/false"
}

Tip

Pros

Key Shards	Auto Unseal	Transit Unseal
Simple	Auto unsealing	AUto matic unsealing
Works on any platform	Set and forget	Set and forget
Config options make it flexible	Utilize integrated features of cloud platform	Platform agnostic
-	-	Multiple clusters

Cons

Key Shards	Auto Unseal	Transit Unseal
Human error prone (risky)	Regional requirements for cloud HSMs	Requires central vault cluster
Requires manual intervention	Cloud / Vendor lock in	High availability required for central cluster
Keys can be required to rotate in case exposed	-	-