Go to main content | Go to main menu

Guide: Secret management in a system development lifecycle

In this guide, we discuss the secret management, why it is important, and different options in choosing the right approach.

30 Oct 2024 DevOps

No description

This text describes different options in secret management. It is not a complete tutorial. It is a collection of our experience and research. Your own work is expected.

In the first part, we will describe what is secret management, motivation behind secret management, and key features of tools which can help us establish a secure environment.

The second part discusses secret management in production on remote machines like servers. Then, this part describes secret management service (SMS) and introduces some industry standard tools.

The third part discusses deployment with Ansible, as well as tools that can help redistribute secrets between developers. Some of the mentioned tools can encrypt secrets, which can then be uploaded to services like Github and Gitlab without undermining security.

And finally, the fourth part discusses why compiled languages are not safe, and why you should not store secrets in them.

The topic of secret detection is discussed in another guide.

TLDR

  • Start somewhere, even if it is not perfect.
  • Hardcoded secrets in the code are always bad.
  • Plaintext secrets should never be stored in git.
  • Optimal production methods include Hashicorp Vault, Azure Key Vault, etc.
  • Less optimal, but still good solutions are Ansible Vault, Pass, Gopass, etc.
  • Whether you use tool X or Y, anything is better than plaintext passwords in your git repository.
  • Something must save the secrets on the disk or in the memory. It is possible to extract them in case you have sufficient access.

What is secret management

Secret management is a practice that allows developers and administrators to securely store sensitive data, such as passwords, keys, and tokens, in a secure environment with strict access controls. Secret management tools include:

Motivation

We’re trying to make it as difficult as possible to compromise the entire system. Many times an attacker can only achieve a partial exploitation of the system, for example by reading files on a web server (Local File Inclusion) or finding the password to a database on Github of a given organization. This allows them to move between services; the attacker can now access the database and further expand the level of compromise of the system from there. It is important to understand that secret management protects the service only from a certain level of access to the system. Once the attacker has access to an account that has sufficient rights, he cannot be prevented from gaining secrets. By the nature of the problem, it is impossible to ensure that a legitimate service has access to secrets, and at the same time, an attacker with the same rights as the service does not have it.

Understanding Secret Management

Secret management is a process of storing and distributing secrets. Secrets are passwords, API keys, certificates, or anything else that is sensitive and should not be publicly available

There is no optimal solution for secret management. Every solution has its pros and cons. It always will be stored on HDD or in memory, so it is always possible to extract it. The goal is to make it as hard as possible. The ideal secret management tool should be:

  • Easy to use
  • Easy to distribute secrets
  • Easy to rotate the secrets and manage access to them

Be aware that secrets might also be found in the following places:

  • Git history
  • Logs
  • Bash history

Production tools (used on remote machines)

There are multiple ways to manage secrets in production. This chapter is about the most common ones. Automation should take care of the secrets to reduce the human factor.

OWASP Secret management cheatsheet describes three features that makes secrets more secure:

  • Secrets pipeline: Having a secrets pipeline that does large parts of the secret management (E.g. creation, rotation, etc.)
  • Using dynamic secrets: When an application starts, it could request its database credentials, which will be dynamically generated and provided for that session. Dynamic secrets are reducing the surface area of credential re-use. If the application’s database credentials are stolen, they will expire.
  • Automated rotation of static secrets: Key rotation is a challenge when implemented manually, and it can lead to mistakes. It is better to automate the rotation of keys, or at least ensure that it sufficiently supports the process.

Achieving this takes work. However, the secret management services can help with this.

Secret management services (SMS)

Services that can be used for secret management:

  • Hashicorp Vault
  • Azure Key Vault
  • Cloud KMS
  • AWS Secrets Manager
  • Infisical
  • Sops …

These solutions are optimal (not perfect). However, it is not worth building the stack yourself without the support infrastructure, operations, and resources for these tools.

A developer without the infrastructure is left with the following tools:

  • Environment variables
  • Files
  • Encrypted files
  • A combination of the above

The reason why they are the optimal solution:

  • You can audit access, as well as manage and rotate the secrets as needed.
  • You can use multi-factor authentication and other security measures (like private key + secret token, IP whitelist, etc.) to protect the secrets.
  • The application uses tokens to access the secrets, not the secrets directly.

Pros:

  • You can rotate the secrets without changing the token and vice versa.
  • You can revoke the access to the secrets.
  • You can audit the access to the secrets.
  • A compromised token does not mean compromised service right away.
  • Easier secret distribution and management.

Cons:

  • You still have to store the tokens somewhere.
  • You still have to be careful with pushing the tokens to git.
  • Usually, you have to pay for the service, or you have to build the infrastructure yourself.
  • You need a secret management server and a client.

Demo

As an example, we will use Hashicorp Vault for storing PostgreSQL credentials. The demo is from the official documentation.

# enable database secrets
vault secrets enable database
# configure database connection
vault write database/config/postgresql \
     plugin_name=postgresql-database-plugin \
     connection_url="postgresql://{{username}}:{{password}}@$POSTGRES_URL/postgres?sslmode=disable" \
     allowed_roles=readonly \
     username="root" \
     password="rootpassword"

#configure role
tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' INHERIT;
GRANT ro TO "{{name}}";
EOF
# create role
vault write database/roles/readonly \
      db_name=postgresql \
      creation_statements=@readonly.sql \
      default_ttl=1h \
      max_ttl=24h

Now, secrets are generated on the fly and valid for 1 hour. They still can be retrieved, but Vault makes it harder to extract them. Also, they can’t be used immediately when retrieving the token with Local File Inclusion. The list of supported databases is available here.

Environment variables and files

This is the easiest way to store secrets. Using only environment variables is not persistent. This is why we utilize files like .env or config.yml to store secrets. These files should be ignored by git and should be distributed by other means, like Ansible, or manually.

Bash

VAR1=secret1 VAR2=secret2 ./my_app

# Print variable
echo $VAR1

PowerShell

$Env:password="secret12321" ./my_app

# Print variable
get-item -path Env:password | select -exp Value

Loading them from a file like .env or config.yml is language-specific.

Files and environment variables are not great, but still, they are better than plaintext passwords in git.

Windows secret management

Although you can use the mentioned tools via WSL, there are tools that are Windows-native.

PowerShell

SecretStore and SecretManagement modules allow working with secrets within PowerShell. By default, secrets are stored locally via Windows Data Protection API. SecretManagement module helps with local and remote vaults that can be registered and unregistered for use in accessing and retrieving secrets.

It supports integration with the following vaults:

The advantage over environment variables is that in order to get the secret, the attacker must get the user account and the files with the secrets. Reading the files alone is not enough.

Example usage
Register-SecretVault -Name SECRET_VAULT -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault

Set-SecretStoreConfiguration -Scope CurrentUser -Authentication Password -PasswordTimeout 60 -Interaction None -Password $pass -Confirm:$false
Set-Secret  -Name test_secret

$pass = Import-CliXml -Path "./encrypted_pass.xml"
Unlock-SecretStore -Password $pass
$secret=(Get-Secret  -Name test_secret -asplaintext)

NOTE: If this module is used on UNIX-like systems, secrets are stored in plaintext. These modules should only be used on Windows.

Resources

Dotnet

Like PowerShell, dotnet has its own very similar secret management module. Secrets are stored in plaintext in the %APPDATA%\Microsoft\UserSecrets\<user_secrets_id>\secrets.json file, but that is enough to keep your code without sensitive information. More on this topic is in the article Safe storage of app secrets in development in ASP.NET Core.

Example

dotnet user-secrets init
dotnet user-secrets set "Movies:ServiceApiKey" "12345"

...

var builder = WebApplication.CreateBuilder(args);
var movieApiKey = builder.Configuration["Movies:ServiceApiKey"];

IIS Web Configuration

You can use the ASP.NET IIS Registration Tool (Aspnet_regiis.exe) to encrypt or decrypt sections of a Web configuration file. ASP.NET will automatically decrypt encrypted configuration elements when the Web.config file is processed. (Source)

Note: Web configuration files must be decrypted after deployment on a particular host, as sharing encrypted configuration between multiple hosts won’t work.

aspnet_regiis -pe "ApiKey" -app "/App" -prov "RsaProtectedConfigurationProvider"

Deployment and development tools (used locally)

Secret management services can also be used in the deployment and development stages.

Ansible vault

Ansible vault is a command-line tool that encrypts and decrypts files. Using it to manage secrets in Ansible playbooks and roles is very easy. 

  • It is integrated with Ansible to encrypt variables in playbooks on the fly.
  • You can encrypt any file, but it is most useful for encrypting variables.
  • Ansible vault is included in the Ansible core.
  • Vault is managed by only one password.
  • There can be multiple vaults in one project.
  • Deployment and source control with the ansible-vault is elegant, if you have a way, how to distribute the master passwords (passwords used to encrypt other passwords; Ansible vault masterpassword encrypts the entire vault) to all developers.

Example use

Encrypting a file

Create file vars.yml with the following content:

---
password: heslo-je-maslo

Ansible vault

# Encrypt a file, you will be prompted for a password
ansible-vault encrypt vars.yml

# Show the encrypted file
ansible-vault view vars.yml

# Edit the encrypted file
ansible-vault edit vars.yml

This will create a file with the following content:

$ANSIBLE_VAULT;1.1;AES256
373263656562383930346536323230... # encrypted content
Encrypt the value of a variable #
echo 'secr3t-Passw0rd12334' | ansible-vault encrypt_string --stdin-name 'password_value'

This will yield the following string

password_value: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62346562393839613965323835666634306566616537393239626432353838326562326164303233
          3831383861303463373432313662653139633237306162630a383163346236633065346138626263
          34343363396236386335303534636530623632333432633334633538666233326365333732333731
          3737383033373738380a623835336661343962386432363130633266666433616462666636636131
          32353162636463373861336435373363623862613135336239663961373663666266

Copy this to the vars.yml file.

Usage in a playbook or role 
# Run ansible command
ansible localhost -m ansible.builtin.debug -a var="password_value" -e "@vars.yml" --ask-vault-pass

# Output
localhost | SUCCESS => {
    "password": "heslo-je-maslo"
}

# Or run the playbook with pass in the file
ansible-playbook playbook.yml --vault-password-file ~/.ansible-prod-password.txt

Amber

Amber is an alternative Ansible vault, more suitable for usage on remote machines. master password is used. No external dependencies are needed. It is like Pass or SecretManager for PowerShell. However, no GPG key or user account is required, just the master password.

Passbolt

Passbolt is the open-source password manager for teams. It is a web application, that allows you to share and store credentials securely. Passbolt has an REST API and CLI client. However, we recommend something other than this tool for usage in the automation process. You cannot create group-level or object-level access tokens, only user-level permissions, which are not suitable for any other use than manual usage.

Pass

Pass is a standard UNIX password manager. It is the CLI tool based on GPG authentication. It can be integrated with git. It is very versatile. You can use multiple GPG keys for the same and different passwords. Pass is similar to Ansible vault; however, it is not integrated directly with Ansible. The community provides integration using a plugin. Definitely usable, but the learning curve is higher. The module community.general.passwordstore provides a way to use Pass in Ansible. Note, that it is not included in the Ansible core.

Pass: Ansible integration

# Setup GPG
gpg --full-generate-key

# Generate password
pass generate <password-name> <length>

# Show password
pass <password-name>

# To get encrypted variable in Ansible
ansible localhost -m ansible.builtin.debug -a var="password" -e "password={{ lookup('community.general.passwordstore', 'heslo') }}"

# Output
localhost | SUCCESS => {
    "password": "jfKuBd0l4Nt3"
}

Gopass

Gopass is a "slightly more awesome standard UNIX password manager for teams". As the name suggests, it is similar to Pass but solves issues like native multi-user support and other valuable features. It can be used in automation. Compatible with Pass storages, so you don’t have to migrate your passwords. The passwordstore Ansible plugin is compatible with it.

Usage 

# Spawn environment with secrets
 gopass env <password-name> /usr/bin/bash -c "printenv <PASSWORD-NAME>"

# Output
jfKuBd0l4Nt3

Git-secret

Git-secret encrypts files in git, using a public key of the specified users.

Compiled languages are not safe

You might argue that you don’t have to manage secrets in compiled languages. Compiled languages like Go, Rust, and C++ are different because they are compiled to machine code, and extracting secrets from binary is harder. However, it is still possible. Sure, you can use encryption, but then you have to store the key somewhere, what constitutes the same problem. This is security by obscurity, which is not a good practice. As stated in the NIST SP800 (page 15): “System security should not depend on the secrecy of the implementation or its components.”

Contact

If you want to know more or need help with deployment, feel free to contact us at csirt@muni.cz.

You are running an old browser version. We recommend updating your browser to its latest version.

More info