In this article, I’ll explain how we manage secrets data at BaseCRM Kubernetes infrastructures using Helm.
Our goal with Helm is to reuse parts of Helm Charts in BaseCRM Kubernetes clusters with a minimal effort and to manage only values and secrets. There is no official secrets management in Helm or Kubernetes, so we decided to change that.
Kubernetes is a platform for automating and scaling containerized application deployments in a clustered world, and it’s developed by Google with big community support. Kubernetes is a portable, flexible and self-healing system for building next generation PaaS infrastructures.
On top of Kubernetes we have Helm. Helm is an official package manager for Kubernetes that helps manage multiple types of applications inside Kubernetes with a templated Charts system.
Helm Charts helps in defining, installing and upgrading reusable application templates inside a Kubernetes cluster even that most complex containing multiple dependent sub-charts. With Helm Charts we can easily create and manage multiple version and stop using direct manifests in Kubernetes. With direct manifest there is only one way to manage by copy-and paste from other manifests which is unacceptable in more complex environments.
With Helm we can split data from charts templates and use with any complex application charts that are reusable in Kubernetes PaaS deployments in multiple clusters. We can easily deploy whole application stack with CI/CD on newly created Kubernetes cluster or just run Helm manually and even deploy whole world from the scratch.
What is the problem we are trying to solve?
We all need passwords and access tokens for infrastructure to be secure, yet easily accessible and distributed according to clear rules. Plus we’d all love to have full change auditability along with secrets versioning, logs and maybe even secret change alerts. At the same time we want to be able to rebuild Kubernetes clusters and there are always some credentials that should not change.
Some may want to keep secrets in Git behind VPN in the buried DataCenter and face real issues with multiple copies, unencrypted drives, secrets rotations. Others would bake into Kubernetes clusters and face issues when cluster fails and they lose all secret data.
On the other hand, there exists a possibility to integrate Kubernetes with Hashicorp Vault service, which implements secrets storage. Kubernetes cluster then points to this value to retrieve secret data.
Our goal was to have the ability to deploy software on Kubernetes clusters without Vault service, which for security reasons should not be a part of any Kubernetes cluster that uses Vaulted data. We want to be able to work with values and secrets on local machines and run dry-runs without any Vault infrastructure connectivity.
How do we do it?
At Base we’ve been using Ansible almost from the beginning. We decided to implement helm for Kubernetes deployments, which probably deserves a separate blog post. When faced with secrets management, we came up with an internal version of the helm-secrets plugin that was in many ways very similar to the Ansible vault. We use PGP and we encrypt the whole secrets file.
It was a simple solution, though from the beginning we understood that it wouldn’t be enough. Soon after we decided to move to managing each YAML secret value separately.
Before any coding, we did list of requirements and research for tools that meet our expectations. We identified Mozilla SOPS as desired secrets backend for our plugin due to list of supported features.
The only thing that we had to handle was to write a simple layer using the first version of helm-secrets that would integrate both with SOPS on backend and Helm on frontend reusing helm plugin functionality.
And this is exactly what we did, and then we decided to share this work publicly. Here it comes, a release of helm-secrets plugin – https://github.com/futuresimple/helm-secrets
The feature list is even longer than described below:
- A simple layer of integration and installation with helm plugin
- Support for Helm YAML structures encryption
- Encryption per value with visual Diff works even on encrypted files
- On the fly decryption for git diff
- On the fly decryption and cleanup for helm install/upgrade/rollback with helm-wrapper command
- Multiple key management solutions like PGP and AWS KMS at the same time in the same secrets file
- Simple adding/removing keys to an encrypted file
- With AWS KMS permissions management for keys without any secrets re-encryption
- Secrets files directory tree separation with recursive .sops.yaml files search
- Extracting sub-elements from an encrypted file structure
- Encrypt only part of a file if needed.
It works with our CI/CD and with multiple teams supporting isolated permissions to specific sub-dirs in Git repository, which contains per project/environment/region secrets for specific Kubernetes clusters. We encrypt secrets using specific KMS keys and master PGP as a fallback.
We use Makefile in our internal charts repository to automate tasks and minimize operational time.
Now some fresh meat of how it works :)
Follow installation instructions on the helm project page
Install helm secrets plugin
helm plugin install https://github.com/futuresimple/helm-secrets
On the helm-secrets repository we can see an example how we can use this plugin with helm charts values directory structure. We use a similar structure with our kubernetes charts.
This example is covered by test.sh in helm-secrets git repository.
example/helm_vars/ ├── .sops.yaml ├── projectX │ ├── .sops.yaml │ ├── production │ │ └── us-east-1 │ │ └── java-app │ │ ├── secrets.yaml │ │ └── value.yaml │ └── sandbox │ └── us-east-1 │ └── java-app │ ├── secrets.yaml │ └── value.yaml ├── projectY │ ├── .sops.yaml │ ├── production │ │ └── us-east-1 │ │ └── java-app │ │ ├── secrets.yaml │ │ └── value.yaml │ └── sandbox │ └── us-east-1 │ └── java-app │ ├── secrets.yaml │ └── value.yaml ├── secrets.yaml └── values.yaml
Let’s follow examples from helm-secrets repository.
- We have two PGP keys (we can also use KMS keys)
– one for projectx
– one for projecty
- All other secrets are managed inside the project space and only one key per project is used
- Keys are isolated per project and only global secrets.yaml file on helm_vars root level can be decrypted by any of these keys
- All recursive depth rules for encrypt / decrypt exist in .sops.yaml files
Encrypt, Decrypt and more
Before encryption example/helm_vars/secrets.yaml
helm-wrapper secrets enc example/helm_vars/secrets.yaml
As a result of the encrypting process, we will get encrypted secrets file with plain keys and encrypted values for those keys.
After the keys section, we have all sops data needed to encrypt or decrypt our secrets structures.
We do have some metadata like sops version, lastmodified or unencrypted_suffix. More about sops file format you can find on Mozilla SOPS project page.
global_secret: ENC[AES256_GCM,data:pTyPdC6YA+z84Q==,iv:aF5hb9CS8Au0B3RWADPtP8fXYzYakU7JJ8ZxzJgHRF0=,tag:c3pCyOf0NpQU7VPL/72XPg==,type:str] sops: .… …. …. unencrypted_suffix: _unencrypted version: 2.0.9
Now decrypt file
helm-wrapper secrets dec example/helm_vars/secrets.yaml
As you can see we can easily work with such files even without decrypting – for example when we do searching for some keys in multiple secret files. We can even decrypt on-the-fly with git diff config for more comfortable work which is unsupported in other solutions.
We can also manage permissions to secrets more flexibly, without re-encrypting all secrets thanks to more advance AWS KMS.
Moreover, viewing and editing secrets using simple commands from the plugin is very helpful in everyday work.
helm-wrapper secrets view example/helm_vars/secrets.yaml
Will output secret file after decryption to stdout
helm-wrapper secrets edit example/helm_vars/secrets.yaml
Edit will open the decrypted file in editor and then after save will encrypt changes as an edited secret file.
All this features help us work on encrypted data without a big effort in almost any scenario.
Helm love on real world infrastructure
Now let’s look how we can deploy something onto Kubernetes.
To make this process transparent and simple we create helm-wrapper. This simple bash wrapper on helm automatically decrypts all defined secrets and uses decrypted data to deploy using helm. If any error occurs or helm successfully deployed chart then all temporary secrets data are cleaned.
Real world example for helloworld application:
AWS_PROFILE=production helm-secrets upgrade --install --timeout 600 --wait helloworld stable/java-app --kube-context=production --namespace=projectx --set global.app_version=bff8fc4 -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/values.yaml -f helm_vars/secrets.yaml -f helm_vars/values.yaml >>>>>> Decrypt Decrypting helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml >>>>>> Decrypt Decrypting helm_vars/secrets.yaml Release "helloworld" has been upgraded. Happy Helming! LAST DEPLOYED: Fri May 5 13:27:01 2017 NAMESPACE: projectx STATUS: DEPLOYED RESOURCES: ==> extensions/v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE helloworld 3 3 3 2 1h ==> v1/Secret NAME TYPE DATA AGE helloworld Opaque 10 1h ==> v1/ConfigMap NAME DATA AGE helloworld 2 1h ==> v1/Service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE helloworld 100.65.221.245 8080/TCP 1h NOTES: Deploy success helloworld-bff8fc4 in namespace projectx >>>>>> Cleanup helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml.dec helm_vars/secrets.yaml.dec
As you can see there is a command, which is used on CI side to deploy the app. Example using our internal universal java app chart, which contains a template for config map, secret, service and deployment.
All this is generated from values specified and evaluated in order using -f option in helm and if there is secret then wrapper decrypts this secrets from specified path on-the-fly and use decrypted as standard values in helm below.
We are using KMS and we need to specify our AWS_PROFILE name to use AWS KMS to decrypt secret files.
If we run with an error then cleanup is always made at the end.
AWS_PROFILE=production helm-wrapper upgrade --install --timeout 600 --wait helloworld stable/java-app --kube-context=wrongcontext --namespace=projectx --set global.app_version=bff8fc4 -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/values.yaml -f helm_vars/secrets.yaml -f helm_vars/values.yaml >>>>>> Decrypt Decrypting helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml >>>>>> Decrypt Decrypting helm_vars/secrets.yaml Error: could not get kubernetes config for context 'wrongcontext': context "wrongcontext" does not exist >>>>>> Cleanup helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml.dec helm_vars/secrets.yaml.dec
All this minimize effort to use secrets in helm with maximum security.
Secure use with GIT
If you leave some decrypted secrets (manually decrypted) then we have two more levels of security by excluding decrypted files in .gitignore and adding a hook for checking if any committed file is encrypted by SOPS.
All of this is included in helm-secrets docs and used by us with our CI/CD process.
Helm-secrets is only a thin wrapper to sops on backend and we can replace each or all commands of this plugin with any other tool in future.
We can also use this plugin to distribute secrets in public git and in the future extend secrets workflow with Kubernetes integrated Vaulting service on the infrastructure side, which will complete this solution.
If you have a smaller infrastructure, you can easily use only helm-secrets plugin and this should be just fine. Hope you’ll enjoy it!
If you feel that this project needs some features please contribute or we can talk on https://github.com/futuresimple/helm-secrets