How to Use Sops-Secrets-Operator to Secure Kubernetes Secrets

(Updated: )
, , ,

Security is essential, especially as automation and GitOps become the de facto standard for managing everything from infrastructure to application deployment. This is particularly true with the increasing adoption of Kubernetes. The trend toward reducing human intervention and managing everything through code suggests a growing trust in automated processes over human operators.

There’s no denying the benefits of managing resources declaratively in a pipeline, but this approach also heightens security concerns. With Git-based workflows, where access to the repository is often shared, how do we ensure users can’t access sensitive information like Kubernetes secrets?

For Kubernetes clusters on cloud providers like AWS, many tools offer native integrations with the provider’s Secret Manager, meaning secrets aren’t stored directly on the cluster but are accessed only when needed. Traditional methods, like pulling secrets from the cloud to local secrets, are also supported. For this approach, External Secrets is a popular choice.

For those who prefer to avoid vendor lock-in and want encrypted files stored directly in the repository, other options exist. Sealed Secrets is one popular option, though I find its UX not very intuitive. Recently, while exploring Nixidy, I came across SOPS Secrets Operator, which is incredibly easy to use, especially for existing SOPS and Age users.

While both Age and SOPS deserve dedicated posts, I’ll give a brief introduction here to help you get started with them.

Let’s dive in!

Age

If you are not interested in theory, feel free to jump to the how to use section.

What is Age?

Age is a modern file encryption format featuring pluggable recipient types and seekable streaming encryption. It provides a lightweight and user-friendly solution for file encryption, ensuring only the intended recipient can decrypt the file. However, Age doesn’t verify the sender’s identity. Interestingly, this is actually a key advantage of Age over PGP for personal use in my opinion, as it removes the need to maintain a list of recipients’ public keys just to identify the sender. In most cases, users are both the sender and the recipient, so this simplification is highly convenient.

How Does It Work?

Age employs a hybrid encryption model. It generates a symmetric key to encrypt the content and then uses an asymmetric key to encrypt the symmetric key in the header.

For symmetric encryption, Age uses ChaCha20-Poly1305 to ensure both encryption and content integrity. For asymmetric encryption, it uses ECC X25519, providing strong security with a minimal footprint.

If you’re interested in more technical details, you can find them here.

The encrypted file is similar to the JWE token I discussed in this post. However, JWE allows you to verify the sender by using a JWT as payload. This JWT is normally signed with sender’s private key, allowing the recipient to validate the sender’s identity.

That’s enough theory, let’s see how we use age.

How to use

Installation

You should find age in most package managers. If you use nix, you can test it with nix shell nixpkgs#age.

Generate age key

Let’s use the age cli to generate a key.

1
age-keygen -o key.txt

The public key will be in the output. You can also get the public key again with below command.

1
age-keygen -y key.txt

Encrypt file

Let’s create a file call data.txt.

1
echo "hello world" > data.txt

Next, let’s store the public key to environment variable

1
pub_key=$(age-keygen -y key.txt)

To encrypt a file, you can run command below. -e suggest it is to encrypt the file. The intended recipient’s public key needs to be passed in with -r flag. If you want your file to be decrypted by multiple recipients, you just pass additional public key with -r flag.

1
age -r ${pub_key} -e data.txt > data.txt.age

Decrypt file

You need the private key to decrypt the file.

1
age --decrypt -i key.txt -o data-decrypt.txt data.txt.age

We can verify the decrypted file with below command.

1
2
➜ cat data-decrypt.txt
hello world

Sops

What is Sops

SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

You can use sops to encrypt file fields using keys from different providers. These keys can be KMS from cloud provider or local keys like age or PGP.

How to use it with Age

We will reuse the age key we generated in the previous section. Let’s create a kubernetes secret first.

1
2
3
4
5
6
7
8
9
cat > ./secret.yaml << EOF
kind: Secret
apiVersion: v1
metadata:
  name: test-credentials
type: Opaque
stringData:
  password: test
EOF

Encrypt file

Let’s save public key to environment variable again.

1
pub_key=$(age-keygen -y key.txt)

Then we can encrypt the file with below command

1
2
3
4
sops --encrypt \
  -age $pub_key \
  --output secret-enc.yaml \
  secret.yaml

This command outputs the encrypted file to secret-enc.yaml. Let’s have a look its content.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
➜ cat secret-env.yaml
kind: ENC[AES256_GCM,data:PaA1cMA7,iv:OaDnlzV9r9e9GAp6XZed22L90oQq7/01SARaqsHAvEQ=,tag:IZugL96rGt6qRTZ80Ltvpw==,type:str]
apiVersion: ENC[AES256_GCM,data:lRI=,iv:Symayu4Vt/xitPXbAR05qVcbdsDqas27V3EcxNDgi1s=,tag:tDef+oPWrqKAyMj3snPIqQ==,type:str]
metadata:
    name: ENC[AES256_GCM,data:MCsvsZe2/nq+2LFHrfilFA==,iv:/gAYRWq547yPhC99FhZdt931lQZaT6F9PyqRsGU5mHc=,tag:GAw0LcZeZHzqFIfFCoMssQ==,type:str]
type: ENC[AES256_GCM,data:DRvepJ6/,iv:Klu+Bh6Zy3PaK0+CKeTYvhfSHUtwqxm3wbisGdMpdp0=,tag:3r3nsZsMhd1bV39mNmOmTA==,type:str]
stringData: null
password: ENC[AES256_GCM,data:VCLOpg==,iv:YPgjn4sgDE9HGcyIzFQ9yOxqPKF14MdMyOlZv9K2yYs=,tag:eK/Fd5QO0D1C4bNxLnbQSg==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1xed6gdv4fwa93uhwf747dmjf6tvq42cc25f4u3cz4r2jp68hr4fqsqjvf9
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0dFFwUnB6dWdPMTZJcVRi
            SkYvc1dBcTl2YWdHTVRqVVJIeW9rZ3BDckhBCm1VczRTd1ZQVWtNZzZPWk1TYmFa
            cURnRTZLRUpLQkwycUl1Um1VZDVRd2sKLS0tIHVwbHp4M2tRcFdaYjYyaDc3M0Rj
            UGNlMHF3M1huMG5RaEFQU1V3VmFpZEUKdDYAGxa67srGYvVWu63Ur1V/qVxMJw22
            CiVaVCzSsLvr+Z5uXJDATvcu829oIQTB74xTImGg9JKHVEPF/aG4lg==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2024-11-02T08:45:37Z"
    mac: ENC[AES256_GCM,data:lP7avYkqkMeK5rDFxJugHS0T0+8+WssbgB+UFwiyhQSQlylzsQDVhL6t1K+VwcAgZR9FI454Ol+0Bdn6oLSTS7UMXVl+gA+5fg4kMvVLhbY2fK/GpJyG21B21syaO09Of4Pae8X5xxm5JUJkdV8K7BPkhZI7o9X1Gpax6eoorjY=,iv:UuZjLZ5eaC7i9H3rAJ7HRlu0mD1TQKINWl430SOEUVk=,tag:Rmhg6gT1Z8pJUu4ff/GP1g==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.9.1

As we can see everything on this file is encrypted and the recipient is our age public key. If we only want to encrypt the stringData, we can run

1
2
3
4
5
sops --encrypt \
  -age $pub_key \
  --output secret-enc.yaml \
  --encrypted-regex '^(data|stringData)$' \
  secret.yaml

Let’s check again and we should see only the content under stringData are encrypted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
➜ cat secret-enc.yaml
kind: Secret
apiVersion: v1
metadata:
    name: test-credentials
type: Opaque
stringData:
    password: ENC[AES256_GCM,data:y3t2uw==,iv:cRy4x+8223nBBn5oLldClScB57cbQUHqZHzz3LoEGv4=,tag:FeU2WkWXhbojgy+xQ7B1hQ=
=,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1xed6gdv4fwa93uhwf747dmjf6tvq42cc25f4u3cz4r2jp68hr4fqsqjvf9
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuakhjRDJleEpxTmxTVitM
            UEJUcDlZdEtpZkpMT0dYTElINndNa1NYTjNzCmgwKytJZUVYaE5LZ1EzdnYwY3BD
            Snk1RHdCNEQ0WkpYSGtjQll3K3NmaHcKLS0tIHlJY1ZObnMwZlZkWnNrVDlOT2RS
            ZEp2R0FtMWFQajVQand0aFpUaGpoVlEKGSAFHrMd8FZPcOOgb3peYTe02SY7LlUy
            qGZpYx+MjFfQwPMpWVF9dwmpdzevF/NCDyHr1w8Qi4DgMSeFoPjr7A==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2024-11-02T08:50:16Z"
    mac: ENC[AES256_GCM,data:yWBSes0LJykHO6KNGK3ZAGBZ/R6Gb7L954In7b35FkNQbMBXQz/yJUiBPj/23bV0YuB15pANCzo9LkNKLxDGzZyPiHuhRc4t2/pU28oSro1ACgUtjXXFCrFupAAxySi5inS3hDT6YKYD6sw9hF0h2Ki30Pt956tYwNtJ/7Wmukw=,iv:BtzyFhUMwLVwfQKHoU6xR/w4vsdNtDDjZRRb9hxUGHI=,tag:/07SnQH8vDYyJfNGgZjhCw==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.1

.sops.yaml

If you do not want to use recipient public key and encrypted_regex for every single command, you can create a .sops.yaml file in the same directory. The config on this file will be used to encrypt the data.

1
2
3
4
5
cat > ./.sops.yaml << EOF
creation_rules:
  - encrypted_regex: ^(data|stringData)$
    age: $pub_key
EOF

Let’s try again

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
➜ sops -e secret.yaml
kind: Secret
apiVersion: v1
metadata:
    name: test-credentials
type: Opaque
stringData:
    password: ENC[AES256_GCM,data:+UlqXQ==,iv:SWOtiUnppmkC/rYRCcU7Df4qc/III47wgEbUBrEg/eM=,tag:gx7RJ87goFeMV4f2qn84wQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1xed6gdv4fwa93uhwf747dmjf6tvq42cc25f4u3cz4r2jp68hr4fqsqjvf9
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWeW9MbDBVUUZGRVduZXQy
            eFZ1Zm1TSHdrc1RHUGkvVEpLbjZFMnRqL2pZCnBkMGlqNm43TGdzZTVkSk10Um5J
            UittYW15TW9JaXNqRDJyZFdOaVVsSFEKLS0tIFRMRTM5bzREdm5UUmlNVUxkeHV1
            amV0K2JWR0Y1UmJsdEtoNTAydFhqZjgK6vznYcF762T4GhpEssbcojgRKiQVd2RY
            fo5PchJJ7+V19vo2vXP9VCI40ouWwMcGKu6fYRLAUoREz2+tP7sMYQ==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2024-11-02T08:57:59Z"
    mac: ENC[AES256_GCM,data:9PHWGriDZrkzX514/qGduom9vl6T+YD5bLLNW4ABeQ2fxPwgUXNGB7eOg/SbGdwefF+jjUnDHDTavMbrc9m7SqcC5w0JCgOb9MdYrNRbbyKEzMFxrxTS+uys/zNCBwofQgkczya7DlBJBYWQgxPus1buM7EeheHfWS0SWYnnbVY=,iv:cogNmggcdpI3E7E1TWsUuuU1ecM59wpr6g6PQcHnQIw=,tag:XHhM12+THFo+fgd7B9X30Q==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.1

Decrypt file

To decrypt the file, sops looks for the key file at its default location. On Linux, this would be $XDG_CONFIG_HOME/sops/age/keys.txt. If $XDG_CONFIG_HOME is not set $HOME/.config/sops/age/keys.txt is used instead. On macOS, this would be $HOME/Library/Application Support/sops/age/keys.txt. On Windows, this would be %AppData%\sops\age\keys.txt.

We need to pass our age key with SOPS_AGE_KEY_FILE environment variable for decryption.

1
2
3
4
5
6
7
8
SOPS_AGE_KEY_FILE=$PWD/key.txt sops -d secret-enc.yaml
kind: Secret
apiVersion: v1
metadata:
  name: test-credentials
type: Opaque
stringData:
  password: test

sops-secrets-operator

Now that we understand how to encrypt files with SOPS, you can start encrypting your secrets and decrypting them in the pipeline before they are applied. While this approach works, is there a way to automate it? What if you need to create multiple secrets at once, or handle secret rotation? This is where sops-secrets-operator comes in handy.

What is sops-secrets-operator

Using sops-secret-operator, you can define multiple Kubernetes secrets as SopsSecret custom resources. The operator then decrypts these resources and makes them available as standard Kubernetes Secrets. It continuously monitors the custom resources for changes, enabling automatic secret rotation.

How to use

Installation

Create Namespace

1
kubectl create namespace sops-operator

Create secret call age-key that has our age key.

1
kubectl create secret generic -n sops-operator age-key --from-file=./key.txt

Add helm repo

1
2
helm repo add sops https://isindir.github.io/sops-secrets-operator/
helm repo update

Install helm release

1
2
3
4
5
6
7
8
9
cat << EOF | helm upgrade --install sops sops/sops-secrets-operator --namespace sops-operator --values -
secretsAsFiles:
- mountPath: /etc/sops-age-key
  name: sops-age-key
  secretName: age-key
extraEnv:
- name: SOPS_AGE_KEY_FILE
  value: /etc/sops-age-key/key.txt
EOF

Create custom resource

Let’s create a custom resource. This resource means the operator should create two secrets in the default namespace. One is called one-token and the other one is jenkins-secret.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat > sops-secrets.yaml << EOF
apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
  name: test-sopssecret
  namespace: default
spec:
  suspend: false
  secretTemplates:
    - name: some-token
      stringData:
        password: test
    - name: jenkins-secret
      labels:
        "jenkins.io/credentials-type": "usernamePassword"
      annotations:
        "jenkins.io/credentials-description": "credentials from Kubernetes"
      stringData:
        username: myUsername
        password: 'Pa$$word'
EOF

Let’s use sops to encrypt this CR. 

1
sops -e sops-secrets.yaml > sops-secrets-enc.yaml

And then apply it.

1
kubectl apply -f sops-secrets-enc.yaml

Verification

We should see these two secrets created.

1
2
3
4
➜ kubectl get secret
NAME             TYPE     DATA   AGE
jenkins-secret   Opaque   2      6s
some-token       Opaque   1      6s

Checking the secret content, the some-token has the correct password test

1
2
➜ kubectl get secret some-token -o yaml | yq .data.password | base64 -d
test

Summary

Security will always be a top priority for running any application or workflow. The combination of Age and SOPS offers both flexibility and robust security. Of course, key management is an additional consideration. However, how much trust you have in cloud providers and the potential cost savings from using local encryption are worth evaluating.

That’s all for today. See you next time!