How to Use Cert Manager on Kubernetes

I’ve got a few readers/viewers asking me to make some content about using Traefik or Kong as Ingress controller. There is no doubt that Kubernetes is one of the hottest topic in the past few years. Normally the more powerful a tool is, the harder it is to be used. I am always reluctant to write a post that is either tool long or difficult to follow. However, deploying applications with helm on Kubernetes is actually quite easy. In today’s post, I would like to show you how to use cert manager to fulfil your TLS needs on Kubernetes.

Cert manager is a very straight forward tool. Users only need two steps to get certificate.

  1. Define an issuer for issuing certificates.
  2. Request certificate from this issuer.

Moreover because certificates are stored in secrets, you can use it with most if not all Kubernetes applications.

Let’s start:

Prerequisites:

A running cluster(I am using Kind), Kubectl and Helm.

(Optional) : Loadbalancer, I am using MetalLB.

Installation

We use below command to install cert manager, it creates namespace cert-manager, install CRDs and set nameservers to 8.8.8.8:53\,1.1.1.1:53 for DNS01 validation.

In case you don’t know, 8.8.8.8 is Google’s DNS server and 1.1.1.1 is Cloudflare’s. Both DNS servers are arguably the fastest right now.

1
2
3
4
5
6
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
--set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'

Let’s wait for all deployment complete.

1
kubectl get deploy -o name -n cert-manager | xargs -n1 -t kubectl rollout status -n cert-manager

Define Issuer

cert-manager supports 6 issuer types. I will cover 3 types that I have used before.

Self-signed

This issuer as its name suggested issues certificate for itself. This is the quickly way to get a TLS certificate and start encrypting data transmission between your applications. You can define self-signed issuer as below and then request your certificate with whatever information you need, the certificate will be generated immediately.

1
2
3
4
5
6
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-cluster-issuer
spec:
selfSigned: {}

Of courses there are drawbacks of using self-signed certificates mainly because it is issued on the fly so it won’t be trusted if there are any checks in place. To find out more information and how to overcome some of issues, please check official doc.

CA

We often see organisations having their internal PKI set up and all their devices will trust certificates issued by internal CA automatically. In order to get certificate issued by internal CA (sometimes it is a requirement), users can create a CA issuer and all certificates issued by this issuer will be under company’s internal root.

To create a CA issuer, we need to have CA key pair.

  • Create Secret with ca key pair.

    Because I plan to create a cluster issuer, I am creating this secret in cert-manager namespace.

    1
    kubectl create secret tls -n cert-manager ca-key-pair --cert=ca.cert.pem --key=ca.key.pem
  • Create ClusterIssuer

    You can see this issuer is using the key pair we just created as CA.

    1
    2
    3
    4
    5
    6
    7
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
    name: fomm-k8s-ca-issuer
    spec:
    ca:
    secretName: ca-key-pair

Similar to self-signed issuer, all certificate requests to CA issuer will be fulfilled automatically. Unlike self-signed, these certificate will be issued under a certain root which means they will be valid if root is trusted on your devices.

ACME

This is the most important issuer type which use ACME protocol to request valid SSL certificates from CAs. Most users will only use this issuer type. I will divide the configuration into three parts.

General Info

Users define basic information of this issuer.

  • email: Please use genuine email here because CA won’t allow bogus email and they will use this email to contact you about expiring certificate.
  • server: ACME directory URL.
  • privateKeySecretRef: This is the secret name that stores account’s private key.
  • (Optional) disableAccountKeyGeneration: If you have your own account keys you want to reuse, please set this to true.
  • (Optional) External Account Binding: Most commercial CAs tend to use external account binding with their ACME servers. If the CA you are using (Sectigo, ZeroSSL, Digicert etc.) requires it, please follow official doc to add it here

For a list of free CA ACME directory URLs, please check my previous post.

1
2
3
4
5
6
7
8
9
10
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: admin@aufomm.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: le-staging-issuer-account-key

HTTP01 validation

This validation requires port 80 to be opened on the server as per RFC8555. If you can’t have this port opened, you can’t use this method. When user request a certificate from issuer using http01 validation method, an ingress object will be created automatically. The setting below are related to what’s on the ingress object.

  • class: This adds kubernetes.io/ingress.class: annotation on the ingress object. Some ingress controllers only need this annotation to be able to detect and satisfy the ingress object.
  • (Optional) ingressTemplate: If you need to add more annotations or labels to the ingress object, you can add it as below.
1
2
3
4
5
6
7
8
9
10
11
12
solvers:
- http01:
ingress:
class: nginx
ingressTemplate:
metadata:
labels:
foo: "bar"
annotations:
"nginx.ingress.kubernetes.io/whitelist-source-range": "0.0.0.0/0,::/0"
"nginx.org/mergeable-ingress-type": "minion"
"traefik.ingress.kubernetes.io/frontend-entry-points": "http"

The ingress object generated from above config looks like below (I was created on an ingress object):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
nginx.org/mergeable-ingress-type: minion
traefik.ingress.kubernetes.io/frontend-entry-points: http
creationTimestamp: "2021-07-24T02:03:29Z"
generateName: cm-acme-http-solver-
generation: 1
labels:
acme.cert-manager.io/http-domain: "652281115"
acme.cert-manager.io/http-token: "1238044331"
acme.cert-manager.io/http01-solver: "true"
foo: bar
name: cm-acme-http-solver-8tb6f
namespace: default
ownerReferences:
- apiVersion: acme.cert-manager.io/v1
blockOwnerDeletion: true
controller: true
kind: Challenge
name: cm-demo-cert-ingress-secret-nppzv-3157103164-4091730226
uid: 774031ed-ec8e-4fec-bb8d-f062c42da97f
resourceVersion: "5843"
uid: 98b9c607-bca9-4ca8-ac85-ce83f952aca0
spec:
rules:
- host: echo.liy.id.au
http:
paths:
- backend:
service:
name: cm-acme-http-solver-nx2qw
port:
number: 8089
path: /.well-known/acme-challenge/OdLznimY3-tC-iPIKva_stZfviL1DTkfa2NbUK-CMhY
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- ip: 172.18.18.150

As soon as the validation completed (which means this host name and path is reachable), certificate will be issued and stored in the secret name we defined on our certificate or ingress object.

To know more settings about HTTP01 validation, please check official doc.

DNS01 validation

DNS validation is the method I alway use. In my opinion there are two main reasons to use this method:

  1. Users don’t need to worry about firewall, Loadbalancer, proxies etc which could potentially block http validation.

  2. Users can request wildcard certificate with DNS01 validation.

However, users need privilege to access domain DNS and not all DNS providers are supported at the moment.

Let me use Cloudflare as an example.

  • First you need to get an API token from Cloudflare. If you aren to sure, please follow this Cloudflare doc. Once we’ve got our token, we need to create secret to store it. As I am creating a ClusterIssuer, I need to put this token in cert-manger namespace.

    1
    kubectl create secret generic cloudflare-api-token-secret -n cert-manager --from-literal=api-token=<CLOUDFLARE_API_TOKEN>
  • Reference api token on dns01 resolver.

    1
    2
    3
    4
    5
    6
    7
    solvers:
    - dns01:
    cloudflare:
    email: <CLOUDFLARE_ACCOUNT_EMAIL>
    apiTokenSecretRef:
    name: cloudflare-api-token-secret
    key: api-token

ACME issuer example

Combine the general info section with DNS section, we’ve got our issuer as below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: your-email@example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: le-staging-issuer-account-key
solvers:
- dns01:
cloudflare:
email: <CLOUDFLARE_ACCOUNT_EMAIL>
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token

I hope you will know what type of issuer you will use and how to define an issuer after reading above. We will start requesting certificates in the next part.

Request Certificate

There are two main ways of requesting a certificate. The first method is to create a certificate object which gets the certificate and stores them in a secret first and then users can use this secret on their application or other Kubernetes resources ingress for example.

The other method is to create ingress objects with cert-manager annotations. Certificate object will be created automatically based on the info on the ingress. Let me show you how these two methods work.

Certificate Object

Personally this is the method I will go for because I am skeptical and need to make sure the certificate exists before using it. There are just too many reasons a certificate request can fail.

You can find an example of certificate object below and I’ve put my comments between lines. You can find the list of certificate specs from here. I only put down whatever I think is useful for my use case.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
# Certificate object name
name: cert-manager-demo-cert
spec:
secretName: cm-demo-cert-secret # Secrete Name to store TLS cert
duration: 2160h # 90d
renewBefore: 360h # 15d
# Although Common Name has been deprecated for a very long time, I still see softwares using common name so you can decide whether or not to keep it.
commonName: li.test
# Subject field might be useful when you are creating your own CA chain. Here is how to write it.
subject:
organizations:
- "foMM Ltd"
countries:
- "AU"
localities:
- "Melbourne"
provinces:
- "Victoria"
# We are requesting end user certificate, so it is not a CA.
isCA: false
# Because everything is automatic, it is better to use new key for new certificate.
# You can also use RSA keys, I only use ECC certs.
privateKey:
rotationPolicy: Always
algorithm: ECDSA
encoding: PKCS8
size: 256
# privateKey:
# rotationPolicy: Always
# algorithm: RSA
# encoding: PKCS8
# size: 2048
usages:
- server auth
- client auth
# At least one of a DNS Name, URI, or IP address is required. Some ACME providers require commonName to be added to dnsNames.
dnsNames:
- "*.li.test"
- "li.test"
issuerRef:
name: my-ca-issuer # ClusterIssuer we are using
kind: ClusterIssuer # Type of the issuer

For users that only use Trusted CAs (mostly Let's Encrypt) to issue their certificate, you don’t need to stress too much about certificate settings. Only CA can decide what’s on the certificate they issue anyway. You can use below to get a RSA 2048 three month validity certificate that secures li.test. This certificate will be renewed 30 days before expiry date automatically.

1
2
3
4
5
6
7
8
9
10
11
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cert-manager-demo-cert
spec:
secretName: cm-demo-cert-secret
dnsNames:
- "li.test"
issuerRef:
name: my-ca-issuer
kind: ClusterIssuer

If you want to know more about how certificate object request certificate from CA servers, please check official doc, it explains very well.

Annotation on Ingress

This really is just a standard ingress resource. The only thing I added is a cert-manager.io/cluster-issuer annotation which tells cert manager to create a certificate for hosts name echo.liy.id.au and store certificate in secret called cm-demo-cert-ingress-secret. Once this ingress object is created, cert-manager generates the certificate objects and request certificate from lets encrypt automatically. The traffic goes to your route will start using valid certificate once the validation is completed.

If you want to know more about cert-manager annotations, please check official doc.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo-route
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
tls:
- hosts:
- echo.liy.id.au
secretName: cm-demo-cert-ingress-secret
rules:
- host: echo.liy.id.au
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: echo-svc
port:
number: 80

That’s all I want to cover today, I hope it is useful to you.