How to Use Cert Manager on Kubernetes

, , ,

I’ve got a few readers/viewers asking me to make some content about using Traefik or Kong as Ingress controller. There is no doubt that Kubernetes is one of the hottest topic in the past few years. Normally the more powerful a tool is, the harder it is to be used. I am always reluctant to write a post that is either tool long or difficult to follow. However, deploying applications with helm on Kubernetes is actually quite easy. In today’s post, I would like to show you how to use cert manager to fulfill your TLS needs on Kubernetes.

Cert manager is a very straight forward tool. Users only need two steps to get certificate.

  1. Define an issuer for issuing certificates.
  2. Request certificate from this issuer.

Moreover because certificates are stored in secrets, you can use it with most if not all Kubernetes applications.

Let’s start:

Prerequisites:

A running cluster, Kubectl and Helm.

Installation

We use below command to install cert manager, it creates namespace cert-manager, install CRDs and set nameservers to 8.8.8.8:53\,1.1.1.1:53 for DNS01 validation.

In case you don’t know, 8.8.8.8 is Google’s DNS server and 1.1.1.1 is Cloudflare’s. Both DNS servers are arguably the fastest right now.

1
2
3
4
5
6
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true \
  --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'

Let’s wait for all deployment complete.

1
kubectl get deploy -o name -n cert-manager | xargs -n1 -t kubectl rollout status -n cert-manager

Define Issuer

cert-manager supports 6 issuer types. I will cover 3 types that I have used before.

Self-signed

This issuer as its name suggested issues certificate for itself. This is the quickly way to get a TLS certificate and start encrypting data transmission between your applications. You can define self-signed issuer as below and then request your certificate with whatever information you need, the certificate will be generated immediately.

1
2
3
4
5
6
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

Of courses there are drawbacks of using self-signed certificates mainly because it is issued on the fly so it won’t be trusted if there are any checks in place. To find out more information and how to overcome some of issues, please check official doc.

CA

We often see organisations having their internal PKI set up and all their devices will trust certificates issued by internal CA automatically. In order to get certificate issued by internal CA (sometimes it is a requirement), users can create a CA issuer and all certificates issued by this issuer will be under company’s internal root.

To create a CA issuer, we need to have CA key pair.

  • Create Secret with ca key pair.

    Because I plan to create a cluster issuer, I am creating this secret in cert-manager namespace.

    1
    kubectl create secret tls -n cert-manager ca-key-pair --cert=ca.cert.pem --key=ca.key.pem

  • Create ClusterIssuer

    You can see this issuer is using the key pair we just created as CA.

    1
    2
    3
    4
    5
    6
    7
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: fomm-k8s-ca-issuer
    spec:
      ca:
        secretName: ca-key-pair

Similar to self-signed issuer, all certificate requests to CA issuer will be fulfilled automatically. Unlike self-signed, these certificate will be issued under a certain root which means they will be valid if root is trusted on your devices.

ACME

This is the most important issuer type which use ACME protocol to request valid SSL certificates from CAs. This issuer consist with two parts. The first part is the general info of the ACME server and the second part is what validation method you want to use with this issuer.

General Info

You need to define your issuer information under spec.acme.

Here is an example:

1
2
3
4
5
6
7
8
9
10
11
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: [email protected]
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: le-staging-issuer-account-key
...
  • email: Please use genuine email here because some CA won’t allow bogus email and they would contact you when your cert is about to expire.
  • server: ACME directory URL. For a list of free CA ACME directory URLs, please check my previous post.
  • privateKeySecretRef: This is the secret name that stores account’s private key. This key is used to register an account as well as signing all messages sent to the ACME server.
  • (Optional) disableAccountKeyGeneration: If you have an existing account keys you want to reuse, please set this to true.
  • (Optional) External Account Binding: Most commercial CAs tend to use external account binding for their ACME servers. If the CA you are using (Sectigo, ZeroSSL, Digicert etc.) requires it, please follow official doc to add it.

HTTP01 validation

As certificate manager use ACME protocol, unsurprisingly it supports HTTP01 validation method. This validation method requires port 80 to be opened on the server (or your load balancer) as per RFC8555. If you can’t use this port, you can’t use this method. When you request a certificate with http01 method, cert manager creates an ingress object and complete the validation on its own.

The minimum requirement for using this method is to tell cert manager what ingress class should be listed on the ingress object it creates.

If you use kong ingress controller, you just need to set your solver as below

1
2
3
4
5
...
    solvers:
    - http01:
        ingress:
          class: kong

class specify what to be added to kubernetes.io/ingress.class: annotation on the ingress object. If for some reason you use different class for your Kong Ingress Controller, you can change the class name here.

As per official doc, cert manager team keep using the annotation in order to be more compatible with as many ingress controllers as possible.

You can extend this validation with different options. For example, if you need to add extra annotation on the ingress object created by cert manager, you can use ingressTemplate. For more information, please check official doc here.

When you request a certificate with http01 validation method, it creates an ingress object. As soon as the validation completed (which means this host name and path is reachable), certificate will be issued and stored in the secret name we defined on our certificate or ingress object.

To know more settings about HTTP01 validation, please check official doc.

DNS01 validation

DNS validation is the method I always use.

There are two main reasons to use this method:

  1. You don’t need to worry about firewall, Loadbalancer, proxies etc which could potentially block http validation.
  2. You can request wildcard certificate with DNS01 validation.

and two reasons NOT to use this method:

  1. Not all DNS providers are supported by default. You can find the list of supported providers here. You can probably use acme-dns if your DNS provider is not on the list though.
  2. cert manager requires credentials to be stored on the cluster to allow it creating DNS records for validation purpose.

Let me use Cloudflare as an example.

  • First you need to get an API token from Cloudflare. If you are not sure how to do ti, please follow this Cloudflare doc. Once we’ve got our token, we need to create secret to store it. As I am creating a ClusterIssuer, I need to put this token in cert-manger namespace.

    1
    kubectl create secret generic cloudflare-api-token -n cert-manager --from-literal=api-token=<CLOUDFLARE_API_TOKEN>

  • Reference API token on DNS01 resolver.

    1
    2
    3
    4
    5
    6
    7
    ...
        solvers:
        - dns01:
            cloudflare:
              apiTokenSecretRef:
                name: cloudflare-api-token
                key: api-token

I hope you will know what type of issuer you will use and how to define an issuer after reading above. We will start requesting certificates in the next part.

Request Certificate

The official doc does a great job of explaining the life cycle of certificate.

In simple words, once a Certificate object is created, it creates a Certificate Request and tells cert manager to use associated Issuer to issue certificate for the requested domain. Issuer will then create an Order and Challenge(s) to fulfill the request. Order will manage the life cycle of the ACME order for this certificate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
                      ┌──────────┐                         ┌─────────────┐
                      │          │                         │             │
       ┌──────────────►  Issuer  ├────────────┐        ┌───►  Challenge  │
       │              │          │            │        │   │             │
       │              └──────────┘            │        │   └─────────────┘
       │                                      │        │
┌──────┴────────┐                        ┌────▼────┐   │   ┌─────────────┐
│               │                        │         │   │   │             │
│  Certificate  │                        │  Order  ◄───┼───►  Challenge  │
│               │                        │         │   │   │             │
└──────▲────────┘                        └────┬────┘   │   └─────────────┘
       │                                      │        │
       │        ┌───────────────────────┐     │        │   ┌─────────────┐
       │        │                       │     │        │   │             │
       └───────►│   CertificateRequest  ◄─────┘        └───►  Challenge  │
                │                       │                  │             │
                └───────────────────────┘                  └─────────────┘                ┼

Annotation Ingress

There are may ways to request a certificate via cert manager. The most commonly used method is to annotate an ingress object and let cert manager does the rest for you.

If you don’t want to specify what kind of private key you want to use for you certificate, the only annotation you need is cert-manager.io/cluster-issuer or cert-manager.io/issuer with the issuer name. To know more about securing ingress object, please check official doc here.

(Optional) Use ECC Cert and mandate key rotate

I encourage you to use ECC private key and mandate key rotate for new certificates. You can do it easily by adding 3 more annotations to your ingress object.

1
2
3
kubectl annotate ingress httpbin-route cert-manager.io/private-key-algorithm=ECDSA
kubectl annotate ingress httpbin-route cert-manager.io/private-key-size=256
kubectl annotate ingress httpbin-route cert-manager.io/private-key-rotation-policy=Always

You can use below command to annotate your ingress object.

1
kubectl annotate ingress httpbin-route cert-manager.io/cluster-issuer=<cluster_issuer_name>

Here is an ingress object example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: le-http-cluster-issuer
    konghq.com/strip-path: "true"
  generation: 1
  name: httpbin-route
spec:
  ingressClassName: kong
  rules:
  - host: kong.demofor.fun
    http:
      paths:
      - backend:
          service:
            name: httpbin-svc
            port:
              number: 80
        path: /test
        pathType: Prefix
  tls:
  - hosts:
    - kong.demofor.fun
    secretName: test-http-cert

Once this ingress object is created, cert manager will create a Certificate object and request this certificate using a ClusterIssuer le-http-cluster-issuer automatically. Once the cert is issued, it will be stored in secret test-http-cert.

Certificate Object

Another method to create certificate is to create a Certificate object manually. This is the method I personally use because I need to make sure the certificate exists before using it. There are just too many reasons a certificate request can fail.

You can find an example of certificate object below and I’ve put my comments between lines. You can find the list of certificate specs from here. I only put down whatever I think is useful for my use case.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-dns-cert
spec:
  secretName: test-dns-cert
  commonName: demofor.fun
  # You can also use RSA keys, I prefer ECC certs.
  privateKey:
    # It is better to use new key for new certificate.
    rotationPolicy: Always
    algorithm: ECDSA
    encoding: PKCS8
    size: 256
  usages:
    - server auth
    - client auth
  # I can add wildcard domain here because I use an issuer using dns01 method.
  dnsNames:
  - "*.demofor.fun"
  - "demofor.fun"
  issuerRef:
    # Issuer name wiut
    name: letsencrypt-dns-staging
    # Type of the issuer
    kind: ClusterIssuer

If you want to know more about what options you can use on Certificate resource, please check official doc here.

Once the certificate is issued, it would be stored in secret test-dns-cert. You can then create your ingress object to use it.

1
kubectl create ingress httpbin-route --class kong --rule 'kong.demofor.fun/test*=httpbin-svc:80,tls=test-dns-cert'

That’s all I want to cover today, see you next time.