How to Use Kong mTLS Plugin

In my previous post, I talked about how to use Vault and JWT Authentication plugins. In today’s post, I want to talk about Mutual TLS Plugin.

Prerequisite :

A Kong Enterprise instance running with database.

cURL and jq

CA and Client certificate: If you don’t have existing client certificates, you can use either mkcert or follow this tutorial to set up your private CA.

You can find official plugin documentation here.

Use Example

I will set up everything via Admin API.

Create a service

1
2
3
4
curl -X POST http://localhost:8001/services \
-H "Content-Type: application/json" \
-H "Accept: application/json, */*" \
-d '{"name":"mtls-service","url":"https://httpbin.org/anything"}'

Create Route

1
2
3
4
curl -X POST http://localhost:8001/services/mtls-service/routes \
-H "Content-Type: application/json" \
-H "Accept: application/json, */*" \
-d '{"name":"mtls-route","paths":["/mtls"]}'

When we visit our route at curl 'http://localhost:8000/mtls' -i, we should get HTTP/1.1 200 OK.

Create ca-certificate object

We need to create ca certificate objects for root and intermediate certificates.

1
curl http://localhost:8001/ca_certificates -F cert=@root.crt
1
curl http://localhost:8001/ca_certificates -F cert=@intermediate.crt

You should get response as below respectively

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"cert": "<PEM_CERT>",
"id": "c383d81a-bffc-4e2a-b0d3-ac56b441a07b",
"cert_digest": "3e9099b5015e8f486c00bcea9d111ee721faba355a89bcf1df69561e3dc6325c",
"tags": null,
"created_at": 1607347576
}

{
"cert": "<PEM_CERT>",
"id": "8796de73-3673-4c4c-bd23-2afbb00f10be",
"cert_digest": "a542bca09c5e4579c619774ae59082bce0f86d261c5a7a5a0f6217c10279ea7c",
"tags": null,
"created_at": 1607347585
}

Enable mTLS plugin

As usual, I enabled plugin globally.

1
2
3
4
5
6
curl --request POST \
--url http://localhost:8001/plugins \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data name=mtls-auth \
--data config.ca_certificates=8796de73-3673-4c4c-bd23-2afbb00f10be \
--data config.ca_certificates=c383d81a-bffc-4e2a-b0d3-ac56b441a07b

If we visit our route again, we will get HTTP/1.1 401 Unauthorized

Access Control

There are two methods of doing access control with mTLS plugin.

The first method is to create a consumer object and map its username or custom_id to CN/Email of your client certificate. When you do that you need to create consumer objects for differnt users.

If you don’t want to use consumer object, you can also use ACL plugin to create groups and then map client certifiate’s CN to this group name.

Use Consumer Object

Create a consumer with username or custom_id the same as the CN/Email on client certificates.

1
2
3
4
curl --request POST \
--url http://localhost:8001/consumers \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data username=<CLIENT_CERT_CN_NAME>

Use ACL plugin

  • Get mTLS plugin id

    1
    export MTLS_PLUGIN_ID=$(curl -s http://localhost:8001/plugins | jq -r '.data[] | select (.name="mtls-auth") | .id')
  • Patch mTLS plugin

    1
    2
    3
    4
    curl --request PATCH \
    --url http://localhost:8001/plugins/$MTLS_PLUGIN_ID \
    --header 'Content-Type: application/x-www-form-urlencoded' \
    --data config.skip_consumer_lookup=true
  • Enable ACL plugin

    1
    2
    3
    4
    5
    curl --request POST \
    --url http://localhost:8001/plugins \
    --header 'Content-Type: application/x-www-form-urlencoded' \
    --data name=acl \
    --data config.allow=<CLIENT_CERT_CN_NAME>

Now when we provide a client certificate that does not have CN matching group name, we will get HTTP/2.0 403 Forbidden and below response.

1
2
3
{
"message": "You cannot consume this service"
}

Access API

mTLS means mutual TLS authentication so we need to access our route via https protocol. If your tls certificate is self-signed, you need to add -k flag to skip certificate checks.

1
curl https://<PROXY_URL>:8443/mtls --key client.key --cert client.crt -k
  • If you use consumer object, you should see X-Consumer-Id and X-Consumer-Username added to the upstream request.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    {
    "args": {},
    "data": "",
    "files": {},
    "form": {},
    "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.78.0",
    "X-Amzn-Trace-Id": "Root=1-60ffa1b7-6763273d3e489c0c14b6e43f",
    "X-Consumer-Id": "8715a4f8-e1bd-44f1-82d2-22724127c47c",
    "X-Consumer-Username": "<CLIENT_CERT_CN_NAME>",
    "X-Forwarded-Host": "kong.li.lan",
    "X-Forwarded-Path": "/mtls",
    "X-Forwarded-Prefix": "/mtls"
    },
    "json": null,
    "method": "GET",
    "origin": "192.168.186.1",
    "url": "https://kong.li.lan/anything"
    }
  • If you use ACL plugin to limit access, you should see X-Client-Cert-Dn, X-Client-Cert-San and X-Authenticated-Groups added.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    {
    "args": {},
    "data": "",
    "files": {},
    "form": {},
    "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Host": "httpbin.org",
    "User-Agent": "xh/0.14.1",
    "X-Amzn-Trace-Id": "Root=1-61b5f805-05b9745a588f96e0190cd347",
    "X-Authenticated-Groups": "<CLIENT_CERT_CN_NAME>",
    "X-Client-Cert-Dn": "<DN_OF_CLIENT_CERT>",
    "X-Client-Cert-San": "<CLIENT_CERT_CN_NAME>",
    "X-Forwarded-Host": "kong.li.lan",
    "X-Forwarded-Path": "/mtls",
    "X-Forwarded-Prefix": "/mtls"
    },
    "json": null,
    "method": "GET",
    "origin": "192.168.0.1",
    "url": "http://kong.li.lan/anything"
    }

Other deployments methods

DBless deployment

Please save below content to kong.yaml and load it to your DBless deployment configuration. Please updat <CLIENT_CERT_CN_NAME> to CN or email of your client certificate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
_format_version: "2.1"
_transform: true

services:
- name: mtls-service
url: https://httpbin.org/anything
routes:
- name: mtls-route
paths:
- /mtls

consumers:
- username: <CLIENT_CERT_CN_NAME>

plugins:
- name: mtls-auth
config:
ca_certificates:
- 7ccf0aed-0181-4c51-99fa-5ccc831cee8b
route: mtls-route

ca_certificates:
- id: 7ccf0aed-0181-4c51-99fa-5ccc831cee8b
cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

After successfully deploying your Kong instance, you should be able to access your api.

Kubernetes Ingress Controller

Please change <CLIENT_CERT_CN_NAME> of your consumer object and put your CA root crtificate in x509 format.

Below example will deploy :

  • Echo deployment
  • Echo Service
  • Mutual TLS plugin
  • Consumer mtls-user
  • Ingress rule to use Mutual TLS Authentication plugin
  • CA certificate object

Please save below to mtls.yaml and use kubectl apply -f mtls.yaml to apply it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo-deployment
spec:
replicas: 1
selector:
matchLabels:
app: echo-pod
template:
metadata:
labels:
app: echo-pod
spec:
containers:
- name: echoheaders
image: k8s.gcr.io/echoserver:1.10
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: echo-service
spec:
selector:
app: echo-pod
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: proxy-to-echo
annotations:
konghq.com/plugins: mtls-auth
konghq.com/strip-path: "true"
spec:
ingressClassName: kong
rules:
- http:
paths:
- backend:
service:
name: echo-service
port:
number: 80
path: /mtls
pathType: Prefix
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: mtls-user
annotations:
kubernetes.io/ingress.class: "kong"
username: <CLIENT_CERT_CN_NAME>
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: mtls-auth
config:
ca_certificates:
- 7ccf0aed-0181-4c51-99fa-5ccc831cee8b
plugin: mtls-auth
---
apiVersion: v1
kind: Secret
metadata:
name: my-ca-cert
annotations:
kubernetes.io/ingress.class: kong
labels:
konghq.com/ca-cert: 'true'
type: Opaque
stringData:
cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
id: 7ccf0aed-0181-4c51-99fa-5ccc831cee8b

Extended information

  • CA Certificate object must include root certificate
  • Multiple CA is supported. You just need to add ca_certificate object id to mTLS plugin under config.ca_certificates.

Consumer matching

This plugin can use CN or DN from client certificate for matching. You can change it by setting config.authenticated_group_by when applying Mutual TLS plugin.

Class 1 S/MIME certificate only has email address and Class 2 S/MIME has both email and Common Name for matching. This means you are not limited to use Email for your consumers when you are using Class 2 S/MIME certificate.

Auto Matching

Users can choose to match username AND/OR custom_id of a consumer against the matching criterial from client certificate. As long as there is 1 match, consumer is authenticated and allowed to use API.

The matching priorities are Email @username > Email @custom_id > Common Name @username > Common Name @custom_id

Users can use config.consumer_by to set what to match.

Manual Matching

If user set config.consumer_by empty [], Kong must use manual mapping, which mean if subject_name is not set on a consumer, authentication fail immediately.

Manual mapping has two fields to use:

  • subject_name
  • ca_certificate.id (optional)

These two fields can only be added via Admin API and are not visible on Kong manager at the moment.

1
2
3
4
curl --request POST \
--url http://localhost:8001/consumers/<CONSUMER_USERNAME>/mtls-auth \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data subject_name=<VALUE_FOR_MATCHING>

User can set subect_name to either an email or common name. Email and subject name has the highest priority, as long as there is a user with subject_name match email, it will match even when consumer_by is set to something else.

mTLS with SELECTIVE upstream

This is not related to to mTLS plugin but I think it is worth mentioning it on this post.

Let’s say you have set up 3 services on Kong and you need Kong to do mTLS with Service 1.

What you need to do is to create client certificate as a certificate object and then reference it on service object.

  • Create Certificate object

    1
    curl http://localhost:8001/certificates -F cert=@cert.pem -F key=@key.pem

    Let’s say we get certificate object id is 9ffed9cd-82a3-4a27-b88a-9816c76bad35

  • Patch service Object

    1
    2
    3
    4
    5
    curl --request PATCH \
    --url http://localhost:8001/services/service_1 \
    --header 'Content-Type: application/x-www-form-urlencoded' \
    --data protocol=https \
    --data client_certificate.id=9ffed9cd-82a3-4a27-b88a-9816c76bad35

(Optional) If you need to build trust stroe to verify upstream server’s tls certificate, you can use ca_certificates on service object as well.

Once this is set, Kong will send the client certificate to all requests related to Service 1.

mTLS with ALL upstreams

Besides using Service Mesh, if you want to add mTLS to all upstream services with a specific client certificate, you can do it easily with two Kong nodes.

The first node will be used as reversed proxy and client to send mTLS certificate to the second node which will be the API gateway to upstream.

What we need to do is to enable client_ssl on the first node. You can find more information about this parameter on official doc here.

Here are the steps (demo in Docker):

  1. Mount client certificate on Reverse Proxy node.

    1
    2
    volumes:
    - /certs/client:/client
  2. Add below environment variables to start Reverse Proxy instance

    1
    2
    3
    KONG_CLIENT_SSL="on"
    KONG_CLIENT_SSL_CERT="/client/client.crt"
    KONG_CLIENT_SSL_CERT_KEY="/client/client.key"
  3. Enable mTLS plugin on API Gateway node, you can enable the plugin globally.

  4. Configure API gateway node as the upstream service on Reverse Proxy.

  5. Set up services on API gateway node to different backend.

  6. Now Traffic between these two Kong nodes are secured with mTLS.

That’s all I want to cover on this post, thank you for reading, see you next time.