How to Use Persistent Private Keys for JWT Signer Plugin in Kong DBless Mode

(Updated: )
, , , , ,

JWT validation has become increasingly important as security standards have improved and become more strictly enforced in the industry. I have written several posts about validating JWT with Kong in the past. One of these posts was about how to use the JWT signer plugin. In that post, I highlighted the benefits of the JWT signer plugin, including:

  • Validating public keys from multiple IDPs
  • Ensuring requests go through Kong by resigning a new token to the backend
  • Introspecting tokens when they are opaque
  • Modifying token claims arbitrarily based on your needs

Despite these benefits, there is a challenge when using this plugin with Kong in DB-less mode. The private key used to sign new tokens is not persistent, meaning Kong generates a new key to sign the new token for each request. As a result, your backend cannot maintain a consistent set of public keys to validate tokens from Kong. This issue arises because all Kong configurations are stored in a single declarative config file when Kong is running in DB-less mode. It is also impossible to rotate the keys since admin api is read-only in this mode.

In today’s post, I will present solutions to address this issue. These solutions depend on the value you use for config.access_token_keyset or config.channel_token_keyset in the JWT signer plugin, which determines where you put your private key for Kong to use.

Let’s get started.

Generate keys

Firstly, we need to generate JWKs. Here is a simple Python script that creates three files in the folder. I will demonstrate how to use these files in the next section.

  • private.json: Contains RS256 and ES256 private keys in JSON format
  • private.yaml: Contains RS256 and ES256 private keys in YAML format
  • public.json: Contains RS256 and ES256 public keys in JSON format

Python library required:

  • joserfc
  • pyyaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
from joserfc.jwk import RSAKey, ECKey
import json
import os
import yaml

def generate_rsa_key(key_size, algorithm):
    rsa_key = RSAKey.generate_key(key_size, algorithm)
    rsa_key.ensure_kid()
    return rsa_key

def generate_ecc_key(curve, algorithm):
    ecc_key = ECKey.generate_key(curve, algorithm)
    ecc_key.ensure_kid()
    return ecc_key

def export_public_key(key):
    return key.as_dict(private=False)

def export_private_key(key):
    return key.as_dict(private=True)

def save_jwk_keys(public_keys, private_keys, directory):
    jwk_public_keys = {"keys": public_keys}
    jwk_private_keys = {"keys": private_keys}

    public_key_path = os.path.join(directory, "public.json")
    private_key_path = os.path.join(directory, "private.json")

    with open(public_key_path, "w") as f:
        json.dump(jwk_public_keys, f, indent=2)

    with open(private_key_path, "w") as f:
        json.dump(jwk_private_keys, f, indent=2)

def save_private_keys_to_yaml(private_keys, directory):
    jwk_private_keys = {"keys": private_keys}
    yaml_private_key_path = os.path.join(directory, "private.yaml")

    with open(yaml_private_key_path, "w") as f:
        yaml.dump(jwk_private_keys, f, default_flow_style=False)

def main():
    current_dir = os.path.dirname(os.path.abspath(__file__))

    rsa_key = generate_rsa_key(2048, {"alg": "RS256"})
    ecc_key = generate_ecc_key("P-256", {"alg": "ES256"})

    rsa_public_key = export_public_key(rsa_key)
    rsa_private_key = export_private_key(rsa_key)
    ecc_public_key = export_public_key(ecc_key)
    ecc_private_key = export_private_key(ecc_key)

    public_keys = [rsa_public_key, ecc_public_key]
    private_keys = [rsa_private_key, ecc_private_key]

    save_jwk_keys(public_keys, private_keys, current_dir)
    save_private_keys_to_yaml(private_keys, current_dir)

if __name__ == "__main__":
    main()

Keys on server

Let’s start with the easier solution first. According to the official documentation:

If you specify config.access_token_keyset or config.channel_token_keyset with either an http:// or https:// prefix, the plugin loads the keys just like it does for config.access_token_jwks_uri and config.channel_token_jwks_uri.

This means the JWT signer plugin can discover keys from an external URL and use them to sign new tokens.

For security reasons, you should restrict access to the private key endpoint and only allow Kong to access it. This can be easily achieved with a service mesh, using traffic permission and OPA. I will discuss this use case further in my upcoming post on End-to-End JWT Validation with Kong Gateway and Kong Mesh.

Since we already have the keys generated in the previous section, let’s use a server to host these keys. For hosting static files, I will use Caddy here. However, you can use any server you prefer—you could even start another Kong instance as the upstream server and use the request termination plugin to return the keys.

Prepare server

Let’s store our keys in the keys/ folder and add the following to Caddyfile and compose.yaml respectively.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
  admin off
  auto_https off
  http_port  8080
}

:8080 {
  route /* {
      respond "Page Not Found" 404
  }

  route /public {
    uri strip_prefix /public
    file_server {
        root /srv/keys
        index public.json
    }
  }

  route /private {
    uri strip_prefix /private
    basic_auth {
        # admin:qwer1234
        admin $2a$14$5MfI7/rMbsV/OMwqxw28geHNqJz14dR.clWwC3IsiXpxwGYfcG3W2
    }
    file_server {
        root /srv/keys
        index private.json
    }
  }
}

In this setup, the Caddy server is running within the same Docker network, kong-ee-net, as my Kong container and is accessible on port 8080.

1
2
3
4
5
6
7
8
9
10
11
12
services:
  jwk:
    image: caddy:2.8
    restart: unless-stopped
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./keys:/srv/keys
    networks:
      - kong-ee-net
networks:
  kong-ee-net:
    external: true

Once configured, the folder structure should look as follows, and you can start the server with docker compose up -d.

1
2
3
4
5
6
7
.
├── Caddyfile
├── compose.yaml
└── keys
    ├── private.json
    ├── private.yaml
    └── public.json

Config plugin

To configure the plugin to read the private key, our config can be as simple as below.

Please note, config.access_token_keyset_client_username and config.access_token_keyset_client_password were introduced in Kong Gateway version 3.7.0.0. If you are running a lower version, the JWT signer plugin will need to access the URL directly without any authentication.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
_format_version: "3.0"
_transform: true
services:
  - name: echo-svc
    url: http://echo:8080
    routes:
    - name: echo-route
      paths:
      - /echo
      plugins:
      - name: jwt-signer
        config:
          access_token_keyset: http://jwk:8080/private
          access_token_keyset_client_username: admin
          access_token_keyset_client_password: qwer1234
          access_token_signing_algorithm: RS256
          access_token_jwks_uri: https://<keycloak>/auth/realms/<realm>/protocol/openid-connect/certs
          channel_token_optional: true

Now, when we send a valid token to Kong, the upstream server will receive a new token signed by Kong, which can be validated using the public key we generated.

Keys in config

What if you are not allowed to host the private keys on a different server? Can you include the private key in the Kong config file instead? According to the official documentation:

If the prefix is not http:// or https:// (such as “my-company” or “kong”), Kong autogenerates JWKS for supported algorithms.

This suggests we might be able to include this keyset in the Kong config and reference the keyset name in our plugin config.

The solution involves using a custom entity named jwt_signer_jwks. Let’s experiment with the generated private.yaml key.

Prepare config

We’ll prepare our configuration again, naming the keyset demo and referencing it in the plugin config. Since the keyset is listed in the same config file, we no longer need config.access_token_keyset_client_username and config.access_token_keyset_client_password.

Next, stop the Caddy server and push below configuration to Kong via the <ADMIN_API>/config endpoint. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
_format_version: "3.0"
_transform: true
services:
  - name: echo-svc
    url: http://echo:8080
    routes:
    - name: echo-route
      paths:
      - /echo
      plugins:
      - name: jwt-signer
        config:
          access_token_keyset: demo
          access_token_signing_algorithm: RS256
          access_token_jwks_uri: https://<keycloak>/auth/realms/<realm>/protocol/openid-connect/certs
          channel_token_optional: true
jwt_signer_jwks:
- name: demo
  keys:
  - alg: RS256
    d: Ob0MJ_TMptvH5hWEC1TrWSR5fYEBJWtuT7imcGE1zkBpvVXedcsw-3xZKtOx5zexJguzcrEsQQUdtzQgGl8bfOg75_HeCr-oPVxt7mdh5cifYg5fqsw-G2_6v0dUkcHqWkO6hk_SdufvRK0Rq1MPwZYRSsD2buaFOn_T7kyepFsfUVsVvDOC5qjdb2ya2-0LQar97INAQU-wvan3ac_hvmm4JzaGfXV9gdcsD3WKMBBGZSj712_Mx-OcAedn_N8-kPQzrqD39qil7mi7ND9s35PhddHrh2zPYkA5SomPqf59fzJnajVpf1-MpOM8rpWbHlpDEKlWcE3ISkhoK2Om0w
    dp: FBXQZnytHU_YtTVD6sRhYypMx0Em6l8g-z55d24cESMlW1AtYFzHB-1yGkWjTJPvVbtfp1xYciLbj4vGdL36Q8y8OomAWEa3e7HFdhGQGO9jkauqxlXiaze2AmSgTIYXjC0y43HJAPmUL4EnW4SYagCDjkp_7H2l7qI6K_-vXGM
    dq: W_mzgvlUwzfAJW3kYKihgopUQ3vBcVZzrm4zfSR72q3lF7JzvDCQEj3gIE3RxoVyGt6h4k9hId7f7IMLaFLI0ToECImgLZeVgl9HLsJm3y7j4KWmH-L8_6kh5KIkzrO8VeSVFeYIpTh_0wcGrnUrNuE7WMMq7xox-FOQdq9Zm4c
    e: AQAB
    kid: B-6RSaK9JfF-GocG14iaoNoqF3HkNyOnAJc2-U6O5nw
    kty: RSA
    n: 9mkjjMGe9JzqQEnJHkgC1cMuFcvqZvMiXmM1Z4JoZe1HZ1vTauS_lpkhDVeoOoGvhvyQ1u0NmRUs6EAR2H6X2fyYmrtCVW5d2uLSvsgzXU4P7n3w1S4vFrcHMpZ7VF506bilmuY_rTibgyTSjfYQ1M_tviwuCqKOTKbYXjsLujAVg2Jbn_KwuXD7dCuPbcYhOPl9b2D1K8y_FAXQxln7bKK7zh6eVixhFG3Puma0lhpZihGH1fqw6PGhzo6ImodZIOrWg5yQVCBc-IEmv87w7y_HOlyLJHjW4DX056j5oEhWLxV8v6wdVFJFAu5grb7rCLpfGDf-m_4ZdHhtS_xRqQ
    p: _d8kNJYTJ_2IRc_3T9nkOtZgKC-z8RdrmkOtJaacFLKoFsmTTo9y09WjqH913U1dDVD9tzlQDscOyFLZKzkTZIYZV4Z8m2FmxBLRoXOXQTJWD3co1O4EGl8nQb9BrHrYEgI5mdSsJwiZTtD9apgqMbcadFNziGWhg30aISv1u-M
    q: -Hn8GkOj_Ep0KAmkqTFg1YthDU6o4I_olQZyaAdwjsbKe4at39v_JKY10jNJkSqbsvjyuVdhuK8x9gGVAlr0uBzX79nDpqLyVWwsAWjhVvvtYUf3s1s3kTffThSX16bh018Ukow7LBHolsQcINa0GCZETCzo0YF7wgzck7HeygM
    qi: MQfJKaQ6glGEatqPDlOxJoNSkblhR_ugM4_l6BqpX23bI2KfcfTP7VEFLULuuoDJDG5iMq_rhoO9W6K9SwFBUuiW_OfcqfH7WE9X5KEAAivCAskMY9sJ9jnQSvafPwCE1vboT1XbFg6BCsgT5Y6hNNlxndj9tGjKkd6h2LcEojk
  - alg: ES256
    crv: P-256
    d: oLjUgBRpm8Ppa5QI1GPunrkqV_Fiw3yEngR3KLY8f3A
    kid: z_gicTJ5EmhY1Dtw_ZLSGA1XXd2GDLuGJWMuAHgG1ho
    kty: EC
    x: ORjsb5gOGH4bwUxDUwO64h3vYLVYDe0OOcK7MWhKLuI
    y: GxBU5PINWLLWrQpYgADO7jXwZXy0qvrFdDRsIhcoV_A

Now, when we send valid tokens to Kong, the upstream server will see that Kong is using our generated key to sign the new tokens.

That’s all I want to share with you today. See you next time!