Debug a Kong Go Plugin Issue on Kubernetes

(Updated: )
, , , , , ,

In my previous post (I know it has been a while…🐢) I talked about how to use Kong Go plugin. I was using Docker for my demo at the time and everything works just fine. Recently I run into an issue where my golden image failed to start on kubernetes but it works fine with docker. After debugging this issue, I think it is worthy to write down the debugging process in the hope that it will help someone in the future.

Let’s get start.

Preparation

I still use go-hello plugin from official repo in our example and build my golden image on top of kong 3.3.

Build golden Image

When it comes to custom plugins or custom nginx template, it is best to build custom images. This way the operation team does not need to worry about managing plugin code in their infrastructure deployment pipeline. Kong also provides an official doc to create this custom image. Let’s follow the official instructions to build our image this time.

  • Download docker-entrypoint.sh
    Go to a folder and run 

    1
    wget https://raw.githubusercontent.com/Kong/docker-kong/master/docker-entrypoint.sh

  • Download Kong installation package
    We will download the latest Kong package for alpine image. You can find their package here. I can download the package with 

    1
    wget https://download.konghq.com/gateway-3.x-ubuntu-focal/pool/all/k/kong/kong_3.3.0_amd64.deb -O kong.deb

  • Create Dockerfile

    You can still use the Dockerfile on my previous post. I just want to show you the official way.

    A few things to call out

    • Use Ubuntu instead of Alpine as base image.
    • Remove /usr/local/bin/resty first as there will be an existing symlink which points to /openresty/bin/resty.
    • Add RUN chmod +x /docker-entrypoint.sh to make sure this entrypoint script is executable.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    FROM golang:latest as builder

    RUN apt-get install git gcc libc-dev curl
    RUN mkdir /go-plugins
    RUN curl https://raw.githubusercontent.com/Kong/go-plugins/master/go-hello.go -o /go-plugins/go-hello.go

    RUN cd /go-plugins && \
        go mod init kong-go-plugin && \
        go get github.com/Kong/go-pdk && \
        go mod tidy && \
        go build go-hello.go

    FROM ubuntu:20.04
      
    COPY kong.deb /tmp/kong.deb
      
    RUN set -ex; \
        apt-get update \
        && apt-get install --yes /tmp/kong.deb \
        && rm -rf /var/lib/apt/lists/* \
        && rm -rf /tmp/kong.deb \
        && chown kong:0 /usr/local/bin/kong \
        && chown -R kong:0 /usr/local/kong \
        && rm /usr/local/bin/resty \
        && ln -s /usr/local/openresty/bin/resty /usr/local/bin/resty \
        && ln -s /usr/local/openresty/luajit/bin/luajit /usr/local/bin/luajit \
        && ln -s /usr/local/openresty/luajit/bin/luajit /usr/local/bin/lua \
        && ln -s /usr/local/openresty/nginx/sbin/nginx /usr/local/bin/nginx \
        && kong version
      
    COPY docker-entrypoint.sh /docker-entrypoint.sh
    RUN chmod +x /docker-entrypoint.sh
    COPY --from=builder  /go-plugins/go-hello /usr/local/bin/
      
    USER kong
      
    ENTRYPOINT ["/docker-entrypoint.sh"]
      
    EXPOSE 8000 8443 8001 8444
      
    STOPSIGNAL SIGQUIT
      
    HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD kong health
      
    CMD ["kong", "docker-start"]

  • Build image
    Let’s build our our image with tag kong-go-hello:3.3-ubuntu

    1
    docker build --no-cache -t kong-go-hello:3.3-ubuntu .

Test golden image

Start container

To save us some time, I will deploy Kong in DBless mode with Docker. If you are not sure what are the options to deploy Kong with docker, please check this post.

1
2
3
4
5
6
7
8
9
docker run --rm -d --name kong-demo \
  -p "8000-8001:8000-8001" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -e "KONG_DATABASE=off" \
  -e "KONG_PLUGINS=bundled,go-hello" \
  -e "KONG_PLUGINSERVER_NAMES=go-hello" \
  -e "KONG_PLUGINSERVER_GO_HELLO_QUERY_CMD=go-hello -dump" \
  kong-go-hello:3.3-ubuntu

Post Config

Let’s save below file to /tmp/kong.yaml 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
_format_version: "3.0"
_transform: true

services:
- name: first-demo
  url: https://httpbin.org/anything
  routes:
  - name: first-demo-route
    paths:
    - /test

plugins:
- config:
    message: hello from kong Go plugin
  enabled: true
  name: go-hello

Now we can apply the config to Kong with curl.

1
curl -X POST http://localhost:8001/config -F config=@/tmp/kong.yaml

Test

Let’s send a request to Kong curl http://localhost:8000/test and we should see below response header.

1
x-hello-from-go: Go says hello from kong Go plugin to localhost:8000

This means our golden image works.

Reproduce issue

Now let me deploy Kong on Kubernetes using this image to reproduce the issue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
helm upgrade -i my-kong kong/kong \
  -n kong --create-namespace --values - <<EOF
  image:
    repository: kong-go-hello
    tag: 3.3-ubuntu
  env:
    log_level: debug
    plugins: bundled,go-hello
    pluginserver_names: go-hello
    pluginserver_go_hello_query_cmd: go-hello -dump
  admin:
    enabled: true
    http:
      enabled: true
  ingressController:
    installCRDs: false
EOF

We should see logs like below which suggests my go plugin server failed to start.

1
2
3
4
5
6
7
➜ kubectl logs -n kong deployments/my-kong-kong proxy
...
2023/06/03 11:14:27 [notice] 1272#0: *298 [kong] process.lua:252 Starting go-hello, context: ngx.timer
2023/06/03 11:14:27 [info] 1272#0: *298 [go-hello:2098] 2023/06/03 11:14:27 removing "/usr/local/kong/go-hello.socket": remove /usr/local/kong/go-hello.socket: read-only file system, context: ngx.timer
2023/06/03 11:14:27 [notice] 1272#0: signal 17 (SIGCHLD) received from 2098
2023/06/03 11:14:27 [notice] 1272#0: *298 [kong] process.lua:268 external pluginserver 'go-hello' terminated: exit 0, context: ngx.timer
...

Debugging process

As we can see I use the same go plugin settings, why did not it work on kubernetes?

The only hint we had was remove /usr/local/kong/go-hello.socket: read-only file system.

If you are familiar with Kong, you will know that Kong stores temporary files and logs in prefix directory. The default prefix directory is /usr/local/kong/ so it makes sense the go-hello.socket is stored in this folder. However, it did not seem right for this folder to be read-only and where did this read-only come from?

Check helm chart

At this point I suspect helm chart is responsible for read-only setting. Let’s check what the chart does.

Firstly, let’s find out the version I am running.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
➜ helm show chart kong/kong
apiVersion: v2
appVersion: "3.2"
dependencies:
- condition: postgresql.enabled
  name: postgresql
  repository: https://charts.bitnami.com/bitnami
  version: 11.9.13
description: The Cloud-Native Ingress and API-management
home: https://konghq.com/
icon: https://s3.amazonaws.com/downloads.kong/universe/assets/icon-kong-inc-large.png
maintainers:
- email: [email protected]
  name: hbagdi
- email: [email protected]
  name: rainest
name: kong
sources:
- https://github.com/Kong/charts/tree/main/charts/kong
version: 2.22.0

On Kubernetes, we use security context to define privilege and access control to a pod or container. For more information, please check official doc.

We can see I am running 2.22.0 which is most up to date version. Next let’s search securityContext on value.yaml file. We can see from these line, Kong’s chart sets containers security context to read-only. 

1
2
3
4
5
6
# securityContext for Kong pods.
securityContext: {}

# securityContext for containers.
containerSecurityContext:
  readOnlyRootFilesystem: true

I also notice that the chart use /kong_prefix/ as the prefix folder on kubernetes. (source) Why do they do it and how does this folder work when the container is read-only?

By searching keyword /kong_prefix, we found below information from chart changelog. 

1
2
- Create and mount emptyDir volumes for `/tmp` and `/kong_prefix` to allow
  for read-only root filesystem securityContexts and PodSecurityPolicys.

Test prefix folder

OK, now we know the prefix folder is /kong_prefix/ which means the socket file should be created in this folder, let’s tell kong where to find the socket with pluginserver_go_hello_socket: /kong_prefix/go-hello.socket and see how it work.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
helm upgrade -i my-kong kong/kong \
  -n kong --create-namespace --values - <<EOF
  image:
    repository: kong-go-hello
    tag: 3.3-ubuntu
  env:
    log_level: debug
    plugins: bundled,go-hello
    pluginserver_names: go-hello
    pluginserver_go_hello_query_cmd: go-hello -dump
    pluginserver_go_hello_socket: /kong_prefix/go-hello.socket
  admin:
    enabled: true
    http:
      enabled: true
  ingressController:
    installCRDs: false
EOF

Initially we see below in the log which suggest our setting works because kong was looking for the socket file from /kong_prefix/go-hello.socket.

1
2023/06/04 05:08:19 [crit] 1272#0: *299 connect() to unix:/kong_prefix/go-hello.socket failed (2: No such file or directory), context: ngx.timer

Then we started to get the same error as before and it seem Kong was still trying to find the socket from /usr/local/kong/, why?

1
2
3
4
2023/06/04 05:09:01 [notice] 1272#0: *298 [kong] process.lua:252 Starting go-hello, context: ngx.timer
2023/06/04 05:09:01 [info] 1272#0: *298 [go-hello:1634] 2023/06/04 05:09:01 removing "/usr/local/kong/go-hello.socket": remove /usr/local/kong/go-hello.socket: read-only file system, context: ngx.timer
2023/06/04 05:09:01 [notice] 1272#0: signal 17 (SIGCHLD) received from 1634
2023/06/04 05:09:01 [notice] 1272#0: *298 [kong] process.lua:268 external pluginserver 'go-hello' terminated: exit 0, context: ngx.timer

Check official doc

On the official doc, I noticed there was a -kong-prefix flag. From reading the description, it seems that we need to use this flag to allow go plugin to find the kong prefix folder.

Check Kong tests

To make sure my understanding is right, I also checked how Kong does tests for Go plugins. As we can see -kong-prefix " .. kong_prefix, is used, it means we definitely need to tell go plugin where to find the prefix folder.

1
2
3
4
5
6
7
8
9
10
11
assert(helpers.start_kong({
  nginx_conf = "spec/fixtures/custom_nginx.template",
  database = strategy,
  dns_hostsfile = dns_hostsfile,
  plugins = "bundled,reports-api,go-hello",
  pluginserver_names = "test",
  pluginserver_test_socket = kong_prefix .. "/go-hello.socket",
  pluginserver_test_query_cmd = "./spec/fixtures/go/go-hello -dump -kong-prefix " .. kong_prefix,
  pluginserver_test_start_cmd = "./spec/fixtures/go/go-hello -kong-prefix " .. kong_prefix,
  anonymous_reports = true,
}))

Test kong-prefix flag

Let’s add -kong-prefix and deploy Kong again. Hooray! No more error complaining plugin servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
helm upgrade -i my-kong kong//kong \
  -n kong --create-namespace --values - <<EOF
  image:
    repository: kong-go-hello
    tag: 3.3-ubuntu
  env:
    log_level: debug
    plugins: bundled,go-hello
    pluginserver_names: go-hello
    pluginserver_go_hello_socket: /kong_prefix/go-hello.socket
    pluginserver_go_hello_query_cmd: go-hello -dump -kong-prefix /kong_prefix
    pluginserver_go_hello_start_cmd: go-hello -kong-prefix /kong_prefix
  admin:
    enabled: true
    http:
      enabled: true
  ingressController:
    installCRDs: false
EOF

Test custom plugin

Let’s test the plugin with CRD.

Preparation

First let’s prepare a httpbin deployment, service and an Ingress.

1
2
3
4
kubectl create deployment httpbin --image=kennethreitz/httpbin
kubectl expose deployment httpbin --name httpbin-svc --port 80
kubectl create ingress httpbin-route --class kong --rule '/test*=httpbin-svc:80'
kubectl annotate ingress httpbin-route konghq.com/strip-path=true

When we send our request, we should see below.

1
2
3
4
5
6
7
8
9
10
11
12
➜ curl http://proxy.li.k8s/test/anything -I
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 428
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sun, 04 Jun 2023 05:31:32 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 2
X-Kong-Proxy-Latency: 0
Via: kong/3.3.0

Apply plugin via CRD

Now let’s apply the plugin globally.

1
2
3
4
5
6
7
8
9
10
11
12
13
kubectl apply -f - <<EOF
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: go-hello-test
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
config:
  message: hello from kong Go plugin on kubernetes
plugin: go-hello
EOF

Test plugin

Now when we send our request again, we should see the response headers as below which means our Go plugin works. 🎉

1
2
3
4
5
6
7
8
9
10
11
12
13
➜ curl http://proxy.li.k8s/test/anything -I
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 428
Connection: keep-alive
x-hello-from-go: Go says hello from kong Go plugin on kubernetes to proxy.li.k8s
Server: gunicorn/19.9.0
Date: Sun, 04 Jun 2023 05:32:35 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 2
X-Kong-Proxy-Latency: 1
Via: kong/3.3.0

That’s all I want to share with you today. See you next time.