How to Log Request and Response Body With Kong

, ,

Kong provides a LOT of logging plugins for users to log request and response. Despite having all these logging plugins, they have one thing in common, you can’t log request and response body with them. Although there seems to be a lot of demand for this feature, this has never been added to Kong. Obviously this is by choice and confirmed by Kong engineer here.

The request/response body is left out of the logs on purpose as it is not bound (we don’t want to log the response body of a video streaming service) and will likely never be, within Kong itself.

Although I agree with this comment, there are always needs to log the request/response body and IMO users should always have the choice to do so. Previously it was only possible to log request and response with a custom logging. Luckily Kong introduced a new PDK since 2.4 version which allows users to set new log fields on the fly and output by one of official logging plugin.

By using this new PDK with pre-function plugin, we can inject request and response body on route/service/global level and then output by a logging plugin.

Let me show how easy this is.

Start Kong

I will deploy Kong in dbless mode and use admin API to push config via /config endpoint.

1
2
3
4
5
6
docker run --rm --detach --name kong \
  -p "8000-8001:8000-8001" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -e "KONG_DATABASE=off" \
  kong:3

This command should give us a kong container with admin API enabled.

Enable file log

Let’s save below file to /tmp/kong.yaml. This is a very simple setup with 1 service, 1 route and file log plugin enable globally.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
_format_version: "3.0"
_transform: true

services:
- name: demo-svc
  url: http://httpbin.org/anything
  routes:
  - name: demo-route
    paths:
    - /test

plugins:
- name: file-log
  config:
    path: /dev/stdout

Next, let’s push this config to Kong.

1
2
3
curl --request POST \
  --url http://localhost:8001/config \
  -F config=@/tmp/kong.yaml

Check original log

Now, let’s send a request to Kong and check the container log.

1
2
3
4
curl --request POST \
  --url http://localhost:8000/test \
  --header 'content-type: application/json' \
  --data '{"message":"this is testing"}'

As we can see from the container log, neither request body nor response body were logged.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
  "response": {
    "status": 200,
    "size": 901,
    "headers": {
      "content-length": "599",
      "access-control-allow-origin": "*",
      "access-control-allow-credentials": "true",
      "via": "kong/3.0.1",
      "x-kong-proxy-latency": "2",
      "server": "gunicorn/19.9.0",
      "connection": "close",
      "x-kong-upstream-latency": "452",
      "date": "Sat, 03 Dec 2022 12:39:08 GMT",
      "content-type": "application/json"
    }
  },
...
  "request": {
    "url": "http://localhost:8000/test",
    "method": "POST",
    "querystring": {},
    "uri": "/test",
    "size": 164,
    "headers": {
      "accept": "*/*",
      "content-length": "29",
      "host": "localhost:8000",
      "user-agent": "curl/7.81.0",
      "content-type": "application/json"
    }
  },
...

Enable pre-function

Let’s create a new config file at /tmp/kong-new.yaml and use pre-function plugin. I’ve also added another route other-route here and enable pre-function plugin ONLY on demo-route.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
_format_version: "3.0"
_transform: true

services:
- name: demo-svc
  url: http://httpbin.org/anything
  routes:
  - name: demo-route
    paths:
    - /test
  - name: other-route
    paths:
    - /demo

plugins:
- name: file-log
  config:
    path: /dev/stdout
- name: pre-function
  route: demo-route
  config:
    access:
    - kong.log.set_serialize_value("request.body", kong.request.get_raw_body())
    body_filter:
    - kong.log.set_serialize_value("response.body", kong.response.get_raw_body())

Let’s push this config.

1
2
3
curl --request POST \
  --url http://localhost:8001/config \
  -F config=@/tmp/kong-new.yaml

and send our request again.

1
2
3
4
curl --request POST \
  --url http://localhost:8000/test \
  --header 'content-type: application/json' \
  --data '{"message":"this is testing"}'

Check new logs

When we check our container log, we should see something similar as below. We can see response.body and request.body is in the log.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
  "response": {
    "status": 200,
    "body": "{\n  \"args\": {}, \n  \"data\": \"{\\\"message\\\":\\\"this is testing\\\"}\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Content-Length\": \"29\", \n    \"Content-Type\": \"application/json\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"curl/7.81.0\", \n    \"X-Amzn-Trace-Id\": \"Root=1-638b4761-0e748a27211a7a1147bc31b6\", \n    \"X-Forwarded-Host\": \"localhost\", \n    \"X-Forwarded-Path\": \"/test\", \n    \"X-Forwarded-Prefix\": \"/test\"\n  }, \n  \"json\": {\n    \"message\": \"this is testing\"\n  }, \n  \"method\": \"POST\", \n  \"origin\": \"172.17.0.1\", \n  \"url\": \"http://localhost/anything\"\n}\n",
    "size": 903,
    "headers": {
      "access-control-allow-origin": "*",
      "access-control-allow-credentials": "true",
      "content-length": "599",
      "via": "kong/3.0.1",
      "x-kong-proxy-latency": "359",
      "server": "gunicorn/19.9.0",
      "connection": "close",
      "x-kong-upstream-latency": "459",
      "date": "Sat, 03 Dec 2022 12:56:01 GMT",
      "content-type": "application/json"
    }
  },
...
  "request": {
    "url": "http://localhost:8000/test",
    "method": "POST",
    "body": "{\"message\":\"this is testing\"}",
    "querystring": {},
    "uri": "/test",
    "size": 164,
    "headers": {
      "accept": "*/*",
      "content-length": "29",
      "host": "localhost:8000",
      "user-agent": "curl/7.81.0",
      "content-type": "application/json"
    }
  },
...
}

If you send request to other-route at /demo, its log does not have request and response body.

One more thing

I agree the response log does not look clean and you might see an issue with other response body. The reason I chose kong.response.get_raw_body() PDK was kong response phase does not support http2, source.

If you know you will send your request in http1.1 only, you can log the response body with pre-function plugin as below.

1
2
3
4
5
6
7
8
- name: pre-function
  route: demo-route
  config:
    access:
    - kong.log.set_serialize_value("request.body", kong.request.get_raw_body())
    - kong.service.request.enable_buffering()
    body_filter:
    - kong.log.set_serialize_value("response.body", kong.service.response.get_body())

That’s all I want to share with you today, I hope you find this useful.