How to Get Real Client IP When Kong Is Behind CDN and Loadbalancer

, , ,

In my IP restriction post I briefly discussed how to set the real ip when Kong is deployed behind a load balancer. I would like to explore this a little bit further to show you how you can find out all IPs in front of Kong and how to start Kong to get the real client IP. Let’s get started.

Request Flow

In order to demonstrate extra hops of client requests, below is the set up we will use today.

1
2
3
4
5
6
7
8
9
10
11
12
                  ┌───────────────┐
┌──────────┐      │               │                         ┌────────┐
│  client  ├──────►               │            ┌────────────►  Kong  │
└──────────┘      │               │            │            └────────┘
┌──────────┐      │               │     ┌──────┴───────┐    ┌────────┐
│  client  ├──────►               ├─────► loadbalancer ├────►  Kong  │
└──────────┘      │               │     └──────┬───────┘    └────────┘
┌──────────┐      │               │            │            ┌────────┐
│  client  ├──────►               │            └────────────►  Kong  │
└──────────┘      │               │                         └────────┘
                  │           CDN │
                  └───────────────┘

I am going to use Cloudflare as CDN, Traefik as Loadbalancer. When client sends a request to kong, Kong should get 3 IP addresses on x-forwarded-header header.

Start Kong

I am going to deploy Kong 3.0 in dbless mode. Kong 3.0 has a LOT of new features. LMDB for storing declarative config IMO is a big one because Kong can finally persist config when it is pushed through /config endpoint when kong is deployed in dbless mode.

Let’s use below command to start Kong

1
2
3
4
5
6
7
8
docker run --detach --rm \
  --name kong \
  --network traefik \
  -p "8001:8001" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -e "KONG_DATABASE=off" \
  kong:3.0-ubuntu

On Traefik side, I just need to add below to my declarative config using file provider.

1
2
3
4
5
6
7
8
9
10
11
http:
  routers:
    konnect-dp:
      service: kong
      middlewares:
      rule: "Host(`test.demofor.fun`)"
  services:
    kong:
      loadBalancer:
        servers:
        - url: "http://kong:8000"

Now when we curl https://test.demofor.fun -is we should see something like below which confirms the request goes through Cloudflare and traefik (because traefik handles TLS for me)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ curl https://test.demofor.fun -is
HTTP/2 404
date: Thu, 27 Oct 2022 10:32:12 GMT
content-type: application/json; charset=utf-8
content-length: 48
permissions-policy: interest-cohort=()
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-kong-response-latency: 1
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=SIowi3dCQMppUMLdFcAk4gEz3xmyU8kl1ifzBH469F933vUCpMFi098ULS4H7JJ7WHpY7M6lEpGii%2BbASLezQI4XDhNwQr29fUmdHPNJ5klYLHJvCb7Ixayxgf5JSQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 760ac432fc89aaf3-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

{"message":"no Route matched with those values"}

Check IPs

Deploy echo server

In order to use IP restriction plugin successfully I would need to know the IP of Cloudflare edge and Traefik. To do that I will run a echo container locally which will returned x-forwarded-header it receives back to me.

1
2
3
4
docker run --detach --rm \
  --network traefik \
  --name echo \
  ealen/echo-server

Push kong config

Now we can push config to Kong. Let’s save below file to test.yaml.

1
2
3
4
5
6
7
8
9
10
_format_version: "3.0"
_transform: true

services:
- name: echo-svc
  url: http://echo
  routes:
  - name: echo-route
    paths:
    - /echo

Then we can use curl -X POST http://localhost:8001/config -F [email protected] to push the config.

Get IPs

Now when we run below command from our host or a different VM

1
curl https://test.demofor.fun/echo -s | jq -r '.request.headers["x-forwarded-for"]'

We should get something like 122.221.122.221, 108.162.249.37, 172.18.0.2.

Understand IPs from x-forwarded-for headers

Let’s take a look at these IP addresses

  • 122.221.122.221: This is my VM’s IP address which is the client IP I would like Kong to get and restrict on.
  • 172.18.0.2: This is a local IP address. If we check our container address, this is the IP of traefik container.
  • 108.162.249.37: This is a public IP address belongs to Cloudflare.

Now we know 122.221.122.221 is the IP I want Kong to get, we need to put 172.18.0.2 and 108.162.249.37 as trusted ips.

Deploy Kong with trusted_ips

Let’s stop and re-create our Kong container with below commands.

1
2
3
4
5
6
7
8
9
10
11
12
13
docker stop kong

docker run --detach --rm \
  --name kong \
  --network traefik \
  -p "8001:8001" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -e "KONG_DATABASE=off" \
  -e "KONG_TRUSTED_IPS=108.162.249.37, 172.18.0.2" \
  -e "KONG_REAL_IP_HEADER=X-Forwarded-For" \
  -e "KONG_REAL_IP_RECURSIVE=on" \
  kong:3.0-ubuntu

Now let’s update our test.yaml file to include the IP restriction plugin. In my example, I want to restrict access from my VM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
_format_version: "3.0"
_transform: true

services:
- name: echo-svc
  url: http://echo
  routes:
  - name: echo-route
    paths:
    - /echo
    plugins:
    - name: ip-restriction
      config:
        deny:
        - 122.221.122.221
        message: "Not Allow"

After pushing the config to Kong we can test from our VM and we probably will see most requests still are allowed. Why is the IP restriction plugin not working?

Deploy Kong with CIDR blocks in trust_ips

Let’s send a few more requests.

1
2
3
4
5
6
7
8
$ curl https://test.demofor.fun/echo -s | jq -r '.request.headers["x-forwarded-for"]'
122.221.122.221, 108.162.249.37, 172.18.0.2

$ curl https://test.demofor.fun/echo -s | jq -r '.request.headers["x-forwarded-for"]'
122.221.122.221, 172.68.146.37, 172.18.0.2

$ curl https://test.demofor.fun/echo -s | jq -r '.request.headers["x-forwarded-for"]'
122.221.122.221, 172.69.62.13, 172.18.0.2

If we take a closer look at the IPs returned from x-forwarded-for in every request, we should notice that the Cloudflare IP address changes in every request.

That means we can’t just put one single IP 108.162.249.37 to our trust list.

We can find cloudflare’s IP address from their website and we can see CIDR blocks like 108.162.192.0/18, 172.64.0.0/13 are listed. Let’s re-create our Kong container with these CIDR blocks in trusted_ips.

1
2
3
4
5
6
7
8
9
10
11
12
13
docker stop kong

docker run --detach --rm \
  --name kong \
  --network traefik \
  -p "8001:8001" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -e "KONG_DATABASE=off" \
  -e "KONG_TRUSTED_IPS=108.162.192.0/18, 172.64.0.0/13, 172.18.0.2" \
  -e "KONG_REAL_IP_HEADER=X-Forwarded-For" \
  -e "KONG_REAL_IP_RECURSIVE=on" \
  kong:3.0-ubuntu

After pushing the same test.yaml, if we curl from our VM again we should be blocked.

1
2
3
4
$ curl https://test.demofor.fun/echo -s
{
  "message":"Not Allow"
}

Summary

IP restriction plugin is very easy to use but it can get complicated when there are multiple hops before your client requests reach Kong. It is important to know what IPs the requests pass through and trust those IPs to make sure Kong gets the real client ip. Using a local echo container or even httpbin.org(if it is allowed) is a good way to find these IPs.

That’s all I want to share with you today, see you next time.