How to Use Istio With Kong Ingress Controller

Transforming software from monolith to micro services is becoming a tread in the past few years. Covid-19 has undoubtedly pushed this transformation further. Previously I had talked a lot about using Kong as the API gateway to secure, transform and monitor client requests to upstream services. What about traffics between internal services? The answer is using Service Mesh.

According to this article on solo.io, Istio has the most market share at the moment. In today’s post, I would show you how to use Kong Ingress Controller with Istio including demos of enabling mTLS, adding AuthorizationPolicy, change traffic distribution using VirtualService and DestinationRule.

Let’s get started.

Prerequisites:

  • Istioctl: If you haven’t installed it, you can follow the official doc.
  • Helm: I need to use helm to install Kong. Helm version is 3.7.1.
  • Your cluster need to be able to provision LoadBalancer. I am using metalLB locally.

Install Istio

As suggested by official documentation, I am going to install istio with minimal configuration profile first. Then I will install Kong Ingress Controller or Istio Ingress Gateway to allow requests entering the mesh.

1
2
3
istioctl install --set profile=minimal \
--set meshConfig.accessLogFile=/dev/stdout \
--skip-confirmation

Install BookInfo

We will use istio’s bookinfo in our demo.

1
2
3
4
5
6
# Create namespace bookinfo
kubectl create namespace bookinfo
# Add lable to namespace for auto sidecar injection.
kubectl label namespace bookinfo istio-injection=enabled
# Install bookinfo app
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo

Next I will show you some basic usage of Istio Ingress Gateway so we can compare it with Kong Ingress Controller. If you are only interested in testing Kong Ingress controller, you can go to the next section.

(Optional) Istio Ingress Gateway

Installation

  1. Add helm chart

    1
    2
    helm repo add istio https://istio-release.storage.googleapis.com/charts
    helm repo update
  2. Install Istio Ingress Gateway

    1
    2
    kubectl create namespace istio-ingress
    helm install istio-ingress istio/gateway -n istio-ingress --wait
  3. Once the pod is ready, let’s save load balancer IP to ISTIO_INGRESS_GATEWAY.

    1
    export ISTIO_INGRESS_GATEWAY=$(kubectl -n istio-ingress get svc istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Let’s curl $ISTIO_INGRESS_GATEWAY

1
curl -s http://$ISTIO_INGRESS_GATEWAY -iv

We would get Connection refused error because istio ingress gateway requires Gateway resource to receive incoming HTTP/TCP connections.

1
2
3
4
5
*   Trying 172.18.18.150:80...
* connect to 172.18.18.150 port 80 failed: Connection refused
* Failed to connect to 172.18.18.150 port 80 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.18.18.150 port 80 after 0 ms: Connection refused

Create Gateway

Let’s create a Gateway resource.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: bookinfo
spec:
selector:
istio: ingress
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
EOF

Then we can try sending the request again and we should be able to connect

1
curl http://$ISTIO_INGRESS_GATEWAY -iv

Create VirtualService

Next step we will use VirtualService to route requests to service productpage at port 9080.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
namespace: bookinfo
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
EOF

Now we can access the product page.

1
curl http://$ISTIO_INGRESS_GATEWAY/productpage

Enable JWT auth

We can also enable JWT authentication on ingress gateway by applying below RequestAuthentication and AuthorizationPolicy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f - << EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: jwt-authenticator
namespace: istio-ingress
spec:
selector:
matchLabels:
istio: ingress
jwtRules:
- issuer: [email protected]
jwksUri: https://raw.githubusercontent.com/istio/istio/release-1.11/security/tools/jwt/samples/jwks.json
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: require-jwt
namespace: istio-ingress
spec:
selector:
matchLabels:
istio: ingress
action: ALLOW
rules:
- from:
- source:
requestPrincipals: ["*"]
EOF

If we access curl http://$ISTIO_INGRESS_GATEWAY/productpage -i again, we will get 403 Forbidden.

1
2
3
4
5
6
7
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Fri, 26 Nov 2021 02:16:18 GMT
server: istio-envoy

RBAC: access denied

To access productpage again we need to pass a JWT token with authorization bearer header in our request.

1
curl http://$ISTIO_INGRESS_GATEWAY/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ"

Let’s move on and try Kong Ingress Controller.

Install Kong

I will install Kong in dbless mode in namespace kong and auto inject istio sidecar proxy.

1
2
3
4
5
6
7
8
# Create Namespace Kong
kubectl create namespace kong
# Label namespace to allow istio sidecar injection
kubectl label namespace kong istio-injection=enabled
# Install Kong with helm
helm upgrade -i my-kong kong/kong -n kong \
--set image.tag=2.6 \
--set ingressController.installCRDs=false --wait

Once the Pod is running, we can save kong proxy ip to KONG_PROXY_IP.

1
export KONG_PROXY_IP=$(kubectl -n kong get svc my-kong-kong-proxy -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Use Kong Ingress Controller

As I mentioned in my previous post, Kong ingress controller use CRDs and annotations to extend native kubernetes objects.

Create Ingress

Let’s start by creating an Ingress object. This ingress object has the same routes as the VirtualSerivce above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kong-istio-ingress
namespace: bookinfo
spec:
ingressClassName: kong
rules:
- http:
paths:
- path: /(productpage|login|logout)
pathType: Exact
backend:
service:
name: productpage
port:
number: 9080
- http:
paths:
- path: /(|static|api/v1/products).*
pathType: Prefix
backend:
service:
name: productpage
port:
number: 9080
EOF

Now we can access /productpage via Kong.

1
curl $KONG_PROXY_IP/productpage

Enable JWT plugin

We will then use JWT plugin to restrict access. I am applying the plugin globally.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
kubectl apply -f - << EOF
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: jwt-user
namespace: bookinfo
annotations:
kubernetes.io/ingress.class: "kong"
username: jwt-user
credentials:
- jwt-key-rs256
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: jwt-auth
namespace: bookinfo
annotations:
kubernetes.io/ingress.class: kong
labels:
global: "true"
plugin: jwt
---
apiVersion: v1
kind: Secret
metadata:
name: jwt-key-rs256
namespace: bookinfo
type: Opaque
stringData:
kongCredType: jwt
key: 'istio[email protected]'
algorithm: RS256
rsa_public_key: |
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxAE7eB6qugXyCAG3yhh7
pkDkT65pHymX+P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV/xFj9VswgsCg4R6o
tmg5PV2He95lZdHtOcU5DXIg/pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD+91gVuoe
JT/DwtGGcp4ignkgXfkiEm4sw+4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoU
qgBo/+4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa/b3y5u
+YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ/EyxOGuHJrLsn00fn
MQIDAQAB
-----END PUBLIC KEY-----
EOF

Now when we access /productpage route again,

1
curl $KONG_PROXY_IP/productpage -i

we should get 401 Unauthorized error.

1
2
3
4
5
6
7
8
9
10
HTTP/1.1 401 Unauthorized
date: Fri, 26 Nov 2021 03:35:48 GMT
content-type: application/json; charset=utf-8
content-length: 26
x-kong-response-latency: 0
server: istio-envoy
x-envoy-upstream-service-time: 0
x-envoy-decorator-operation: my-kong-kong-proxy.kong.svc.cluster.local:80/*

{"message":"Unauthorized"}

If we pass in an authorization bearer header, we are allowed to access the service again.

1
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ"

Next we will explore some istio features and see how it works with Kong ingress controller.

Use Istio features

Monitor Mesh

Monitoring might be one of the most important features that you use service mesh for. I will use official sample yaml file to install kiali dashboard and prometheus and jaeger will be used to get some data from mesh

1
2
3
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/jaeger.yaml

Once pods is ready, we can access dashboard with istioctl dashboard kiali.

Let’s go to Graph and then let’s tick Security in the Display dropdown. Please make sure we are checking Namespace bookinfo. After that, let’s send some requests.

1
2
3
4
5
while true
do
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ" > /dev/null
sleep 0.5
done

We should see below traffic flow.

As we can see not all traffics are encrypted.

Enable mTLS

Zero Trust architecture requires ALL traffics to be encrypted and verified. We can enable mTLS easily with istio to achieve this goal. What we need to do is to apply below PeerAuthentication in root namespace istio-system. This will enable strict mTLS in the mesh.

1
2
3
4
5
6
7
8
9
10
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "default"
namespace: "istio-system"
spec:
mtls:
mode: STRICT
EOF

If we visit curl $KONG_PROXY_IP/productpage now we will get curl: (56) Recv failure: Connection reset by peer error because the sidecar in Kong pod is expecting a client certificate to be passed in with the request. As Kong ingress controller is our entry point, we want to allow requests coming in. What we need to do is to disable PeerAuthentication in kong namespace so the traffic between Kong and Bookinfo remains encrypted and the client requests enter the mesh via Kong.

1
2
3
4
5
6
7
8
9
10
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: kong
spec:
mtls:
mode: DISABLE
EOF

Once that’s added, let’s try curl $KONG_PROXY_IP/productpage again. This time you would get upstream connect error or disconnect/reset before headers. reset reason: connection termination error. To fix this issue we need to annotate productpage service and our ingress.

1
2
kubectl -n bookinfo annotate service productpage ingress.kubernetes.io/service-upstream=true
kubectl -n bookinfo annotate ingress kong-istio-ingress konghq.com/preserve-host=false

The first annotation ingress.kubernetes.io/service-upstream=true allows Kong to send request to this service ip address instead of the pod IP.

The second annotation konghq.com/preserve-host=false helps us to NOT preserve request host header. The reason is that we want productpage to get the request from sidecar.

After applying annotations we can send our request again and we should see below in graph. As we can see ALL traffic are secured now.

Split Traffics

Kong can be used to load balance client requests to different backends. However when the backend services are not directly connected to Kong, there is not much Kong can do. For example, in our demo Kong has no control over which review pod productpage send requests to. In order to route requests as we want them to, we can use VirtualService and DistinationRule to fine tune traffic inside mesh.

Let me give you an example. Let’s continue sending our request to /productpage.

1
2
3
4
5
while true
do
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ" > /dev/null
sleep 0.1
done

Then go to Kiali Dashboard -> Graph and then select Traffic Distribution under Display dropdown. Because the default load balancing algorithm is ROUND_ROBIN, we can see requests are almost equally distributed amoung 3 subsets.

Let’s apply below VirtualService and DestinationRule. These two objects split the traffic to different review subsets based on its assigned weight. In the example I want 10% of traffic goes to v1, 60% goes to v2 and 30% goes to v3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: slipt-reviews
namespace: bookinfo
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 10
- destination:
host: reviews
subset: v2
weight: 60
- destination:
host: reviews
subset: v3
weight: 30
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: reviews-subsets
namespace: bookinfo
spec:
host: reviews
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
EOF

After a while we should see the traffic distrubution changes.

That’s all I want to show you today. This is just the tip of the iceberg, istio have a ton of more features that you can explore.

Thank you for reading, see you next time.