Explore Kubernetes Gateway API With Kong Ingress Controller

Kubernetes has become the standard for orchestrating containerized applications, offering declarative and automated management. Traditionally, ingress objects have been employed to handle external traffic to micro services within the cluster. However, to make Ingress resources effective, an ingress controller must be operational in the cluster. Despite its widespread use, Ingress comes with limitations, such as exclusively managing L7 traffic, restricting routing to services within the same namespace, and lacking features like header and query string matching.

To address these limitations, Gateway API has been introduced. It exposes a more versatile proxy API that supports multiple protocols beyond HTTP and models various infrastructure components. This offers enhanced deployment and management options for cluster operations. Roles and personas also play a crucial role in Gateway API, delineating distinct responsibilities for different teams within the system.

In this post, I’ll explore the standard version of Gateway API and showcase how you can utilize the Kong Ingress Controller to understand the capabilities offered by Gateway API.

Install Gateway API CRD

The Gateway API has certain CRDs that require manual installation. As previously mentioned, I will install the CRDs from the standard channel and explore Package v1 which includes:

  • Gateway
  • GatewayClass
  • HTTPRoute
1
2
kubectl apply -f \
https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Infrastructure Ops

With the CRDs installed, the next step is to define GatewayClass objects, which describe different types of Gateway instances. The official documentation can be a bit confusing, especially when it mentions, "Note: GatewayClass serves the same function as the networking.IngressClass resource." In reality, GatewayClass defines how you want to deploy your Gateway instances with different configurations.

Infrastructure providers can offer both internal and external GatewayClass options, enabling distinct features for different gateway instances. For instance, for an internal gateway, you may want to enable stream listening, while for an external gateway, you would only enable proxy listening.

Create GatewayClass

Let’s create the GatewayClass kong-demo in our cluster and set the controller name to be kic-gateway-controller. This is the default value for Kong Ingress Controller to reconciles any resources attached to this GatewayClass. For additional information, please refer to Kong official doc.

1
2
3
4
5
6
7
8
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: kong-demo
annotations:
konghq.com/gatewayclass-unmanaged: 'true'
spec:
controllerName: konghq.com/kic-gateway-controller

Cluster Operations

Gateway objects are meant to spin up gateway instances dynamically based on the GateayClass it reference to and the listeners they define. However, the Kong Gateway Operator is still in tech preview, we will deploy Kong manually.

Deploy Kong Ingress Controller

We will use helm to install, let’s add Kong repo and update first.

1
2
helm repo add kong https://charts.konghq.com
helm repo update

We need to make sure the kong/ingress chart version is 0.10.2 or later.

1
helm search repo kong/ingress -o json | jq .[].version

Next we can install Kong Ingress Controller with release name my-kong.

1
2
3
4
5
6
7
helm upgrade -i my-kong kong/ingress -n kong \
--set gateway.image.tag=3.5 \
--set gateway.env.router_flavor=expressions \
--set gateway.admin.enabled=true \
--set gateway.admin.http.enabled=true \
--set controller.ingressController.installCRDs=false \
--create-namespace

If you use MetalLB and it does not assign LoadBalancer IP automatically, you can annotate your proxy service <helm_release_name>-kong-proxy with the desired IP as shown below.

1
2
3
kubectl -n kong annotate \
service my-kong-gateway-proxy \
metallb.universe.tf/loadBalancerIPs=192.168.18.150

Create Gateway

As we are deploying Kong Ingress Controller manually, it’s crucial to ensure that the listener on the Gateway object matches the Kong Proxy service. You can create the Gateway in the same namespace as your HTTPRoute. If you want to use the same Gateway to route traffic across the cluster, define a Gateway in one namespace and specifically allow HTTPRoute to use it. In the example below, I create the Gateway kong-gw-demo in the kong namespace. Then, I allow HTTPRoute from namespaces with the label shared-gateway-access: "true" to use this Gateway.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: kong-gw-demo
namespace: kong
spec:
gatewayClassName: kong-demo
listeners:
- name: proxy
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"

If you want to limit the usage of Gateway to specific namespaces without extra label, you can define your gateway as below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: kong-gw-demo
namespace: kong
spec:
gatewayClassName: kong-demo
listeners:
- name: proxy
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- httpbin
- nginx

Create demo Apps

Next we will deploy two applications for our demos.

1
2
3
4
kubectl create namespace httpbin
kubectl label namespace httpbin shared-gateway-access=true
kubectl -n httpbin create deployment httpbin --image=mccutchen/go-httpbin
kubectl -n httpbin expose deployment httpbin --name httpbin-svc --port 8080
1
2
3
4
kubectl create namespace nginx
kubectl label namespace nginx shared-gateway-access=true
kubectl -n nginx create deployment nginx --image=nginx:alpine
kubectl -n nginx expose deployment nginx --name nginx-svc --port 80

HTTPRoute

The HTTPRoute defines how you want to match a request based on conditions (matches), process it (filters), and forward the request to a backend service (backendRefs). With everything set up, let’s delve into the HTTPRoute API specification to explore its capabilities.

HTTPRouteMatch

Let’s explore the criteria that can be utilized for matching requests.

HTTPPathMatch

Similar to using Ingress, we can use annotations on HTTPRoute to enhance routing capability.

The first criteria is to match requests path. As we can see from below examples, path supports Exact, PathPrefix and RegularExpression. This is similar to the Ingress Path types except for the Regex one.

  • Exact

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: nginx-route
    namespace: nginx
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: Exact
    value: /nginx
    backendRefs:
    - name: nginx-svc
    kind: Service
    port: 80
  • PathPrefix

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080
  • RegularExpression

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-regex-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: RegularExpression
    value: /api/v(1|2)
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080

HTTPHeaderMatch

We can match request by header on HTTPRoute directly now. You need to use konghq.com/headers.* annotation to achieve the same on Ingress. Header matching support Exact and RegularExpression.

  • Exact

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    headers:
    - name: "x-version"
    type: Exact
    value: "2"
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080
  • RegularExpression

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    headers:
    - name: "x-version"
    type: RegularExpression
    value: "(1|2)"
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080

HTTPQueryParamMatch

This is supported by using expressions router flavour only.

With expressions and KIC, we can also match request by query string. Same as match by header, query string match supports Exact and RegularExpression.

  • Exact

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    queryParams:
    - type: Exact
    name: version
    value: 2
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080
  • RegularExpression

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: 'true'
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    queryParams:
    - type: RegularExpression
    name: version
    value: (1|2)
    backendRefs:
    - name: httpbin-svc
    kind: Service
    port: 8080

HTTPMethod

Continuing, we can also match requests based on the HTTP method. In the following example, we match GET and POST methods at the path /test.

Note: There is a limitation to match one method per match path. In my opinion it should support multiple methods (as an array) but this is considered a breaking change hence it is not being done at the moment. GitHub issue

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: httpbin
annotations:
konghq.com/strip-path: 'true'
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
method: "POST"
- path:
type: PathPrefix
value: /test
method: "GET"
backendRefs:
- name: httpbin-svc
kind: Service
port: 8080

Host name Match

If we want to use host name to the matching rule, we can put the host names on the HTTPRoute object. As we did not specify hostname on our Gateway Listener, the listener allows all hosts names to be matched. In the example below, the HTTPRoute can match all three domain names proxy.li.k8s, test.proxy.li.k8s or test.proxy.li.test with path /test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: httpbin
annotations:
konghq.com/strip-path: 'true'
spec:
hostnames:
- proxy.li.k8s
- test.proxy.li.k8s
- test.proxy.li.test
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
backendRefs:
- name: httpbin-svc
kind: Service
port: 8080

The cluster operator can also impose restrictions on the hostnames that the listener should respond to. By adding hostnames to the listener section, only the allowed hostnames will be matched to the HTTPRoute. Hostnames prefixed with a wildcard label (*) are interpreted as a suffix match. This means that a match for *.example.com would include both test.example.com and foo.test.example.com, but not example.com.

Now, let’s update our Gateway object kong-gw-demo to include hostname: "*.li.k8s", above HTTPRoute will only match the requests that has path /test and hostname proxy.li.k8s or test.proxy.li.k8s, not test.proxy.li.test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: kong-gw-demo
namespace: kong
spec:
gatewayClassName: kong-demo
listeners:
- name: proxy
hostname: "*.li.k8s"
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"

Here is a table explaining how hostname match on Gateway and HTTPRoute.

Gateway HTTPRoute Matched Hostname
proxy.li.k8s *.li.k8s proxy.li.k8s
————– —————– ——————
*.li.k8s *.li.k8s *.li.k8s
————– —————– ——————
*.li.k8s proxy.li.k8s proxy.li.k8s
test.proxy.li.k8s test.proxy.li.k8s
proxy.li.net Not match

HTTPRouteFilter

HTTPRouteFilter helps to process request or respoonse right through the request lifecycle. While Kong plugins offer additional capabilities, here we’ll delve into what can be accomplished using the native Gateway API.

RequestHeaderModifier

As name suggested, you can modify request headers directly on the HTTPRoute. It supports three actions:

  • set: Set overwrites the request with the given header (name, value) before the action.
  • add: Add adds the given header(s) (name, value) to the request before the action. It appends to any existing values associated with the header name.
  • remove: Remove the given header(s) from the HTTP request before the action. The value of Remove is a list of HTTP header names. Note that the header names are case-insensitive.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: httpbin
annotations:
konghq.com/strip-path: "true"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: x-test-id
value: fomm
set:
- name: User-Agent
value: fomm-local
remove: ["Accept"]
backendRefs:
- name: httpbin-svc
kind: Service
port: 8080

ResponseHeaderModifier

Similar to RequestHeaderModifier, you can also change Response header. It also support the same 3 actions, set,add and remove.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: httpbin
annotations:
konghq.com/strip-path: "true"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
add:
- name: x-test-id
value: fomm
set:
- name: User-Agent
value: fomm-local
remove: ["Access-Control-Allow-Origin", "Access-Control-Allow-Credentials"]
backendRefs:
- name: httpbin-svc
kind: Service
port: 8080

RequestRedirect

This filter is designed to short-circuit the request and redirect it to an entirely different destination. Consequently, the backendRef cannot be used with this filter, as it doesn’t align with the concept of redirection. If you don’t specify the path, port, host, etc., it will utilize the original request information. In the example below, the request is short-circuited to httpbin server.

There are two types, ReplacePrefixMatch and ReplaceFullPath. However, from testing it seems Kong Ingress Controller does not support ReplacePrefixMatch at the moment. You can find ReplaceFullPath below.

  • ReplaceFullPath

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: httpbin-route
    namespace: httpbin
    annotations:
    konghq.com/strip-path: "true"
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gw-demo
    namespace: kong
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /test
    filters:
    - type: RequestRedirect
    requestRedirect:
    scheme: http
    hostname: httpbin.org
    path:
    type: ReplaceFullPath
    replaceFullPath: /anything/paprika
    statusCode: 301
    port: 80

ExtensionRef

ExtensionRef is an implementation-specific extension to the filter behavior. For Kong Ingress Controller we can use it to apply KongPlugin on the HTTPRoute. This is similar to adding konghq.com/plugins annotation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: key-auth-example
namespace: httpbin
plugin: key-auth
config:
key_names:
- apikey
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: acl-example
namespace: httpbin
plugin: acl
config:
allow:
- group1
hide_groups_header: true
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: httpbin
annotations:
konghq.com/strip-path: "true"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
filters:
- type: ExtensionRef
extensionRef:
group: configuration.konghq.com
kind: KongPlugin
name: key-auth-example
- type: ExtensionRef
extensionRef:
group: configuration.konghq.com
kind: KongPlugin
name: acl-example
backendRefs:
- name: httpbin-svc
kind: Service
port: 8080

BackendRefs

BackendRef is a reference to a backend where the request should be forwarded to.

Cross-namespace routing

As mentioned earlier, the Ingress object cannot route requests to a service in a different namespace. However, Gateway API has changed this limitation, allowing users to route traffic to services in different namespaces. To achieve this, we need to use ReferenceGrant to permit requests to flow through.

I need to patch namespace kong to have label shared-gateway-access=true so KIC will response to the HTTPRoute in namespace kong.

1
kubectl label namespace kong shared-gateway-access=true

Next we can create our HTTPRoute in kong namespace and route request to httpbin service in httpbin namespace.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin-route
namespace: kong
annotations:
konghq.com/strip-path: "true"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
hostnames:
- proxy.li.k8s
rules:
- matches:
- path:
type: PathPrefix
value: /test
backendRefs:
- name: httpbin-svc
namespace: httpbin
kind: Service
port: 8080
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: httpbin-grant-kong
namespace: httpbin
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: kong
to:
- group: ""
kind: Service

Weighted routing

Another useful built-in feature is traffic splitting by setting weights on different backends. In the example below, traffic is split 90% to httpbin and 10% to nginx.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx-httpbin-route
namespace: kong
annotations:
konghq.com/strip-path: 'true'
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kong-gw-demo
namespace: kong
rules:
- matches:
- path:
type: PathPrefix
value: /test
backendRefs:
- name: httpbin-svc
namespace: httpbin
kind: Service
namespace: httpbin
port: 8080
weight: 90
- name: nginx-svc
kind: Service
namespace: nginx
port: 80
weight: 10
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: httpbin-grant-kong
namespace: httpbin
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: kong
to:
- group: ""
kind: Service
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: nginx-grant-kong
namespace: nginx
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: kong
to:
- group: ""
kind: Service

Summary

Kubernetes Gateway API is undoubtedly the future, and I encourage you to start exploring this new standard. Many more features are continuously being added too.

There’s still a lot for us to delve into, including TLS termination and cert-manager integration, which significantly differ from Ingress. Additionally, L4 routing is another aspect we may want to discuss, but it’s probably better to cover these topics in separate posts.

That’s all I want to share with you today, and I look forward to seeing you next time.