Automating Hugo Deployments in Kubernetes with Keel

The problem
#

For a while I have been using Hugo to generate this site using Gitea Actions to build the image and Helm deploy the site. It was running within my Kubernetes cluster primarily by using a Helm chart I created but always ran into the same issue.

I wanted the ability to create a new post, commit and deploy without having to tag the new image and update the AppVersion in the Helm chart.

That’s it.

Every time I wanted to update the site it would take around 5-10 minutes tagging, waiting for the build, rebuilding the Helm chart, waiting for the pipelines to run just to get my new content out.

The solution
#

I decided all I needed to realistically do was to build the :latest image and provide a way for the cluster to automatically update when it detected a difference between the currently running image and the image in my repo.

Now you might be worried about using the :latest as was I but I decided to go with it for the following reasons:

Simplicity of deployment
Not having to rely on my Gitea repo cleanups to reduce the file size
Finally but most importantly, Hugo is fantastic at failing hard during the build process.

With the last point in hand I know it “should” never build a crap image and as a result the :latest image should be safe. Additionally there is nothing mission critical to to this site so convienience wins out here.

The implementation
#

This section will have a lot of detail to show each step. I won’t go into the full details of the Helm chart to keep it simple but if you are interested drop me an email and I might post another post showing how I build the healthcheck and the Helm chart to work together. There is a full directory structure at the end of this post which will give you an idea of roughly what is going on with it.

You could build the image yourself, not use Helm, not build the image with the healthcheck built in. Pick and choose what you need, this is just for reference.

Building the Docker image
#

Dockerfile
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Use Hugo with extended features as the build stage
FROM hugomods/hugo:exts AS build
WORKDIR /src
COPY . /src
# Build the Hugo site using the --minify flag to reduce the size of the generated files
RUN hugo --minify

# Use a lightweight nginx image for serving the static files
FROM nginx:alpine
# Clear the default nginx HTML directory
RUN rm -rf /usr/share/nginx/html/*
# Copy the built Hugo site from the build stage to the nginx HTML directory
COPY --from=build /src/public /usr/share/nginx/html
RUN chown -R nginx:nginx /usr/share/nginx/html
# Copy the entrypoint script and nginx configuration file
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
# Update the default nginx config to allow for the /health endpoint
COPY nginx.conf /etc/nginx/conf.d/default.conf
CMD ["/usr/local/bin/entrypoint.sh"]

entrypoint.sh
#

Just to provide some extra detail on what is going on in this script we provide a termination handler which changes the endpoint of the /health from 200 to 503 so that Kubernetes is aware that the pod is about to die so traffic stops being routed to it. This means that users should never get an invalid response while the image is upgraded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#!/bin/sh
term_handler() {
    echo "Changing /health endpoint to return 503"
    sed -i 's/return 200/return 503/' /etc/nginx/conf.d/default.conf
    nginx -s reload

    echo "Waiting for 10 seconds"
    sleep 10

    echo "Stopping Nginx"
    nginx -s quit

    exit 0
}
trap 'term_handler' SIGTERM SIGINT SIGQUIT
nginx -g 'daemon off;' &
NGINX_PID=$!
wait $NGINX_PID

nginx.conf
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
server {
    listen 80;
    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
        error_page 404 /404.html;
    }

    location /health {
        access_log off;
        return 200 'OK';
        add_header Content-Type text/plain;
    }
}

Gitea actions pipeline for the image build
#

In this stage you can see it builds new changes on the main branch, publishes to my registry using the latest tag and merges the amd64 and arm64 builds into a single manifest to allow any architecture in my cluster to run it.

Although this is Gitea Actions it aligns with GitHub actions and should realistically work the same.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
name: Build Docker image
run-name: Build Docker image
on:
  push:
    branches:
      - main
    paths-ignore:
      - 'helm/**'

env:
  REGISTRY_URL: git.leece.im

jobs:
  Build:
    strategy:
      matrix:
        platform: [amd64, arm64]
    runs-on: ubuntu-latest-${{ matrix.platform }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          ref: ${{ github.ref }}
      - name: Log in to Gitea Docker Registry
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY_URL }}
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - name: Build and push Docker image
        run: |
          docker build --no-cache -t ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-${{ matrix.platform }} . --push

  Create-Manifest:
    runs-on: ubuntu-latest
    needs: Build
    steps:
      - name: Log in to Gitea Docker Registry
        uses: docker/login-action@v1
        with:
          registry: git.leece.im
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - name: Create and push Docker manifest for the latest tag
        run: |
          docker manifest create ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-amd64 ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-arm64
          docker manifest push ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest

Setting up Keel to automate image upgrades
#

To automatically update the latest image to the most recent version I struggled to find something to do this, but while doing research for a automated dev environment I came across Keel.

Keel is helpful for automatically upgrading images within deployments either on a webhook or an interval. It is build primary for matching against semantic versioning which is helpful for development enviroments and has many other policies listed here.

As I use FluxCD for managing my clusters I created a HelmRelease file as shown below but you could also follow regular Helm setup or other deployment methods here.

I do not normally put these together in a single file but for simplicity they are together below. Additionally you may also see valuesFrom defined in the HelmRelease, this points to a secret for Discord webhooks to notify me on upgrades. There is no functional differences defined in here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
---
apiVersion: v1
kind: Namespace
metadata:
  name: keel
  annotations:
    environment: production
    owner: ash@leece.im
  labels:
    name: keel
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: keel
  namespace: flux-system
spec:
  interval: 10m
  url: https://charts.keel.sh
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: keel
  namespace: flux-system
spec:
  chart:
    spec:
      chart: keel
      reconcileStrategy: ChartVersion
      version: 1.0.*
      sourceRef:
        kind: HelmRepository
        name: keel
  interval: 10m0s
  releaseName: keel
  targetNamespace: keel
  valuesFrom:
    - kind: Secret
      name: keel-values

Once this is deployed you will then see a pod appear, this will be used to monitor the repo, deployment and force the image to pull.

1
2
3
ash@Ashs-Mac-mini ~ % k get pods -n keel
NAME                    READY   STATUS    RESTARTS   AGE
keel-8667f984c8-gwvwz   1/1     Running   0          4d9h

Helm chart changes
#

Now we have Keel deployed we can go ahead and modify our deployment in our Helm chart to add some annotations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "leeceim.fullname" . }}
  labels:
    {{- include "leeceim.labels" . | nindent 4 }}
  annotations:
    keel.sh/policy: "force"
    keel.sh/trigger: "poll"
    keel.sh/match-tag: "true"
#    keel.sh/pollSchedule: "@every 5m"
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
...

The annotations do the following:

keel.sh/policy: "force" allows us to force update the tag even if it doesn’t match a semantic version.
keel.sh/trigger: "poll" means we want to poll at a set interval rather than wait for a webhook. I may set one up in the future but for now this is the easiest method.
keel.sh/match-tag: "true" means only the exact defined tag in the deployment will be upgraded.
keel.sh/pollSchedule: "@every 5m" although it is commented out we can override the default poll interval which can be defined in the Helm chart.

These could be defined in the values.yaml and templated into the deployment if you wish to do so.

Now we need to make a couple more changes.

In the default Helm template that gets provided with helm create the following is set in the values.yaml

1
2
3
image:
  ...
  pullPolicy: IfNotPresent

Ensure this is set to Always

1
2
3
image:
  ...
  pullPolicy: Always

Finally, ensure the appVersion is set to “latest” in Chart.yaml

1
appVersion: "latest"

Now once the updated Helm chart is deployed to the cluster we should be good to go. Go update the image however you want to do it, pipelines, manually etc… and the new latest image should be fetched within the interval defined or the default provided by Keel. You can check the logs of the Keel pod to see if there are any errors.

Conclusion
#

By implementing this streamlined deployment pipeline with Keel, I’ve reduced the time it takes to publish new content from 5-10 minutes down to just the time needed to write and commit my changes.

This approach strikes a good balance between simplicity and reliability for a non-critical site like this blog. While using :latest tags in production is often discouraged, Hugo’s strong build validation helps mitigate the risks and the convenience gained is worth it for my use case.

If you have any questions about this setup or suggestions for improvements, feel free to reach out! You can find ways to contact me on my home page.

Directory reference
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
total 56
drwxr-xr-x@ 3 ash  staff    96 Mar 20 00:18 archetypes
drwxr-xr-x@ 4 ash  staff   128 Jun 14 00:20 content
-rw-r--r--@ 1 ash  staff   360 Jun 14 01:35 Dockerfile
-rwxr-xr-x@ 1 ash  staff   370 Mar 20 00:18 entrypoint.sh
-rw-r--r--@ 1 ash  staff   139 Jun 14 00:17 go.mod
-rw-r--r--@ 1 ash  staff   251 Jun 14 00:17 go.sum
drwxr-xr-x@ 6 ash  staff   192 Jun 14 02:26 helm
-rw-r--r--@ 1 ash  staff  2270 Jun 14 00:39 hugo.yaml
-rw-r--r--@ 1 ash  staff   296 Mar 20 00:18 nginx.conf
-rw-r--r--@ 1 ash  staff   771 Jun 14 00:21 README.md
drwxr-xr-x@ 2 ash  staff    64 Jun 13 23:55 static
drwxr-xr-x@ 2 ash  staff    64 Jun 14 00:14 themes

./archetypes:
total 8
-rw-r--r--@ 1 ash  staff  102 Mar 20 00:18 default.md

./content:
total 8
drwxr-xr-x@ 4 ash  staff  128 Jun 18 11:10 posts
-rw-r--r--@ 1 ash  staff  132 Jun 14 00:20 search.md

./content/posts:
total 16
-rw-r--r--@ 1 ash  staff  1503 Jun 18 11:25 k8s-hugo-pipeline.md
-rw-r--r--@ 1 ash  staff   514 Jun 14 02:26 welcome.md

./helm:
total 16
-rw-r--r--@  1 ash  staff  1158 Jun 14 02:26 Chart.yaml
drwxr-xr-x@ 10 ash  staff   320 Jun 14 12:21 templates
-rw-r--r--@  1 ash  staff  2371 Jun 14 02:17 values.yaml

./helm/templates:
total 56
-rw-r--r--@ 1 ash  staff  1782 Mar 20 00:18 _helpers.tpl
-rw-r--r--@ 1 ash  staff  2303 Jun 14 02:26 deployment.yaml
-rw-r--r--@ 1 ash  staff   991 Mar 20 00:18 hpa.yaml
-rw-r--r--@ 1 ash  staff  2079 Mar 20 00:18 ingress.yaml
-rw-r--r--@ 1 ash  staff  1744 Mar 20 00:18 NOTES.txt
-rw-r--r--@ 1 ash  staff   361 Mar 20 00:18 service.yaml
-rw-r--r--@ 1 ash  staff   389 Mar 20 00:18 serviceaccount.yaml
drwxr-xr-x@ 3 ash  staff    96 Mar 20 00:18 tests

./static:
total 0

./themes:
total 0

Reply by Email

The problem#

The solution#

The implementation#

Building the Docker image#

Dockerfile#

entrypoint.sh#

nginx.conf#

Gitea actions pipeline for the image build#

Setting up Keel to automate image upgrades#

Helm chart changes#

Conclusion#

Directory reference#