The problem#
For a while I have been using Hugo to generate this site using Gitea Actions to build the image and Helm deploy the site. It was running within my Kubernetes cluster primarily by using a Helm chart I created but always ran into the same issue.
I wanted the ability to create a new post, commit and deploy without having to tag the new image and update the AppVersion in the Helm chart.
That’s it.
Every time I wanted to update the site it would take around 5-10 minutes tagging, waiting for the build, rebuilding the Helm chart, waiting for the pipelines to run just to get my new content out.
The solution#
I decided all I needed to realistically do was to build the :latest image and provide a way for the cluster to automatically update when it detected a difference between the currently running image and the image in my repo.
Now you might be worried about using the :latest as was I but I decided to go with it for the following reasons:
- Simplicity of deployment
- Not having to rely on my Gitea repo cleanups to reduce the file size
- Finally but most importantly, Hugo is fantastic at failing hard during the build process.
With the last point in hand I know it “should” never build a crap image and as a result the :latest image should be safe. Additionally there is nothing mission critical to to this site so convienience wins out here.
The implementation#
This section will have a lot of detail to show each step. I won’t go into the full details of the Helm chart to keep it simple but if you are interested drop me an email and I might post another post showing how I build the healthcheck and the Helm chart to work together. There is a full directory structure at the end of this post which will give you an idea of roughly what is going on with it.
You could build the image yourself, not use Helm, not build the image with the healthcheck built in. Pick and choose what you need, this is just for reference.
Building the Docker image#
Dockerfile#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| # Use Hugo with extended features as the build stage
FROM hugomods/hugo:exts AS build
WORKDIR /src
COPY . /src
# Build the Hugo site using the --minify flag to reduce the size of the generated files
RUN hugo --minify
# Use a lightweight nginx image for serving the static files
FROM nginx:alpine
# Clear the default nginx HTML directory
RUN rm -rf /usr/share/nginx/html/*
# Copy the built Hugo site from the build stage to the nginx HTML directory
COPY --from=build /src/public /usr/share/nginx/html
RUN chown -R nginx:nginx /usr/share/nginx/html
# Copy the entrypoint script and nginx configuration file
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
# Update the default nginx config to allow for the /health endpoint
COPY nginx.conf /etc/nginx/conf.d/default.conf
CMD ["/usr/local/bin/entrypoint.sh"]
|
entrypoint.sh#
Just to provide some extra detail on what is going on in this script we provide a termination handler which changes the endpoint of the /health from 200 to 503 so that Kubernetes is aware that the pod is about to die so traffic stops being routed to it. This means that users should never get an invalid response while the image is upgraded.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| #!/bin/sh
term_handler() {
echo "Changing /health endpoint to return 503"
sed -i 's/return 200/return 503/' /etc/nginx/conf.d/default.conf
nginx -s reload
echo "Waiting for 10 seconds"
sleep 10
echo "Stopping Nginx"
nginx -s quit
exit 0
}
trap 'term_handler' SIGTERM SIGINT SIGQUIT
nginx -g 'daemon off;' &
NGINX_PID=$!
wait $NGINX_PID
|
nginx.conf#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| server {
listen 80;
root /usr/share/nginx/html;
index index.html;
location / {
try_files $uri $uri/ =404;
error_page 404 /404.html;
}
location /health {
access_log off;
return 200 'OK';
add_header Content-Type text/plain;
}
}
|
Gitea actions pipeline for the image build#
In this stage you can see it builds new changes on the main branch, publishes to my registry using the latest tag and merges the amd64 and arm64 builds into a single manifest to allow any architecture in my cluster to run it.
Although this is Gitea Actions it aligns with GitHub actions and should realistically work the same.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
| name: Build Docker image
run-name: Build Docker image
on:
push:
branches:
- main
paths-ignore:
- 'helm/**'
env:
REGISTRY_URL: git.leece.im
jobs:
Build:
strategy:
matrix:
platform: [amd64, arm64]
runs-on: ubuntu-latest-${{ matrix.platform }}
steps:
- name: Checkout
uses: actions/checkout@v2
with:
ref: ${{ github.ref }}
- name: Log in to Gitea Docker Registry
uses: docker/login-action@v1
with:
registry: ${{ env.REGISTRY_URL }}
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- name: Build and push Docker image
run: |
docker build --no-cache -t ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-${{ matrix.platform }} . --push
Create-Manifest:
runs-on: ubuntu-latest
needs: Build
steps:
- name: Log in to Gitea Docker Registry
uses: docker/login-action@v1
with:
registry: git.leece.im
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- name: Create and push Docker manifest for the latest tag
run: |
docker manifest create ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-amd64 ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest-arm64
docker manifest push ${{ env.REGISTRY_URL }}/${{ github.repository }}:latest
|
Setting up Keel to automate image upgrades#
To automatically update the latest image to the most recent version I struggled to find something to do this, but while doing research for a automated dev environment I came across Keel.
Keel is helpful for automatically upgrading images within deployments either on a webhook or an interval. It is build primary for matching against semantic versioning which is helpful for development enviroments and has many other policies listed here.
As I use FluxCD for managing my clusters I created a HelmRelease file as shown below but you could also follow regular Helm setup or other deployment methods here.
I do not normally put these together in a single file but for simplicity they are together below. Additionally you may also see valuesFrom defined in the HelmRelease, this points to a secret for Discord webhooks to notify me on upgrades. There is no functional differences defined in here.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
| ---
apiVersion: v1
kind: Namespace
metadata:
name: keel
annotations:
environment: production
owner: ash@leece.im
labels:
name: keel
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: keel
namespace: flux-system
spec:
interval: 10m
url: https://charts.keel.sh
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: keel
namespace: flux-system
spec:
chart:
spec:
chart: keel
reconcileStrategy: ChartVersion
version: 1.0.*
sourceRef:
kind: HelmRepository
name: keel
interval: 10m0s
releaseName: keel
targetNamespace: keel
valuesFrom:
- kind: Secret
name: keel-values
|
Once this is deployed you will then see a pod appear, this will be used to monitor the repo, deployment and force the image to pull.
1
2
3
| ash@Ashs-Mac-mini ~ % k get pods -n keel
NAME READY STATUS RESTARTS AGE
keel-8667f984c8-gwvwz 1/1 Running 0 4d9h
|
Helm chart changes#
Now we have Keel deployed we can go ahead and modify our deployment in our Helm chart to add some annotations:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "leeceim.fullname" . }}
labels:
{{- include "leeceim.labels" . | nindent 4 }}
annotations:
keel.sh/policy: "force"
keel.sh/trigger: "poll"
keel.sh/match-tag: "true"
# keel.sh/pollSchedule: "@every 5m"
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
...
|
The annotations do the following:
keel.sh/policy: "force" allows us to force update the tag even if it doesn’t match a semantic version.keel.sh/trigger: "poll" means we want to poll at a set interval rather than wait for a webhook. I may set one up in the future but for now this is the easiest method.keel.sh/match-tag: "true" means only the exact defined tag in the deployment will be upgraded.keel.sh/pollSchedule: "@every 5m" although it is commented out we can override the default poll interval which can be defined in the Helm chart.
These could be defined in the values.yaml and templated into the deployment if you wish to do so.
Now we need to make a couple more changes.
In the default Helm template that gets provided with helm create the following is set in the values.yaml
1
2
3
| image:
...
pullPolicy: IfNotPresent
|
Ensure this is set to Always
1
2
3
| image:
...
pullPolicy: Always
|
Finally, ensure the appVersion is set to “latest” in Chart.yaml
Now once the updated Helm chart is deployed to the cluster we should be good to go. Go update the image however you want to do it, pipelines, manually etc… and the new latest image should be fetched within the interval defined or the default provided by Keel. You can check the logs of the Keel pod to see if there are any errors.
Conclusion#
By implementing this streamlined deployment pipeline with Keel, I’ve reduced the time it takes to publish new content from 5-10 minutes down to just the time needed to write and commit my changes.
This approach strikes a good balance between simplicity and reliability for a non-critical site like this blog. While using :latest tags in production is often discouraged, Hugo’s strong build validation helps mitigate the risks and the convenience gained is worth it for my use case.
If you have any questions about this setup or suggestions for improvements, feel free to reach out! You can find ways to contact me on my home page.
Directory reference#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| total 56
drwxr-xr-x@ 3 ash staff 96 Mar 20 00:18 archetypes
drwxr-xr-x@ 4 ash staff 128 Jun 14 00:20 content
-rw-r--r--@ 1 ash staff 360 Jun 14 01:35 Dockerfile
-rwxr-xr-x@ 1 ash staff 370 Mar 20 00:18 entrypoint.sh
-rw-r--r--@ 1 ash staff 139 Jun 14 00:17 go.mod
-rw-r--r--@ 1 ash staff 251 Jun 14 00:17 go.sum
drwxr-xr-x@ 6 ash staff 192 Jun 14 02:26 helm
-rw-r--r--@ 1 ash staff 2270 Jun 14 00:39 hugo.yaml
-rw-r--r--@ 1 ash staff 296 Mar 20 00:18 nginx.conf
-rw-r--r--@ 1 ash staff 771 Jun 14 00:21 README.md
drwxr-xr-x@ 2 ash staff 64 Jun 13 23:55 static
drwxr-xr-x@ 2 ash staff 64 Jun 14 00:14 themes
./archetypes:
total 8
-rw-r--r--@ 1 ash staff 102 Mar 20 00:18 default.md
./content:
total 8
drwxr-xr-x@ 4 ash staff 128 Jun 18 11:10 posts
-rw-r--r--@ 1 ash staff 132 Jun 14 00:20 search.md
./content/posts:
total 16
-rw-r--r--@ 1 ash staff 1503 Jun 18 11:25 k8s-hugo-pipeline.md
-rw-r--r--@ 1 ash staff 514 Jun 14 02:26 welcome.md
./helm:
total 16
-rw-r--r--@ 1 ash staff 1158 Jun 14 02:26 Chart.yaml
drwxr-xr-x@ 10 ash staff 320 Jun 14 12:21 templates
-rw-r--r--@ 1 ash staff 2371 Jun 14 02:17 values.yaml
./helm/templates:
total 56
-rw-r--r--@ 1 ash staff 1782 Mar 20 00:18 _helpers.tpl
-rw-r--r--@ 1 ash staff 2303 Jun 14 02:26 deployment.yaml
-rw-r--r--@ 1 ash staff 991 Mar 20 00:18 hpa.yaml
-rw-r--r--@ 1 ash staff 2079 Mar 20 00:18 ingress.yaml
-rw-r--r--@ 1 ash staff 1744 Mar 20 00:18 NOTES.txt
-rw-r--r--@ 1 ash staff 361 Mar 20 00:18 service.yaml
-rw-r--r--@ 1 ash staff 389 Mar 20 00:18 serviceaccount.yaml
drwxr-xr-x@ 3 ash staff 96 Mar 20 00:18 tests
./static:
total 0
./themes:
total 0
|