The Problem

While talking with a friend that works in IT, he repeated something that I’ve heard before many times: “It’s really difficult to guarantee consistency of developer experience.” It’s not that we don’t have the tools, but they’re scattered, and the workflow for deploying something are not integrated. Besides, it can extremely time consuming, and prone to mistakes.

But what if there could be a “paved road” approach, where we enforce strict configurations and policies, and establish guardrails, therefore guaranteeing that we can guarantee security, isolation, limits, and the ability to audit the system any time we need to? This is what I’m trying to build with k8s-mtp: a multi tenant platform based on Kubernetes that abstracts away things like RBAC, NetworkPolicy, ResourceQuotas, and a bunch of other important but difficult things, to guarantee that the industry best-practices are always in place.

While I’m no stranger to building dashboards, there was always something that bothered me when I was using Grafana for displaying the metrics I wanted to look at: it was a web app! Now, while web apps are much maligned, they do serve a purpose. But for the purpose of displaying rather the rather simple information that Prometheus collect (yes, simple: a complex metric is a useless metric IMO), a web app seems overkill. And now that TUIs are back en vogue, I thought it could be a fun exercise to build a TUI for displaying Prometheus metrics. Hence, gopromtui: a lightweight, terminal-native solution for displaying Prometheus metrics without any additional infrastructure.

After finishing my previous project, I felt an itch to continue building. One of the things that was noticeable during that process was that my coding skills were not being pushed: while the app I built did teach me about integration of Prometheus inside a Golang application, there wasn’t much else new for me there. So, for my next project, I chose to concentrate on building software.

The question then became: what should I build? It occurred to me that if I wanted to concentrate on improving my software development skills, that I should choose a domain that I have some experience in, so as to remove an extra layer of complexity for the project. As I have some experience already with Kubernetes, and given that I love CLIs, the union of those two things seemed like a natural choice. I even already had a cluster running that I could easily test things on!

We’ve now arrived at the last stretch of this journey. As mentioned in the previous post, we will now try and establish some ways to increase the visibility on what’s happening in our cluster. This will imply two things: on one hand, we will improve the app that was built for this project, with the objective of imbuing it with an integration to an event monitoring system; and then we will build the event monitoring system itself.

After installing and setting up all the different pieces of our CI/CD pipeline, we still need to do a bit of work to make sure that this process is as automated as possible, as is working in tandem to our benefit. This post is mostly about that.

Ingress and SSL

Even though cilium was installed in our system, we still need to configure the different environments to use it as its ingress, as well as configuring a load balancer and certificate manager. Thankfully these are all relatively straight-forward tasks. Let us start with the load balancer, which is going to be metallb. This is achieved by installing the following manifest: kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.2/config/manifests/metallb-native.yaml. Feel free to download it beforehand to double check its contents. After that is done and all the pods and services are up and running, we create the file metallb-config.yaml with the following contents: