tl;dr:

  • 3459 lines of code added (+3378 net)
  • 48 test functions across 9 new test files
  • 30 commits: 25 for new features and tests, 2 fixes, and 3 chores
  • Complete build pipeline rewrite: Docker replaced with ko (daemonless); no more external actions
  • Dex integration with JWT validation middleware + rate limiting
  • Full REST API: 8 endpoints for tenants + member CRUD
  • CLI tool: login via Dex password grant, 9 subcommands for management

Part 3 ended with a promise: “authentication, an API gateway with rate limiting built-in, and if I’m feeling generous both a CLI tool and a Web UI”. On the surface, I failed because I’m not delivering on the full promise, as there’s no Web UI (yet!). But there’s so much more that was added besides what was promised. We have a lot of tests! We have an even better CI/CD pipeline! We eliminated external dependencies! We have Helm charts! And, of course, we have all the other promises that were made before. I feel like this is a good trade-off: less show-off, but better foundations.

If part 3 was about building walls, this fourth part is about building doors that can lock properly.

What’s new#

A revamped CI pipeline; and a package registry as well#

This is probably the most exciting part for me. And it started out of pure laziness (which seems to be a somewhat recurring theme here). The story is simple: as I was testing the newly written CI workflow, the machine where I was testing it had gone through a kernel update, and thus when starting the docker daemon it failed since a module that was needed couldn’t be loaded. The solution is simple: a reboot. But this bothered me a bit because I was already not a fan of having to use docker on the pipeline, and reboots suck. Turns out there was a better way.

Enter ko. This beautiful piece of software leverages the fact that Go deployment is extremely simple, which makes you go from the standalone binary straight into the OCI layers in a single step. So, instead of needing to have the docker daemon running, a Dockerfile, and multiple (slow) build steps, you just compile the go binary, run ko, and an image that you can publish to a container registry comes out. Yes, it’s that simple.

But I wasn’t done. Of course, you still need a base image, as ko doesn’t handle that for you. The solution? Do it yourself. I was surprised at how simple this is, but again, this is just the beauty of building in Go. Three files, 6 lines of code:

FROM scratch
COPY passwd /etc/passwd
COPY group /etc/group
USER 65532:65532

nonroot:x:65532:

nonroot:x:65532:65532:nonroot:/home/nonroot:/sbin/nologin

There we go. I now have a base image that is rootless. And since our git forge also has a package registry, I can just publish it there and reference it elsewhere! With this, I can create a .ko.yaml file that handles all the steps I need to have our single binaries into images that I can publish in our registry.

defaultBaseImage: git.assilvestrar.club/lourenco/k8s-mtp/base:latest
builds:
  - id: api
    main: ./cmd/api
    platforms: [linux/amd64]
  - id: controller
    main: ./cmd/controller
    platforms: [linux/amd64]
  - id: webhook
    main: ./cmd/webhook
    platforms: [linux/amd64]

And one last bit of goodness: I keep harping on about out software supply chain security is so important, and how self-hosting is the way to go. But it turns out that talk is cheap, and I wasn’t following what I preached. Because it is truly hard sometimes, and there are some situations you don’t even notice creeping up on you. actions/checkout is one such situation. You don’t even think about it, but this single line is responsible for pulling an ungodly amount of Typescript code (we’re talking about thousands of lines of bundled JavaScript). All of this to checkout a git repository. This doesn’t make any sense.

So I did what any sane engineer would do, and built my own action. In less than 30 lines of yaml, including 6 lines of bash, I was able to reproduce 95%+ of the use cases that actions/checkout has. Don’t believe me? Here it goes:

name: 'Git checkout'
description: 'Self-hosted git checkout'
inputs:
  token:
    description: 'Gitea personal access token'
    required: true
  server_url:
    description: 'Gitea server URL (without .git suffix)'
    default: ${{ gitea.server_url }}
  repository:
    description: 'Repository path (owner/repo)'
    default: ${{ gitea.repository }}
  ref:
    description: 'Commit SHA, branch, or tag to checkout'
    default: ${{ gitea.sha }}
  depth:
    description: 'Shallow clone depth'
    default: '1'
runs:
  using: composite
  steps:
    - run: |
        rm -rf .git
        git init
        git remote add origin "${{ inputs.server_url }}/${{ inputs.repository }}"
        git config http.${{ inputs.server_url }}/.extraheader "Authorization: Bearer ${{ inputs.token }}"
        git fetch origin "${{ inputs.ref }}" --depth="${{ inputs.depth }}"
        git checkout FETCH_HEAD
      shell: bash

That’s it. And with this, I can simply replace uses: actions/checkout@v5 on my pipeline for uses: lourenco/actions/checkout@v1. That’s it. Everything works as it should, and no more importing external dependencies into this project. I’ve got to admit that I’m way too proud about this. But it is exactly the kind of change I love to implement, and it goes exactly into the philosophy of being self-reliant when you can. It’s that simple, sometimes; you just have to stop and think.

So, our CI/CD pipeline becomes extremely simple and effective, with our own actions, no docker, and a self-hosted package registry:

name: CI
on:
  push:
    branches: [main]
jobs:
  ci:
    runs-on: self-hosted
    env:
      GOPRIVATE: git.assilvestrar.club
    steps:
      - name: Checkout
        uses: lourenco/actions/checkout@v1
        with:
          token: ${{ secrets.REPO_SECRET }}
      - name: Lint
        run: make lint
      - name: Test
        run: make test
      - name: Ko Build & Push
        run: |
          echo "${{ secrets.REGISTRY_TOKEN }}" | ko login git.assilvestrar.club -u ${{ secrets.REGISTRY_USER }} --password-stdin
          for bin in api controller webhook; do
            KO_DOCKER_REPO=git.assilvestrar.club/lourenco/k8s-mtp/$bin ko build --bare --platform=linux/amd64 ./cmd/$bin/
          done
          rm /var/lib/act_runner/.docker/config.json
      - name: Helm Lint
        run: helm lint deploy/charts/k8s-mtp
      - name: Helm Package & Push
        run: |
          helm package deploy/charts/k8s-mtp
          echo "${{ secrets.REGISTRY_TOKEN }}" | helm registry login git.assilvestrar.club -u ${{ secrets.REGISTRY_USER }} --password-stdin
          helm push k8s-mtp-*.tgz oci://git.assilvestrar.club/lourenco/charts

Tests, so I’m not flying blind#

Tests are always a complicated part of any project. Not that they’re difficult to write, but they feel like a drag on the pace. You have to stop advancing in terms of features, and start solidifying what you already have. A good engineer will feel like this is a waste of time, while a great engineer will know that this is essential. Remember how last time around, 60% of the commits were about fixing bugs that appeared at runtime? Had I had a test suite in place, that would’ve probably been way less. And while it might be a “drag” at first, making sure that you’re not breaking the foundations is super important.

I have now implemented tests for 4 different parts of the project: config, store, tenant reconciler, and webhook. In total, I have 48 tests across 9 files, that cover the most significant parts of the project. And while that’s not nothing, there’s still a long way to get full coverage of the whole project. Still, this is a start that allows me to build on much firmer ground. And turns out that this was easier than what I was expecting. Even though k8s-mtp makes use of postgresql for runtime, it turns out that through the usage of the go-sqlmock package, the testing of the SQL queries that are used through the project can be mocked without having to spin up a real database. E.g.:

func TestListTenants_Success(t *testing.T) {
		h, mock := newTestHandler(t)
	
		mock.ExpectQuery(`SELECT.* FROM tenants.*WHERE deleted_at IS NULL`).
			WillReturnRows(sqlmock.NewRows([]string{"id", "name", "namespace", "tier", "owner_email"}).
				AddRow("uid-1", "test-tenant", "tenant-test-tenant", "free", "test@test.com"))
	
		req := httptest.NewRequest("GET", "/", nil)
		w := httptest.NewRecorder()
		h.ListTenants()(w, req)
		if w.Code != http.StatusOK {
			t.Errorf("expected status 200, got %v", w.Result().Status)
		}
	
		var tenants []TenantResponse
		json.NewDecoder(w.Body).Decode(&tenants)
		if len(tenants) != 1 {
			t.Errorf("expected 1 tenant, got %d", len(tenants))
		}
		if tenants[0].Name != "test-tenant" {
			t.Errorf("expected tenant name to be `test-tenant`, instead got %s", tenants[0].Name)
		}
		if err := mock.ExpectationsWereMet(); err != nil {
			t.Errorf("unmet expectations: %v", err)
		}
	}

As you can see, nothing here depends on having a running SQL database on the background. Instead, the usage of a mock library allows us to simulate the queries and results of said queries; and to assert whether the rest of the software is running properly given those queries and results. This is a godsend for quick and effective development, as we don’t have to deal with the time of spinning a database every time we want to test our packages. Sweet!

Authentication with Dex#

As I was advancing through the development of k8s-mtp, there was one decision that kept bugging me: how am I going to deal with authentication? On one hand, there was the possibility of learning a fair bit more about the OIDC and JWT specs; but if there’s something I’ve learned from other projects in the past, is that this is a sure way to head to burnout town. In the end, it felt like my time would be better spent trying to hone in on what I could contribute to make this project special, and hand rolling my own authentication suite was for sure contrary to that objective. So, I ended up settling on using Dex for this purpose.

The match was obvious: written in Golang, FOSS, and K8s native with CRDs. There were other options like Keycloak, but Dex is much more lightweight and can be deployed as a single binary as well. This made everything much more pleasurable to write, as it was just a matter of adding it to the Helm charts I was already providing for deploying the project. With 3 new files, for template, values and configmap changes, we were able to cleanly integrate this into our existing Helm charts. Mission accomplished!

For bootstrapping, I chose the obvious route: a staticPasswords connector that allows us to get tokens without having to install and configure an external IdP. With this, I was in a good place to build the JWT validation pipeline: first, it fetches the JWKS, which then caches by making use of the key id, and then validate the issuer and the audience, and it finishes by injecting the subject into the context.

Of note in this part is the fact that I chose to construct the public keys from scratch, with a pretty simple function:

func (j *jwk) publicKey() (any, error) {
	n, err := base64.RawURLEncoding.DecodeString(j.N)
	if err != nil {
		return nil, fmt.Errorf("unable to decode the rsa modulus")
	}
	e, err := base64.RawURLEncoding.DecodeString(j.E)
	if err != nil {
		return nil, fmt.Errorf("unable to decode the rsa exponent")
	}

	ei := 0
	for _, b := range e {
		ei = ei<<8 | int(b)
	}
	return &rsa.PublicKey{N: new(big.Int).SetBytes(n), E: ei}, nil
}

Rate limiting, written from scratch#

It’s absolutely true that there are a ton of rate-limiting libraries already in existence, ready to plug and play. But I had a feeling that I could learn something new here, and turns out I was right. Let’s have a look at the whole implementation, which is less than 40 lines of code:

func (rl *RateLimiter) Limit(next http.HandlerFunc) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		key := UserFromContext(r.Context())
		if key == "" {
			key = r.RemoteAddr
		}

		rl.mu.Lock()
		userBucket, ok := rl.buckets[key]
		if !ok {
			userBucket = &bucket{tokens: float64(rl.burst), lastRefill: time.Now()}
			rl.buckets[key] = userBucket
		}
		now := time.Now()
		elapsed := now.Sub(userBucket.lastRefill).Seconds()
		userBucket.tokens += elapsed * rl.rate
		if userBucket.tokens > float64(rl.burst) {
			userBucket.tokens = float64(rl.burst)
		}
		userBucket.lastRefill = now
		userBucket.tokens -= 1
		allowed := userBucket.tokens >= 0
		rl.mu.Unlock()

		if !allowed {
			w.Header().Set("Retry-After", fmt.Sprintf("%.0f", math.Ceil(1.0/rl.rate)))
			http.Error(w, "too many requests, no more tokens", http.StatusTooManyRequests)
		} else {
			next(w, r)
		}
	}
}

The important part here sits between the rl.mu.Lock() and rl.mu.Unlock() lines. A very simple algorithm that could be explained thusly:

1. lock the bucket
2. check if we have the bucket by its key; if not, create it
3. add tokens based on the rate and the time since last refill
4. if the new amount is bigger than the burst amount, set it to burst
5. set last refill to current time
6. consume token
7. check if amount of tokens is >= 0 (to avoid race conditions, do it before unlock)
8. unlock the bucket

It truly is that simple. I provide per-user isolation, by associating a user with a token with a bucket, and measuring by the amount of tokens that sit in the bucket, I know whether the user has tripped our rate-limits or not. And it works, because I built the tests that prove it. And one last bit that is also super important, is the setting of the Retry-After header, in case the user has gone over the limits: this allows a properly configured client to know when it can try again, which is an extremely important information.

The REST API aka the doors that lock#

I previously spent a lot of time building walls and guardrails and golden paths. But there was something else that was needed. A door might be a good analogy, if I add to it the fact that it has built-in protections about what can come in, as this is where the authentication and rate-limiting handlers I built before.

Let’s start by leveraging a pattern that Go has introduced in its stdlib’s http package relatively recently: the ServeMux, or the HTTP request multiplexer. This allows us to do pattern matching of incoming requests without having to pull in external frameworks or write a bunch of boilerplate to handle this exact function. Sometimes, I just love Golang! Look at how simple this has become:

func (s *Server) NewMux() *http.ServeMux {
	mux := http.NewServeMux()
	mux.Handle("/health", s.loggingMiddleware(s.healthHandler()))
	mux.Handle("GET /api/v1/tenants", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.tenantHandler.ListTenants()))))
	mux.Handle("POST /api/v1/tenants", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.tenantHandler.CreateTenant()))))
	mux.Handle("GET /api/v1/tenants/{id}", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.tenantHandler.GetTenant()))))
	mux.Handle("PUT /api/v1/tenants/{id}", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.tenantHandler.UpdateTenant()))))
	mux.Handle("DELETE /api/v1/tenants/{id}", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.tenantHandler.DeleteTenant()))))
	mux.Handle("POST /api/v1/tenants/{id}/members", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.memberHandler.AddMember()))))
	mux.Handle("DELETE /api/v1/tenants/{id}/members/{userId}", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.memberHandler.RemoveMember()))))
	mux.Handle("GET /api/v1/tenants/{id}/members", s.loggingMiddleware(s.rl.Limit(s.auth.Auth(s.memberHandler.ListMembers()))))

	return mux
}

You can see the pattern: pretty much all requests are being wired in a particular order: first the logging, then the rate-limiter, then the auth, and finally the specific handler I want. This might be a bit verbose, but it’s very clear what’s happening here. And thanks to the multiplexer functionality of the ServeMux type, I can do matching between the {id} parameter and pass it to the functions through the context. With this, I can cleanly separate the CRDs that are needed for the internal operations, from the API types that we want to expose to the admin and operators of k8s-mtp: the v1.Tenant type has a dozen or so fields, while the CreateTenantRequest type only has 3. This makes a huge difference in terms of ergonomics of operation.

The handlers for the tenants and members all follow a similar pattern: they receive a JSON request, which is decoded and validated; from that, they build the request for the action that they need to execute (say, listing members), make it persist, and then depending on the result from this process, emit a response. To look at the example mentioned just now:

func (m *MemberHandler) ListMembers() http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "application/json")
		id := r.PathValue("id")
		members, err := m.store.ListMembers(r.Context(), id)
		if err != nil {
			m.logger.Error("error getting the members", "err", err)
			http.Error(w, `{"error": "error getting the members"}`, http.StatusInternalServerError)
			return
		}
		var resp []MemberResponse
		for _, member := range members {
			resp = append(resp, MemberResponse{
				UserID:    member.UserID,
				Role:      member.Role,
				CreatedAt: member.CreatedAt.Format(time.RFC3339),
			})
		}
		w.WriteHeader(http.StatusOK)
		json.NewEncoder(w).Encode(resp)
	}
}

A CLI for login and operation#

The last bit I added was a simple CLI that allows an admin to do several things, most importantly: 1) login into Dex, and 2) operate k8s-mtp. This will be subject to enhancements later on, but for now it was kept pretty bare bones, just to guarantee that everything was wired up correctly. I do subcommand dispatch with just the flag package and switch statements, as there was no need to complicate and bring in the “big guns”. Keep it simple, stupid!

Perhaps the most important piece of code in this CLI is the handler for the login flow. By making use of term’s ReadPassword function, we end up handling sensitive input properly, and thus save the token we get from Dex in our config without having to expose the password. The rest is, once again, standard operational procedure at this point: we POST a request with the encoded password, we validate the answer, decode it, and save it:

func loginCmd(configPath string) {
	cfg, err := loadConfig(configPath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error loading config")
		os.Exit(1)
	}
	var username string
	_, _ = fmt.Print("Username: ")
	_, err = fmt.Scanln(&username)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error reading the username")
		os.Exit(1)
	}
	_, _ = fmt.Print("Password: ")
	password, err := term.ReadPassword(int(os.Stdin.Fd()))
	if err != nil {
		fmt.Fprintln(os.Stderr, "Error reading password:", err)
		os.Exit(1)
	}
	fmt.Println()

	data := url.Values{
		"grant_type": {"password"},
		"username":   {username},
		"password":   {string(password)},
		"scope":      {"openid"},
	}

	req, err := http.NewRequest("POST", cfg.DexURL+"/token", strings.NewReader(data.Encode()))
	if err != nil {
		fmt.Fprintln(os.Stderr, "Error requesting the token:", err)
		os.Exit(1)
	}
	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	req.Header.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(cfg.ClientID+":")))

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		fmt.Fprintln(os.Stderr, "Error requesting auth:", err)
		os.Exit(1)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		fmt.Fprintf(os.Stderr, "Expected status 200, instead got %d", resp.StatusCode)
		os.Exit(1)
	}

	type token struct {
		IDToken string `json:"id_token"`
	}
	var tokenResp token
	if err = json.NewDecoder(resp.Body).Decode(&tokenResp); err != nil {
		fmt.Fprintln(os.Stderr, "Error decoding the json response")
		os.Exit(1)
	}

	cfg.Token = tokenResp.IDToken
	if err = saveConfig(configPath, cfg); err != nil {
		fmt.Fprintln(os.Stderr, "Error saving config")
		os.Exit(1)
	} else {
		fmt.Fprintf(os.Stdout, "Configuration saved to %s", configPath)
	}
}

What I learned meanwhile#

Type assertions can be weird#

There was quite a few accidents along the way, some of them were unearthed right away while building, some took a bit longer to surface. One of these bugs was a fun one. I had built a very simple helper, to allow me to return the user of the request from the context. This was the helper, in its first version:

func UserFromContext(ctx context.Context) string {
	return ctx.Value(userCtxKey).(string)
}

Can you imagine what’s wrong here? Supposedly nothing, as I’m asserting that I want a string type from the context’s Value. But this was panicking, somehow. The solution? Good old user, _ := ... pattern that’s so typical of Go. Not everything is perfect in Go-land. I kid you not, this was the extent of the solution:

func UserFromContext(ctx context.Context) string {
	user, _ := ctx.Value(userCtxKey).(string)
	return user
}

Pick your battles#

As you can see, this turned out to be a huge sprint of new stuff added (and a huge blog post, to boot). And it’s not like I spent time building unnecessary parts of this project. But in truth I ended up cramming three different beasts (i.e., infrastructure, tests, and new functionalities) inside a single sprint. By now, I should’ve known better that this is not the way to go, and I feel a bit burned out but having spent so much time building a so little time sharing, as this was part of the objective when I started the project. And this was without having rolled my own authentication provider! Something to think about for the future.

But, all in all, I can’t complain. What I achieved here was significant. The level of the project is improving with each commit, and I can only feel hopeful that what comes next will be even better! After all, we do start from a much better place than before, with the improvements that were made to the infra and tests. It was costly, but there was a very good reason for it.

What’s next#

Thank you for going on this trip with me. The next stop is: Observability! There will be metrics, dashboards, audit logging, and maybe cost allocation? All aboard the k8s-mtp train!