Overview

GitOps is a word that gets thrown around a lot these days. People are talking about it a lot becuase it solves a lot of problems that have plagued the industry for a long time. Deploying to production can be an insanely complicated process for some people, requiring multiple build steps and established procedures requiring human intervention or hacky scripts somebody wrote at 11 PM who just wanted to go home for the day. GitOps allows you to syncronize your collection of kubernetes manifest files between a repository and your cluster. That way if things go south, you can destroy your cluster and rebuild in just a few minutes. Additionally you can bring over your code review, change control, and audit procedures that you use with your application development to your cluster configuration, since now changes are no different than a PR to your application’s codebase.

When I first learned about GitOps, I really wanted to try it out on my homelab. I initially went with Argo CD, simply because that’s what everyone else was using. Although Argo CD is cool and all, there was one major deal-breaker for me. Argo CD requires nearly 8GB of RAM for normal operation and a lot of CPU power, which was more than double what I had in my test k3s cluster. What’s the point of a GitOps tool if there’s no room left on the cluster to deploy applications? As a broke college student I figured there had to be a better (and cheaper) way. The solution I found was Flux CD. It’s very lightweight, and all of it’s configuration is stored in your source control, so no need to mess with any fancy PV setups in k8s.

Flux works on top of any existing k3s or k8s cluster. I’ll be using k3s for this tutorial. You’ll need about 512MB of RAM and a few milli-CPU available in your cluster. One important thing that I found is that the Flux documentation is a little confusing at times, and I often had to delete the Flux deployment and re-deploy to implement the change. This guide will attempt to get it right the first time (assuming your needs are the same as mine).

Setup

This tutorial assumes that you already have a Kubernetes cluster of some kind already up and running. If you do not have a cluster, there are several options available to you. Minikube is an excellent choice if you only have one (somewhat-powerful) machine. k3sup is a good tool if you have a few spare computers and want to quickly turn them into Kubernetes nodes. And finally there are always hosted options such as EKS and GKE, but be aware these can get pricy very quickly. (If you enjoy a challenge and have a couple free weekends, there’s also Kubernetes The Hard Way).

All the commands below should be run using a computer with a Kubeconfig file that can connect to your cluster. In my case I have a dedicated LXC container that is inside an isolated network that I have created for my k3s cluster. Start by installing the Flux CLI tool:

# Linux
curl -s https://fluxcd.io/install.sh | sudo bash
# MacOS
brew install fluxcd/tap/flux
# Windows (requires Chocolatey)
choco install flux

Next go to GitHub and create a personal access token. Make sure that you check all the boxes under repo and read:packages. Fill in the appropriate values in the script below to bootstrap Flux into your cluster.

# On Windows, replace this with 'set'.
export GITHUB_TOKEN=<gh_token> # Your GitHub personal access token
export GITHUB_USER=<gh_username> # Your GitHub username
export GITHUB_REPO=<gh_repository> # Name of an empty or non-existant repository on your GitHub account (e.g. cluster-iac)
flux bootstrap github --components-extra=image-reflector-controller,image-automation-controller --owner=$GITHUB_USER --repository=$GITHUB_REPO --branch=main --path=./clusters/prod --personal --read-write-key

Now what does this command do? First we tell Flux that we want to bootstrap a new cluster, and we want to store our Infrastucture-as-Code (IaC) on GitHub. We then list some extra components we want to install. These will be important later when we set up automatic image update scanning. We then pass some information about GitHub to Flux. Flux will automatically create $GITHUB_REPO and watch the main branch for updates. At the very end we note that the key we attached is a --read-write-key. This extra argument is critical for Flux to be able to write back to GitHub when image updates are found. Once you run this command give Flux a few minutes to deploy everything and fully initialize. In the meantime, you should see a new private repository on GitHub that has some Kubernetes manifest files in it. Congratulations, you now have a functioning GitOps setup!

Files Walkthrough

If you log in to GitHub.com, you should see your new repository. In this repository there should be some new files. What we’re interested in is the clusters/prod directory, as this is where configuration specific to your cluster will live. It’s important to know that you can have multiple Flux clusters use the same repository. This way you can re-use your manifests and configuration across all your clusters.

In this directory, you can add several kinds of files. If you add a regular Kubernetes manifest file (*.yaml), Flux will automatically apply it when you commit, and will ensure that it stays in sync (one important caveat to this is that if you manually edit a resource using kubectl, it will get overwritten the next time Flux tries to reconcile your manifest files).

Another kind of file you can add is a Kustomization file. I have a kustomization file that points to another directory in the repository.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m0s
  dependsOn:
    - name: infrastructure
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/prod
  prune: true
  validation: client

This Kustomization file will apply all the manifest/Kustomization files located in apps/prod. In turn I have a Kustomization file there that loads whatever apps I want from apps/base like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/app1
  - ../base/app2
  - ../base/app3

Each of these app directories has a Kustomization files that lists all the regular Kubernetes manifest files that are needed for that app, located in the same directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - pgcluster.yaml
  - deployment.yaml
  - secrets.yaml
  - service.yaml
  - ingress.yaml
namespace: app1

This may seem a little redundant, but this gives me the flexability to keep all my app definitions in one place. Then I can enable and disable apps by commenting out one line in apps/prod/kustomization.yaml. There is no “right” way to organize your files as long as it works for you.

Wrap-up

You are now ready to jump into the world of GitOps. Coming soon is another tutorial on how to deploy your own app, completely from scratch, using only GitHub, GitHub Actions, and Flux.