<link>https://example.com/</link></image><item><title>Stage II: CI/CD Pipeline, GitOps & Ephemeral Secrets</h1> <article> <h1>Stage II: CI/CD Pipeline, GitOps & Ephemeral Secrets</h1> <p>Sat, 16 May 2026 00:00:00 +0000</p> <p>With a highly available K3s foundation established on my KVM environment in Stage I, the cluster was functional but not yet “enterprise-grade.” Manually applying manifests leads to configuration drift, and standard Kubernetes Secrets violate zero-trust security principles.</p> <p>This post covers <strong>Stage II</strong> of the roadmap: establishing a strict CI/CD GitOps pipeline and securing the cluster’s sensitive credentials for my stateful workloads (specifically my n8n workflow engine and PostgreSQL database).</p> <h2 id="table-of-contents">Table of Contents</h2> <ol> <li> </li> <li> </li> <li> </li> <li> </li> <li> </li> <li> </li> </ol> <h2 id="github-actions">1. Continuous Integration with GitHub Actions</h2> <p>Before deployment can happen, the code must be built into an immutable artifact. I utilized <strong>GitHub Actions</strong> to automatically build my application code into Docker images and push them to the container registry on every main branch commit. This ensures that if a deployment breaks, I can instantly roll back to a previously tagged, working image.</p> <p>Here is a view of the automated pipeline successfully building and pushing the containerized application upon a new commit:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" > </div> </div></figure> </p> <h2 id="argocd">2. The GitOps Philosophy: ArgoCD & Helm</h2> <p>The core rule of GitOps is simple: <strong>The Git repository is the single source of truth.</strong> I deployed <strong>ArgoCD</strong> to operate as a continuous reconciliation loop, tracking my GitHub repository. Rather than syncing raw YAML, ArgoCD dynamically renders my deployments using <strong>Helm</strong> and <strong>Kustomize</strong> before applying them to the K3s cluster.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">argoproj.io/v1alpha1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Application</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">whatsapp-bot-sync</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">argocd</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">project</span><span class="p">:</span><span class="w"> </span><span class="l">default</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">repoURL</span><span class="p">:</span><span class="w"> </span><span class="s1">'https://github.com/danack10/k3s-whatsapp-chatbot.git'</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetRevision</span><span class="p">:</span><span class="w"> </span><span class="l">HEAD</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">apps/whatsapp-bot/base</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">destination</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">server</span><span class="p">:</span><span class="w"> </span><span class="s1">'https://kubernetes.default.svc'</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">whatsapp-bot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">syncPolicy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">automated</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">prune</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selfHeal</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">syncOptions</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">CreateNamespace=True</span><span class="w"> </span></span></span></code></pre></div><p>Once applied, the GitOps synchronization kicks in. The following dashboard view shows ArgoCD mapping and deploying the entire application stack:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" > </div> </div></figure> </p> <p><strong>The Challenge:</strong> Setting up secure authentication for ArgoCD to pull from my private repository was tricky. I wanted to use a dynamic GitHub App rather than a static Personal Access Token (PAT). Ensuring the GitHub App had the correct permissions scoped strictly to the <code>k3s-whatsapp-chatbot</code> repo took some trial and error. Additionally, before my edge ingress was fully established, I had to utilize <code>kubectl port-forward</code> routed entirely through my Tailscale mesh to access the ArgoCD UI, keeping the interface completely hidden from the public internet.</p> <h2 id="secrets-problem">3. The Problem with K8s Secrets</h2> <p>Deploying standard applications via GitOps is straightforward, but deploying secrets presents a major security flaw. You cannot push raw or base64-encoded secrets into a Git repository.</p> <p>To solve this, I deployed <strong>HashiCorp Vault</strong> as the centralized, encrypted credential store inside the cluster.</p> <h2 id="vault">4. Declarative Security with Vault & OpenTofu</h2> <p>Many engineers deploy Vault and then manually configure it using CLI commands (<code>vault write auth/kubernetes...</code>). To maintain my declarative infrastructure standards, I bypassed the CLI entirely and used the <strong>OpenTofu Vault Provider</strong> to configure Vault’s state as code. This allowed me to declaratively define my Kubernetes authentication roles, policies, and secret engines.</p> <p><strong>The Challenge:</strong> Learning to manage persistent state in Kubernetes was a huge learning curve. Stateful workloads like Vault, n8n, and PostgreSQL require robust Persistent Volume Claims (PVCs). At one point, my <code>vault-0</code> pod entered a failure state. The “junior developer” instinct is to simply delete the PVC, tear the Vault down, and rebuild it from scratch. However, in an enterprise environment with thousands of production secrets, deleting the database is catastrophic! Instead of wiping it, I architected a proper recovery process: safely restarting the pod, maintaining the PVC integrity, and gracefully unsealing the vault using my Shamir key shares.</p> <h2 id="eso">5. Bridging the Gap: External Secrets Operator (ESO)</h2> <p>While Vault securely stores the credentials, my stateful workloads (n8n and Postgres) still need a way to read them without hardcoding Vault API logic into the application containers.</p> <p>I implemented the <strong>External Secrets Operator (ESO)</strong> to act as an automated courier. It authenticates with Vault, reads the secure payload, and dynamically injects it into a standard native Kubernetes Secret.</p> <p>Here we define two distinct <code>ExternalSecret</code> manifests in a single multi-document YAML file. The first manifest pulls static database credentials, while the second pulls an ephemeral GitHub token:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="c"># 1. Manifest for static n8n database credentials</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">external-secrets.io/v1beta1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ExternalSecret</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">n8n-creds-sync</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">whatsapp-bot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">refreshInterval</span><span class="p">:</span><span class="w"> </span><span class="s2">"1h"</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">secretStoreRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">vault-backend-global</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterSecretStore</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">target</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">bot-secrets</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">data</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">secretKey</span><span class="p">:</span><span class="w"> </span><span class="l">DB_POSTGRESDB_USER</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">remoteRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">n8n/config</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">property</span><span class="p">:</span><span class="w"> </span><span class="l">POSTGRES_USER</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">secretKey</span><span class="p">:</span><span class="w"> </span><span class="l">DB_POSTGRESDB_PASSWORD</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">remoteRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">n8n/config</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">property</span><span class="p">:</span><span class="w"> </span><span class="l">POSTGRES_PASSWORD</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">secretKey</span><span class="p">:</span><span class="w"> </span><span class="l">N8N_ENCRYPTION_KEY</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">remoteRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">n8n/config</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">property</span><span class="p">:</span><span class="w"> </span><span class="l">ENCRYPTION_KEY</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="c"># 2. Manifest for dynamic GitHub Token</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">external-secrets.io/v1beta1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ExternalSecret</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">github-token-sync</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">whatsapp-bot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">refreshInterval</span><span class="p">:</span><span class="w"> </span><span class="s2">"45m"</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">secretStoreRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">vault-github-backend</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">SecretStore</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">target</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">github-auth-secret</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">data</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">secretKey</span><span class="p">:</span><span class="w"> </span><span class="l">GITHUB_TOKEN</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">remoteRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">github-app/token/whatsapp-bot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">property</span><span class="p">:</span><span class="w"> </span><span class="l">token</span><span class="w"> </span></span></span></code></pre></div><p>Because this <code>ExternalSecret</code> manifest contains no actual passwords (only reference pointers to Vault), it is perfectly safe to commit to GitHub and deploy via ArgoCD. The n8n and Postgres pods then consume <code>pg-secret-live</code> natively.</p> <p>Here is the verification from the cluster showing the ExternalSecret successfully synchronized with Vault, proving the ephemeral credential pipeline is fully functional:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" > </div> </div></figure> </p> <h2 id="outcomes">6. Stage II Outcomes</h2> <p>By the end of Phase 6, the cluster achieved a true DevSecOps deployment pipeline:</p> <ul> <li><strong>Zero Manual Intervention:</strong> Code pushed to Git is built by Actions, and automatically synchronized by ArgoCD.</li> <li><strong>Persistent & Stateful:</strong> n8n, PostgreSQL, and Vault are securely backed by PVCs.</li> <li><strong>Zero-Trust Credentials:</strong> No sensitive data exists in the repository. Configuration is handled declaratively via OpenTofu, and credentials are injected ephemerally by Vault and ESO.</li> </ul> <p>With the deployment engine automated and secured, the cluster is ready to be exposed to the internet.</p> <p><strong>Next up:</strong> In , I cover how I secured the edge perimeter using <strong>Cloudflare Tunnels</strong> and established full-stack observability with <strong>Prometheus and Grafana</strong>.</p> </article> <article> <h1>Stage I: Architecting a Bare-Metal KVM & K3s Foundation</h1> <p>Thu, 14 May 2026 00:00:00 +0000</p> <p>Building Kubernetes in the cloud is easy—cloud providers abstract all the hard networking and compute layers away from you. To truly understand Cloud Native architecture, I decided to build a zero-trust environment from the ground up.</p> <p>This post covers <strong>Stage I</strong> of my 10-Phase Engineering Roadmap: establishing the virtualized compute layer, deploying a zero-trust mesh VPN, bootstrapping the K3s cluster, and configuring edge routing.</p> <h2 id="table-of-contents">Table of Contents</h2> <ol> <li> </li> <li> </li> <li> </li> <li> </li> <li> </li> </ol> <h2 id="base-compute">1. The Base Compute Environment & Zero-Trust Access</h2> <p>To simulate a true enterprise environment, I am running a bare-metal <strong>Kali Linux</strong> host. However, I am not running the cluster directly on the host OS. Instead, I am using <strong>KVM (Kernel-based Virtual Machine)</strong> to spin up an isolated Ubuntu server guest.</p> <p>Furthermore, to enforce zero-trust from day one, SSH port 22 is completely blocked from the public internet. Access to the Ubuntu KVM is handled exclusively via <strong>Tailscale</strong>, creating an encrypted wireguard mesh tunnel that allows me to SSH into the root environment securely from anywhere.</p> <p>To confirm the virtualized compute layer was running smoothly, here is the output verifying the KVM guest status on the Kali host:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" ></div> </div></figure> </p> <h2 id="opentofu">2. Declarative Infrastructure with OpenTofu</h2> <p>To eliminate manual configuration drift right from the start, I used <strong>OpenTofu</strong> with the <code>libvirt</code> provider to manage the underlying state of the Ubuntu virtual machine.</p> <p>Here is a snippet of how I structured the infrastructure provisioning:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">terraform</span> { </span></span><span class="line"><span class="cl"> <span class="k">required_providers</span> { </span></span><span class="line"><span class="cl"><span class="n"> libvirt</span> <span class="o">=</span> { </span></span><span class="line"><span class="cl"><span class="n"> source</span> <span class="o">=</span> <span class="s2">"dmacvicar/libvirt"</span> </span></span><span class="line"><span class="cl"><span class="n"> version</span> <span class="o">=</span> <span class="s2">"0.7.6"</span> </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl">} </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">provider</span> <span class="s2">"libvirt"</span> { </span></span><span class="line"><span class="cl"><span class="n"> uri</span> <span class="o">=</span> <span class="s2">"qemu:///system"</span> </span></span><span class="line"><span class="cl">}<span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># Download the official Ubuntu Cloud Image </span></span></span><span class="line"><span class="cl"><span class="c1"># 1. The Base Image (Downloads and stores the raw Ubuntu image) </span></span></span><span class="line"><span class="cl"><span class="k">resource</span> <span class="s2">"libvirt_volume" "ubuntu_base"</span> { </span></span><span class="line"><span class="cl"><span class="n"> name</span> <span class="o">=</span> <span class="s2">"ubuntu-base.qcow2"</span> </span></span><span class="line"><span class="cl"><span class="n"> pool</span> <span class="o">=</span> <span class="s2">"default"</span> </span></span><span class="line"><span class="cl"><span class="n"> source</span> <span class="o">=</span> <span class="s2">"${path.module}/noble-server-cloudimg-amd64.img"</span> </span></span><span class="line"><span class="cl"><span class="n"> format</span> <span class="o">=</span> <span class="s2">"qcow2"</span> </span></span><span class="line"><span class="cl">}<span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># 2. The Actual VM Drive (Clones the base and stretches it to 30GB) </span></span></span><span class="line"><span class="cl"><span class="k">resource</span> <span class="s2">"libvirt_volume" "ubuntu_image"</span> { </span></span><span class="line"><span class="cl"><span class="n"> name</span> <span class="o">=</span> <span class="s2">"ubuntu-24.04.qcow2"</span> </span></span><span class="line"><span class="cl"><span class="n"> pool</span> <span class="o">=</span> <span class="s2">"default"</span> </span></span><span class="line"><span class="cl"><span class="n"> base_volume_id</span> <span class="o">=</span> <span class="k">libvirt_volume</span><span class="p">.</span><span class="k">ubuntu_base</span><span class="p">.</span><span class="k">id</span> </span></span><span class="line"><span class="cl"><span class="n"> size</span> <span class="o">=</span> <span class="m">32212254720</span><span class="c1"> # 30GB - SSD </span></span></span><span class="line"><span class="cl"><span class="n"> format</span> <span class="o">=</span> <span class="s2">"qcow2"</span> </span></span><span class="line"><span class="cl">}<span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># Inject the cloud-init file (which now contains Tailscale) </span></span></span><span class="line"><span class="cl"><span class="k">resource</span> <span class="s2">"libvirt_cloudinit_disk" "commoninit"</span> { </span></span><span class="line"><span class="cl"><span class="n"> name</span> <span class="o">=</span> <span class="s2">"commoninit.iso"</span> </span></span><span class="line"><span class="cl"><span class="n"> user_data</span> <span class="o">=</span> <span class="k">file</span><span class="p">(</span><span class="s2">"${path.module}/cloud_init.cfg"</span><span class="p">)</span> </span></span><span class="line"><span class="cl"><span class="n"> pool</span> <span class="o">=</span> <span class="s2">"default"</span> </span></span><span class="line"><span class="cl">}<span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># Define the Virtual Machine </span></span></span><span class="line"><span class="cl"><span class="k">resource</span> <span class="s2">"libvirt_domain" "k3s_node"</span> { </span></span><span class="line"><span class="cl"><span class="n"> name</span> <span class="o">=</span> <span class="s2">"k3s-server"</span> </span></span><span class="line"><span class="cl"><span class="n"> memory</span> <span class="o">=</span> <span class="s2">"6144"</span><span class="c1"> # 6GB RAM </span></span></span><span class="line"><span class="cl"><span class="n"> vcpu</span> <span class="o">=</span> <span class="m">2</span><span class="c1"> # 2 CPU Cores </span></span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n"> cloudinit</span> <span class="o">=</span> <span class="k">libvirt_cloudinit_disk</span><span class="p">.</span><span class="k">commoninit</span><span class="p">.</span><span class="k">id</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="k">network_interface</span> { </span></span><span class="line"><span class="cl"><span class="n"> network_name</span> <span class="o">=</span> <span class="s2">"default"</span> </span></span><span class="line"><span class="cl"><span class="n"> wait_for_lease</span> <span class="o">=</span> <span class="kt">true</span> </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="k">disk</span> { </span></span><span class="line"><span class="cl"><span class="n"> volume_id</span> <span class="o">=</span> <span class="k">libvirt_volume</span><span class="p">.</span><span class="k">ubuntu_image</span><span class="p">.</span><span class="k">id</span> </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="k">console</span> { </span></span><span class="line"><span class="cl"><span class="n"> type</span> <span class="o">=</span> <span class="s2">"pty"</span> </span></span><span class="line"><span class="cl"><span class="n"> target_port</span> <span class="o">=</span> <span class="s2">"0"</span> </span></span><span class="line"><span class="cl"><span class="n"> target_type</span> <span class="o">=</span> <span class="s2">"serial"</span> </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl">} </span></span></code></pre></div><p><strong>The Challenge:</strong> The most frustrating part of this setup was dealing with libvirt socket permissions. By default, the OpenTofu provider running on the Kali host was getting <code>permission denied</code> when trying to connect to the <code>qemu:///system</code> socket. I had to properly configure the <code>libvirt</code> group policies and Polkit rules on the host to allow the declarative pipeline to automatically provision the VM without requiring interactive root prompts.</p> <h2 id="k3s-setup">3. Bootstrapping Bare-Metal K3s</h2> <p>For container orchestration, I chose <strong>K3s</strong>. It is lightweight enough to run highly efficiently on the Ubuntu guest, but fully conformant for enterprise-grade deployments.</p> <p>To install and bootstrap the cluster securely, I explicitly passed arguments to bind to my Tailscale interface and disabled the default local storage path to maintain strict control:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -sfL <span class="o">[</span>https://get.k3s.io<span class="o">](</span>https://get.k3s.io<span class="o">)</span> <span class="p">|</span> <span class="nv">INSTALL_K3S_EXEC</span><span class="o">=</span><span class="s2">"--node-ip <TAILSCALE_IP_ADDRESS> --bind-address <TAILSCALE_IP_ADDRESS> --tls-san <TAILSCALE_IP_ADDRESS>"</span> sh - </span></span></code></pre></div><p>Once the script executed, the node successfully registered itself using the secure Tailscale interface, as shown in the cluster node status:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" > </div> </div></figure> </p> <h2 id="traefik-ingress">4. Edge Routing & Traefik Challenges</h2> <p>Networking on local KVM Kubernetes is drastically different than the cloud. You don’t have AWS or GCP to automatically provision an Elastic Load Balancer (ELB) for you.</p> <p>I utilized <strong>Traefik</strong> as the primary Ingress Controller, deploying it via Helm to manage routing for my internal services.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">networking.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Ingress</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">n8n-ingress</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">whatsapp-bot </span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l">traefik</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">rules</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="l"><Magic-DNS-FROM-TAILSCALE> </span><span class="w"> </span><span class="c"># MagicDNS name</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">http</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">paths</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">pathType</span><span class="p">:</span><span class="w"> </span><span class="l">Prefix</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">backend</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">n8n-service </span><span class="w"> </span><span class="c"># n8n service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">number</span><span class="p">:</span><span class="w"> </span><span class="m">5678</span><span class="w"> </span><span class="c"># 5678 default n8n port</span><span class="w"> </span></span></span></code></pre></div><p><strong>The Fix:</strong> Originally, exposing services from an isolated KVM guest required complex iptables rules, sysctl IP forwarding, and dealing with port 80/443 conflicts on the Kali host. To completely bypass this networking headache, I utilized Tailscale MagicDNS. By binding K3s and Traefik directly to the Tailscale interface inside the VM, I created a secure overlay network. Now, Traefik routes traffic seamlessly via MagicDNS, completely bypassing the Kali host’s physical network bridge and eliminating the need for complex port-forwarding.</p> <p>With the local DNS override in place and Traefik configured, I could securely access the internal dashboard. Here is the active HTTP routing rule proving the configuration works:</p> <p> <figure > <div class="flex justify-center "> <div class="w-full" > </div> </div></figure> </p> <h2 id="outcomes">5. Stage I Outcomes</h2> <p>By the end of Phase 4, the foundation was set:</p> <ul> <li><strong>Infrastructure is codified:</strong> The Ubuntu VM environment can be reliably rebuilt using OpenTofu.</li> <li><strong>Orchestration is live:</strong> K3s is running natively and efficiently.</li> <li><strong>Access is secured:</strong> The host and guest are protected behind a Tailscale zero-trust perimeter.</li> </ul> <p>With the base compute and networking established, the cluster was ready for application workloads.</p> <p><strong>Next up:</strong> In , I dive into how I layered <strong>GitHub Actions</strong> and <strong>ArgoCD</strong> on top of this foundation to enforce GitOps compliance, and how I secured the cluster credentials using <strong>HashiCorp Vault</strong>.</p> </article> <article> <h1>Cloud Native Architecture: Zero-Trust Bare Metal Kubernetes</h1> <p>Wed, 28 Jan 2026 00:00:00 +0000</p> <p>An ongoing, 10-phase infrastructure build demonstrating modern DevSecOps principles. This active laboratory project is transforming bare-metal hardware into a highly available, self-healing, and secure Kubernetes environment using GitOps methodologies.</p> <h2 id="overview">Overview</h2> <p>I wanted to move beyond simple cloud provider tutorials and understand how the underlying compute layers actually work. I engineered this cluster to enforce zero-trust security and eliminate manual configuration drift, proving that enterprise-grade automation can be built from bare metal using virtualization, secure tunneling, and GitOps.</p> <h2 id="infrastructure-capabilities">Infrastructure Capabilities</h2> <h3 id="1-virtualized-compute--zero-trust-access">1. Virtualized Compute & Zero-Trust Access</h3> <ul> <li><strong>KVM Hypervisor</strong> - Ubuntu guest virtual machine running efficiently on a bare-metal Kali Linux host.</li> <li><strong>Tailscale Mesh VPN</strong> - Hardened, zero-trust SSH access tunnel completely isolating the host from the public internet.</li> <li><strong>Declarative Provisioning</strong> - Utilizing OpenTofu to dynamically provision and manage the infrastructure state.</li> </ul> <h3 id="2-container-orchestration--networking">2. Container Orchestration & Networking</h3> <ul> <li><strong>K3s Orchestration</strong> - Lightweight, highly available Kubernetes deployment.</li> <li><strong>Dynamic Ingress</strong> - Traefik configured as the primary ingress controller to manage robust routing and load balancing.</li> </ul> <h3 id="3-cicd--gitops">3. CI/CD & GitOps</h3> <ul> <li><strong>Continuous Integration</strong> - GitHub Actions automates the building and pushing of Docker images for immutable rollbacks.</li> <li><strong>ArgoCD Synchronization</strong> - Cluster state is bound directly to the Git repository using Helm and Kustomize.</li> <li><strong>Zero Manual Drift</strong> - ArgoCD automatically detects and overwrites any manual changes, enforcing strict GitOps compliance.</li> </ul> <h3 id="4-secret-management">4. Secret Management</h3> <ul> <li><strong>HashiCorp Vault</strong> - Centralized, encrypted storage for all sensitive credentials and API keys.</li> <li><strong>External Secrets Operator (ESO)</strong> - Dynamically injects Vault secrets directly into Kubernetes pods.</li> </ul> <h2 id="system-architecture">System Architecture</h2> <pre><code>┌─────────────────┐ (Builds Docker Image) ┌───────────────────────┐ │ GitHub Actions │────────────────────────────▶│ Container Registry │ │ (CI Pipeline) │ │ (GHCR / Hub) │ └─────────────────┘ └───────────────────────┘ │ │ │ (Updates Manifests) │ (Pulls Image) ▼ ▼ ┌──────────────┐ ┌───────────────┐ ┌───────────────────────┐ │ │ │ │ │ Kubernetes (K3s) │ │ Git Repo │────▶│ ArgoCD │────▶│ ┌───────────────────┐ │ │ (Manifests) │ │ (Controller) │ │ │ Traefik Ingress │ │ │ │ │ │ │ └───────────────────┘ │ └──────────────┘ └───────────────┘ │ ┌───────────────────┐ │ │ │ k3s-whatsapp-bot │ │ ┌──────────────┐ ┌───────────────┐ │ │ (n8n + Postgres) │ │ │ │ │ External │ │ └───────────────────┘ │ │ HashiCorp │◀────│ Secrets │◀────│ ┌───────────────────┐ │ │ Vault │ │ Operator │ │ │ ESO Injector │ │ │ │ │ │ │ └───────────────────┘ │ └──────────────┘ └───────────────┘ └───────────────────────┘ [ Infrastructure: KVM Ubuntu Guest on Kali Metal | Secured via Tailscale ] </code></pre> <h2 id="engineering-outcomes">Engineering Outcomes</h2> <ul> <li>🚀 <strong>Full CI/CD</strong>: 100% automated pipeline from code push (GitHub Actions) to cluster synchronization (ArgoCD).</li> <li>🔒 <strong>Security</strong>: Host isolated via Tailscale; zero hardcoded secrets via Vault and ESO.</li> <li>📉 <strong>Configuration Drift</strong>: Reduced to 0% through strict ArgoCD reconciliation loops.</li> </ul> <h2 id="technical-deep-dives-architecture-series">Technical Deep Dives (Architecture Series)</h2> <p>To explore the raw code, YAML manifests, and how I solved specific architectural challenges across the lifecycle, read my detailed engineering write-ups:</p> <ul> <li>📝 <strong> </strong> - <em>Deep dive into Phases 1-4: Virtualizing Ubuntu on Kali via KVM, Tailscale SSH tunnels, and OpenTofu provisioning.</em></li> <li>📝 <strong> </strong> - <em>Deep dive into Phases 5-6: Docker builds via GitHub Actions, Helm/ArgoCD drift elimination, and Vault/ESO injection.</em></li> <li>📝 <strong> </strong> (Coming Soon) - <em>Deep dive into Phases 7-8: Cloudflare Tunnels and Prometheus/Grafana telemetry.</em></li> <li>📝 <strong> </strong> (Coming Soon) - <em>Deep dive into Phases 9-10: Executing automated attack paths against the cluster and Building a SIEM alerting pipeline.</em></li> </ul> <h2 id="the-10-phase-engineering-roadmap">The 10-Phase Engineering Roadmap</h2> <p>This cluster is designed as a living DevSecOps laboratory. I am currently executing Phase 7 of a comprehensive, capability-driven lifecycle.</p> <p><strong>Stage I: The Compute Foundation (✅ Completed)</strong></p> <ul> <li>✅ <strong>Phase 1: Base Hypervisor</strong> - Bare-metal Kali Linux hosting KVM.</li> <li>✅ <strong>Phase 2: Virtualization & Access</strong> - OpenTofu provisioning of the Ubuntu guest and zero-trust Tailscale SSH tunneling.</li> <li>✅ <strong>Phase 3: Container Orchestration</strong> - High-availability K3s cluster bootstrapping.</li> <li>✅ <strong>Phase 4: Edge Routing</strong> - Dynamic ingress and load balancing via Traefik.</li> </ul> <p><strong>Stage II: Automation & Zero-Trust (✅ Completed)</strong></p> <ul> <li>✅ <strong>Phase 5: CI/CD Pipeline</strong> - GitHub Actions building Docker images and ArgoCD/Helm synchronizing the GitOps state.</li> <li>✅ <strong>Phase 6: Ephemeral Secrets</strong> - Zero-trust credential injection via HashiCorp Vault and ESO.</li> </ul> <p><strong>Stage III: Perimeter Defense & Observability (⏳ In Progress)</strong></p> <ul> <li>⏳ <strong>Phase 7: Zero-Trust Perimeter</strong> - Integrating Cloudflare Tunnels for unexposed, secure ingress.</li> <li>⏳ <strong>Phase 8: Full-Stack Telemetry</strong> - Deploying Prometheus & Grafana for cluster observability.</li> </ul> <p><strong>Stage IV: Purple Teaming Laboratory (📅 Planned)</strong></p> <ul> <li>📅 <strong>Phase 9: Offensive Simulation (Red)</strong> - Executing automated attack paths against the cluster to validate resilience.</li> <li>📅 <strong>Phase 10: Threat Detection (Blue)</strong> - Building a SIEM alerting pipeline to capture the offensive testing telemetry.</li> </ul> </article> </main></body></html>