GitOps |

Stage II: CI/CD Pipeline, GitOps & Ephemeral Secrets

Sat, 16 May 2026 00:00:00 +0000

With a highly available K3s foundation established on my KVM environment in Stage I, the cluster was functional but not yet “enterprise-grade.” Manually applying manifests leads to configuration drift, and standard Kubernetes Secrets violate zero-trust security principles.

This post covers Stage II of the roadmap: establishing a strict CI/CD GitOps pipeline and securing the cluster’s sensitive credentials for my stateful workloads (specifically my n8n workflow engine and PostgreSQL database).

1. Continuous Integration with GitHub Actions

Before deployment can happen, the code must be built into an immutable artifact. I utilized GitHub Actions to automatically build my application code into Docker images and push them to the container registry on every main branch commit. This ensures that if a deployment breaks, I can instantly roll back to a previously tagged, working image.

Here is a view of the automated pipeline successfully building and pushing the containerized application upon a new commit:

2. The GitOps Philosophy: ArgoCD & Helm

The core rule of GitOps is simple: The Git repository is the single source of truth. I deployed ArgoCD to operate as a continuous reconciliation loop, tracking my GitHub repository. Rather than syncing raw YAML, ArgoCD dynamically renders my deployments using Helm and Kustomize before applying them to the K3s cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: whatsapp-bot-sync
 namespace: argocd
spec:
 project: default
 source:
 repoURL: 'https://github.com/danack10/k3s-whatsapp-chatbot.git'
 targetRevision: HEAD
 path: apps/whatsapp-bot/base
 destination:
 server: 'https://kubernetes.default.svc'
 namespace: whatsapp-bot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=True

Once applied, the GitOps synchronization kicks in. The following dashboard view shows ArgoCD mapping and deploying the entire application stack:

The Challenge: Setting up secure authentication for ArgoCD to pull from my private repository was tricky. I wanted to use a dynamic GitHub App rather than a static Personal Access Token (PAT). Ensuring the GitHub App had the correct permissions scoped strictly to the k3s-whatsapp-chatbot repo took some trial and error. Additionally, before my edge ingress was fully established, I had to utilize kubectl port-forward routed entirely through my Tailscale mesh to access the ArgoCD UI, keeping the interface completely hidden from the public internet.

3. The Problem with K8s Secrets

Deploying standard applications via GitOps is straightforward, but deploying secrets presents a major security flaw. You cannot push raw or base64-encoded secrets into a Git repository.

To solve this, I deployed HashiCorp Vault as the centralized, encrypted credential store inside the cluster.

4. Declarative Security with Vault & OpenTofu

Many engineers deploy Vault and then manually configure it using CLI commands (vault write auth/kubernetes...). To maintain my declarative infrastructure standards, I bypassed the CLI entirely and used the OpenTofu Vault Provider to configure Vault’s state as code. This allowed me to declaratively define my Kubernetes authentication roles, policies, and secret engines.

The Challenge: Learning to manage persistent state in Kubernetes was a huge learning curve. Stateful workloads like Vault, n8n, and PostgreSQL require robust Persistent Volume Claims (PVCs). At one point, my vault-0 pod entered a failure state. The “junior developer” instinct is to simply delete the PVC, tear the Vault down, and rebuild it from scratch. However, in an enterprise environment with thousands of production secrets, deleting the database is catastrophic! Instead of wiping it, I architected a proper recovery process: safely restarting the pod, maintaining the PVC integrity, and gracefully unsealing the vault using my Shamir key shares.

5. Bridging the Gap: External Secrets Operator (ESO)

While Vault securely stores the credentials, my stateful workloads (n8n and Postgres) still need a way to read them without hardcoding Vault API logic into the application containers.

I implemented the External Secrets Operator (ESO) to act as an automated courier. It authenticates with Vault, reads the secure payload, and dynamically injects it into a standard native Kubernetes Secret.

Here we define two distinct ExternalSecret manifests in a single multi-document YAML file. The first manifest pulls static database credentials, while the second pulls an ephemeral GitHub token:

---
# 1. Manifest for static n8n database credentials
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: n8n-creds-sync
 namespace: whatsapp-bot
spec:
 refreshInterval: "1h"
 secretStoreRef:
 name: vault-backend-global
 kind: ClusterSecretStore
 target:
 name: bot-secrets
 data:
 - secretKey: DB_POSTGRESDB_USER
 remoteRef:
 key: n8n/config
 property: POSTGRES_USER

 - secretKey: DB_POSTGRESDB_PASSWORD
 remoteRef:
 key: n8n/config
 property: POSTGRES_PASSWORD

 - secretKey: N8N_ENCRYPTION_KEY
 remoteRef:
 key: n8n/config
 property: ENCRYPTION_KEY

---
# 2. Manifest for dynamic GitHub Token
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: github-token-sync
 namespace: whatsapp-bot
spec:
 refreshInterval: "45m"
 secretStoreRef:
 name: vault-github-backend
 kind: SecretStore
 target:
 name: github-auth-secret
 data:
 - secretKey: GITHUB_TOKEN
 remoteRef:
 key: github-app/token/whatsapp-bot
 property: token

Because this ExternalSecret manifest contains no actual passwords (only reference pointers to Vault), it is perfectly safe to commit to GitHub and deploy via ArgoCD. The n8n and Postgres pods then consume pg-secret-live natively.

Here is the verification from the cluster showing the ExternalSecret successfully synchronized with Vault, proving the ephemeral credential pipeline is fully functional:

6. Stage II Outcomes

By the end of Phase 6, the cluster achieved a true DevSecOps deployment pipeline:

Zero Manual Intervention: Code pushed to Git is built by Actions, and automatically synchronized by ArgoCD.
Persistent & Stateful: n8n, PostgreSQL, and Vault are securely backed by PVCs.
Zero-Trust Credentials: No sensitive data exists in the repository. Configuration is handled declaratively via OpenTofu, and credentials are injected ephemerally by Vault and ESO.

With the deployment engine automated and secured, the cluster is ready to be exposed to the internet.

Next up: In , I cover how I secured the edge perimeter using Cloudflare Tunnels and established full-stack observability with Prometheus and Grafana.

Cloud Native Architecture: Zero-Trust Bare Metal Kubernetes

Wed, 28 Jan 2026 00:00:00 +0000

An ongoing, 10-phase infrastructure build demonstrating modern DevSecOps principles. This active laboratory project is transforming bare-metal hardware into a highly available, self-healing, and secure Kubernetes environment using GitOps methodologies.

Overview

I wanted to move beyond simple cloud provider tutorials and understand how the underlying compute layers actually work. I engineered this cluster to enforce zero-trust security and eliminate manual configuration drift, proving that enterprise-grade automation can be built from bare metal using virtualization, secure tunneling, and GitOps.

Infrastructure Capabilities

1. Virtualized Compute & Zero-Trust Access

KVM Hypervisor - Ubuntu guest virtual machine running efficiently on a bare-metal Kali Linux host.
Tailscale Mesh VPN - Hardened, zero-trust SSH access tunnel completely isolating the host from the public internet.
Declarative Provisioning - Utilizing OpenTofu to dynamically provision and manage the infrastructure state.

2. Container Orchestration & Networking

K3s Orchestration - Lightweight, highly available Kubernetes deployment.
Dynamic Ingress - Traefik configured as the primary ingress controller to manage robust routing and load balancing.

3. CI/CD & GitOps

Continuous Integration - GitHub Actions automates the building and pushing of Docker images for immutable rollbacks.
ArgoCD Synchronization - Cluster state is bound directly to the Git repository using Helm and Kustomize.
Zero Manual Drift - ArgoCD automatically detects and overwrites any manual changes, enforcing strict GitOps compliance.

4. Secret Management

HashiCorp Vault - Centralized, encrypted storage for all sensitive credentials and API keys.
External Secrets Operator (ESO) - Dynamically injects Vault secrets directly into Kubernetes pods.

System Architecture

┌─────────────────┐ (Builds Docker Image) ┌───────────────────────┐
│ GitHub Actions │────────────────────────────▶│ Container Registry │
│ (CI Pipeline) │ │ (GHCR / Hub) │
└─────────────────┘ └───────────────────────┘
│ │
│ (Updates Manifests) │ (Pulls Image)
▼ ▼
┌──────────────┐ ┌───────────────┐ ┌───────────────────────┐
│ │ │ │ │ Kubernetes (K3s) │
│ Git Repo │────▶│ ArgoCD │────▶│ ┌───────────────────┐ │
│ (Manifests) │ │ (Controller) │ │ │ Traefik Ingress │ │
│ │ │ │ │ └───────────────────┘ │
└──────────────┘ └───────────────┘ │ ┌───────────────────┐ │
│ │ k3s-whatsapp-bot │ │
┌──────────────┐ ┌───────────────┐ │ │ (n8n + Postgres) │ │
│ │ │ External │ │ └───────────────────┘ │
│ HashiCorp │◀────│ Secrets │◀────│ ┌───────────────────┐ │
│ Vault │ │ Operator │ │ │ ESO Injector │ │
│ │ │ │ │ └───────────────────┘ │
└──────────────┘ └───────────────┘ └───────────────────────┘
[ Infrastructure: KVM Ubuntu Guest on Kali Metal | Secured via Tailscale ]

Engineering Outcomes

🚀 Full CI/CD: 100% automated pipeline from code push (GitHub Actions) to cluster synchronization (ArgoCD).
🔒 Security: Host isolated via Tailscale; zero hardcoded secrets via Vault and ESO.
📉 Configuration Drift: Reduced to 0% through strict ArgoCD reconciliation loops.

Technical Deep Dives (Architecture Series)

To explore the raw code, YAML manifests, and how I solved specific architectural challenges across the lifecycle, read my detailed engineering write-ups:

📝 - Deep dive into Phases 1-4: Virtualizing Ubuntu on Kali via KVM, Tailscale SSH tunnels, and OpenTofu provisioning.
📝 - Deep dive into Phases 5-6: Docker builds via GitHub Actions, Helm/ArgoCD drift elimination, and Vault/ESO injection.
📝 (Coming Soon) - Deep dive into Phases 7-8: Cloudflare Tunnels and Prometheus/Grafana telemetry.
📝 (Coming Soon) - Deep dive into Phases 9-10: Executing automated attack paths against the cluster and Building a SIEM alerting pipeline.

The 10-Phase Engineering Roadmap

This cluster is designed as a living DevSecOps laboratory. I am currently executing Phase 7 of a comprehensive, capability-driven lifecycle.

Stage I: The Compute Foundation (✅ Completed)

✅ Phase 1: Base Hypervisor - Bare-metal Kali Linux hosting KVM.
✅ Phase 2: Virtualization & Access - OpenTofu provisioning of the Ubuntu guest and zero-trust Tailscale SSH tunneling.
✅ Phase 3: Container Orchestration - High-availability K3s cluster bootstrapping.
✅ Phase 4: Edge Routing - Dynamic ingress and load balancing via Traefik.

Stage II: Automation & Zero-Trust (✅ Completed)

✅ Phase 5: CI/CD Pipeline - GitHub Actions building Docker images and ArgoCD/Helm synchronizing the GitOps state.
✅ Phase 6: Ephemeral Secrets - Zero-trust credential injection via HashiCorp Vault and ESO.

Stage III: Perimeter Defense & Observability (⏳ In Progress)

⏳ Phase 7: Zero-Trust Perimeter - Integrating Cloudflare Tunnels for unexposed, secure ingress.
⏳ Phase 8: Full-Stack Telemetry - Deploying Prometheus & Grafana for cluster observability.

Stage IV: Purple Teaming Laboratory (📅 Planned)

📅 Phase 9: Offensive Simulation (Red) - Executing automated attack paths against the cluster to validate resilience.
📅 Phase 10: Threat Detection (Blue) - Building a SIEM alerting pipeline to capture the offensive testing telemetry.