From ff761a3e90281b1813ec054ab9fe92f426b5706a Mon Sep 17 00:00:00 2001 From: ximenchuifeng Date: Wed, 15 Jul 2026 01:11:21 +0800 Subject: [PATCH] docs(gitops): plan argocd offline bootstrap --- .../2026-07-15-argocd-offline-bootstrap.md | 365 ++++++++++++++++++ 1 file changed, 365 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md diff --git a/docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md b/docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md new file mode 100644 index 0000000..8a00ddf --- /dev/null +++ b/docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md @@ -0,0 +1,365 @@ +# Argo CD Offline Bootstrap Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan. + +**Goal:** Make Argo CD on `192.168.1.173` restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades. + +**Architecture:** Vendor the exact upstream Argo CD `v3.4.4` install manifest in `mixingx-deploy`, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to `IfNotPresent`. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures. + +**Tech Stack:** Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules. + +**Global constraints:** Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub `.zcode/plans/` files. Stop before applying if any required image is absent from the local k3s image store. + +--- + +### Task 1: Add a failing render-verification contract + +**Files:** +- Create: `bootstrap/argocd-offline/verify-render.sh` +- Create: `bootstrap/argocd-offline/tests/expected-workloads.txt` + +**Step 1: Define the exact supported workload/container inventory** + +Add the seven workload names and their expected main/init container names to `tests/expected-workloads.txt`: + +```text +Deployment/argocd-applicationset-controller argocd-applicationset-controller +Deployment/argocd-dex-server dex copyutil +Deployment/argocd-notifications-controller argocd-notifications-controller +Deployment/argocd-redis redis secret-init +Deployment/argocd-repo-server argocd-repo-server copyutil +Deployment/argocd-server argocd-server +StatefulSet/argocd-application-controller argocd-application-controller +``` + +**Step 2: Write the render verifier before the overlay exists** + +Implement `verify-render.sh ` with `set -euo pipefail`. It must fail unless all of these are true: + +- The render contains only the expected Harbor image references for Argo CD, Dex, and Redis. +- No upstream `quay.io`, `ghcr.io`, or `public.ecr.aws` image reference remains in a workload. +- No `imagePullPolicy: Always` remains. +- Each inventory entry is present and each named container/init container has an explicit `imagePullPolicy: IfNotPresent` in its workload block. +- The expected images are exactly: + +```text +192.168.1.119:8888/argoproj/argocd:v3.4.4 +192.168.1.119:8888/dexidp/dex:v2.45.0 +192.168.1.119:8888/library/redis:8.2.3-alpine +``` + +Keep this verifier dependency-free: Bash plus standard `awk`, `grep`, and `sed` only. + +**Step 3: Run it and confirm the contract fails** + +Run: + +```bash +bootstrap/argocd-offline/verify-render.sh /dev/null +``` + +Expected: non-zero exit with a clear missing-workload or missing-image error. + +**Step 4: Commit the verification contract** + +```bash +git add bootstrap/argocd-offline/verify-render.sh \ + bootstrap/argocd-offline/tests/expected-workloads.txt +git commit -m "test(gitops): define argocd offline render contract" +``` + +### Task 2: Vendor Argo CD and build the offline Kustomize overlay + +**Files:** +- Create: `bootstrap/argocd-offline/vendor/install-v3.4.4.yaml` +- Create: `bootstrap/argocd-offline/kustomization.yaml` +- Create: `bootstrap/argocd-offline/patches/application-controller.yaml` +- Create: `bootstrap/argocd-offline/patches/applicationset-controller.yaml` +- Create: `bootstrap/argocd-offline/patches/dex-server.yaml` +- Create: `bootstrap/argocd-offline/patches/notifications-controller.yaml` +- Create: `bootstrap/argocd-offline/patches/redis.yaml` +- Create: `bootstrap/argocd-offline/patches/repo-server.yaml` +- Create: `bootstrap/argocd-offline/patches/server.yaml` + +**Step 1: Vendor the exact upstream release** + +Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff: + +```bash +curl -fsSL \ + https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \ + -o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml +shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml +git diff --stat +``` + +Expected: a single vendored upstream manifest, with no generated or unrelated files. + +**Step 2: Add the Kustomize image rewrites** + +Create `kustomization.yaml` with the vendored manifest as its only resource and these exact replacements: + +```yaml +images: + - name: quay.io/argoproj/argocd + newName: 192.168.1.119:8888/argoproj/argocd + newTag: v3.4.4 + - name: ghcr.io/dexidp/dex + newName: 192.168.1.119:8888/dexidp/dex + newTag: v2.45.0 + - name: public.ecr.aws/docker/library/redis + newName: 192.168.1.119:8888/library/redis + newTag: 8.2.3-alpine +``` + +Reference all seven strategic-merge patches from the same file. + +**Step 3: Patch every main and init container explicitly** + +Each patch must target one exact Deployment or StatefulSet in namespace `argocd`. Include the expected container names from Task 1 and set only: + +```yaml +imagePullPolicy: IfNotPresent +``` + +Do not alter replicas, commands, probes, security contexts, volumes, or application resources. + +**Step 4: Render and make the verification contract pass** + +```bash +tmp="$(mktemp)" +kubectl kustomize bootstrap/argocd-offline > "$tmp" +bootstrap/argocd-offline/verify-render.sh "$tmp" +rm -f "$tmp" +``` + +Expected: verifier exits zero. Also inspect the render directly: + +```bash +kubectl kustomize bootstrap/argocd-offline \ + | grep -E 'image:|imagePullPolicy:' \ + | sort -u +``` + +Expected: only the three Harbor images and `imagePullPolicy: IfNotPresent`; no `Always`. + +**Step 5: Commit the vendored source and overlay** + +```bash +git add bootstrap/argocd-offline +git commit -m "feat(gitops): add offline argocd bootstrap overlay" +``` + +### Task 3: Add the safe, idempotent server apply workflow + +**Files:** +- Create: `bootstrap/argocd-offline/apply.sh` +- Create: `bootstrap/argocd-offline/README.md` +- Modify: `bootstrap/argocd-offline/verify-render.sh` + +**Step 1: Add a testable cache-check mode** + +Extend the verifier or add a private function in `apply.sh` that compares the three exact expected references with: + +```bash +k3s ctr images list -q +``` + +The comparison must be exact, print every missing image, and return non-zero before any `kubectl apply`. It must not contact a registry. + +**Step 2: Implement `apply.sh`** + +The script must: + +1. Use `set -euo pipefail` and resolve all paths relative to its own directory. +2. Require UID 0, `k3s`, and `k3s kubectl kustomize`. +3. Render into a temporary file and remove it with `trap`. +4. Run `verify-render.sh` against that file. +5. Verify all three images exist in k3s containerd. +6. Snapshot current Argo CD Application names to a temporary file. +7. Apply only the rendered file with `k3s kubectl apply -f`. +8. Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1. +9. Fail if any Argo CD pod reports `ErrImagePull` or `ImagePullBackOff`. +10. Confirm the pre-existing Application names are still present, then print pod state. + +Use a bounded rollout timeout such as `300s`; do not loop indefinitely. + +**Step 3: Document the only supported install/upgrade path** + +README must include: + +- Purpose and offline limitations: only exact cached tags can start without Harbor. +- Pinned upstream URL and SHA-256. +- Required cached image list. +- Install/recovery command: `sudo ./apply.sh`. +- Verification commands for pods, pull policies, and Applications. +- Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply. +- Warning not to run `kubectl apply` directly against upstream `install.yaml`. + +**Step 4: Test failure paths without changing the cluster** + +Run locally: + +```bash +bootstrap/argocd-offline/apply.sh +``` + +Expected: non-root or missing-k3s failure before apply. + +Run the render verification again and shell syntax checks: + +```bash +bash -n bootstrap/argocd-offline/apply.sh +bash -n bootstrap/argocd-offline/verify-render.sh +tmp="$(mktemp)" +kubectl kustomize bootstrap/argocd-offline > "$tmp" +bootstrap/argocd-offline/verify-render.sh "$tmp" +rm -f "$tmp" +``` + +Expected: all syntax/render checks pass. + +**Step 5: Commit the operator workflow** + +```bash +git add bootstrap/argocd-offline/apply.sh \ + bootstrap/argocd-offline/verify-render.sh \ + bootstrap/argocd-offline/README.md +git commit -m "feat(gitops): add safe argocd offline apply workflow" +``` + +### Task 4: Publish the deploy source of truth and Hub pointer + +**Files:** +- Modify: Hub submodule pointer `mixingx-deploy` + +**Step 1: Rebase and re-run verification before publishing deploy** + +```bash +git fetch origin +git rebase origin/master +bash -n bootstrap/argocd-offline/apply.sh +bash -n bootstrap/argocd-offline/verify-render.sh +tmp="$(mktemp)" +kubectl kustomize bootstrap/argocd-offline > "$tmp" +bootstrap/argocd-offline/verify-render.sh "$tmp" +rm -f "$tmp" +git status --short +``` + +Expected: clean deploy worktree and all verification passes. + +**Step 2: Push deploy master** + +```bash +git push origin master +``` + +Expected: remote `master` advances to the verified offline-bootstrap commits. + +**Step 3: Update only the Hub submodule pointer** + +From the Hub root: + +```bash +git fetch origin +git rebase origin/develop/v.1.0.1 +git add mixingx-deploy +git diff --cached --submodule=log +``` + +Expected: the staged diff contains only the `mixingx-deploy` gitlink. The existing untracked `.zcode/plans/` files remain unstaged. + +**Step 4: Commit and push the Hub pointer** + +```bash +git commit -m "chore(gitops): update deploy offline argocd bootstrap" +git push origin develop/v.1.0.1 +``` + +### Task 5: Recover and verify Argo CD on `192.168.1.173` + +**Files:** +- No repository changes expected. + +**Step 1: Capture the pre-change cluster inventory** + +Over SSH, save or print: + +```bash +sudo k3s kubectl -n argocd get applications.argoproj.io -o name +sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide +sudo k3s ctr images list -q +``` + +Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying. + +**Step 2: Transfer the committed bootstrap directory** + +From the Hub machine: + +```bash +scp -r mixingx-deploy/bootstrap/argocd-offline \ + mx@192.168.1.173:/tmp/argocd-offline +``` + +Use the committed directory that was just pushed; do not assemble a manifest manually on the server. + +**Step 3: Apply through the checked entry point** + +```bash +ssh mx@192.168.1.173 \ + 'sudo /tmp/argocd-offline/apply.sh' +``` + +Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images. + +**Step 4: Verify the permanent live state** + +Run: + +```bash +sudo k3s kubectl -n argocd get pods +sudo k3s kubectl -n argocd get deploy,statefulset \ + -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}' +sudo k3s kubectl -n argocd get applications.argoproj.io -o name +``` + +Expected: + +- Every Argo CD pod is Ready. +- Every listed container and init container reports `IfNotPresent`. +- No pod is in `ErrImagePull` or `ImagePullBackOff`. +- The Application inventory matches the pre-change snapshot. + +**Step 5: Prove reconciliation and reboot persistence without widening scope** + +Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification. + +### Task 6: Final repository and remote verification + +**Files:** +- No changes expected. + +**Step 1: Verify branch alignment and clean deploy state** + +```bash +git -C mixingx-deploy status --short +git -C mixingx-deploy rev-parse HEAD +git -C mixingx-deploy rev-parse origin/master +git status --short +git rev-parse HEAD +git rev-parse origin/develop/v.1.0.1 +git submodule status mixingx-deploy +``` + +Expected: + +- Deploy `HEAD` equals `origin/master`. +- Hub `HEAD` equals `origin/develop/v.1.0.1`. +- Hub points at the same deploy commit. +- Only the user's pre-existing untracked `.zcode/plans/` files remain; no implementation files are unstaged. + +**Step 2: Record the handoff facts** + +Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.