# Argo CD Offline Bootstrap Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan. **Goal:** Make Argo CD on `192.168.1.173` restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades. **Architecture:** Vendor the exact upstream Argo CD `v3.4.4` install manifest in `mixingx-deploy`, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to `IfNotPresent`. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures. **Tech Stack:** Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules. **Global constraints:** Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub `.zcode/plans/` files. Stop before applying if any required image is absent from the local k3s image store. --- ### Task 1: Add a failing render-verification contract **Files:** - Create: `bootstrap/argocd-offline/verify-render.sh` - Create: `bootstrap/argocd-offline/tests/expected-workloads.txt` **Step 1: Define the exact supported workload/container inventory** Add the seven workload names and their expected main/init container names to `tests/expected-workloads.txt`: ```text Deployment/argocd-applicationset-controller argocd-applicationset-controller Deployment/argocd-dex-server dex copyutil Deployment/argocd-notifications-controller argocd-notifications-controller Deployment/argocd-redis redis secret-init Deployment/argocd-repo-server argocd-repo-server copyutil Deployment/argocd-server argocd-server StatefulSet/argocd-application-controller argocd-application-controller ``` **Step 2: Write the render verifier before the overlay exists** Implement `verify-render.sh ` with `set -euo pipefail`. It must fail unless all of these are true: - The render contains only the expected Harbor image references for Argo CD, Dex, and Redis. - No upstream `quay.io`, `ghcr.io`, or `public.ecr.aws` image reference remains in a workload. - No `imagePullPolicy: Always` remains. - Each inventory entry is present and each named container/init container has an explicit `imagePullPolicy: IfNotPresent` in its workload block. - The expected images are exactly: ```text 192.168.1.119:8888/argoproj/argocd:v3.4.4 192.168.1.119:8888/dexidp/dex:v2.45.0 192.168.1.119:8888/library/redis:8.2.3-alpine ``` Keep this verifier dependency-free: Bash plus standard `awk`, `grep`, and `sed` only. **Step 3: Run it and confirm the contract fails** Run: ```bash bootstrap/argocd-offline/verify-render.sh /dev/null ``` Expected: non-zero exit with a clear missing-workload or missing-image error. **Step 4: Commit the verification contract** ```bash git add bootstrap/argocd-offline/verify-render.sh \ bootstrap/argocd-offline/tests/expected-workloads.txt git commit -m "test(gitops): define argocd offline render contract" ``` ### Task 2: Vendor Argo CD and build the offline Kustomize overlay **Files:** - Create: `bootstrap/argocd-offline/vendor/install-v3.4.4.yaml` - Create: `bootstrap/argocd-offline/kustomization.yaml` - Create: `bootstrap/argocd-offline/patches/application-controller.yaml` - Create: `bootstrap/argocd-offline/patches/applicationset-controller.yaml` - Create: `bootstrap/argocd-offline/patches/dex-server.yaml` - Create: `bootstrap/argocd-offline/patches/notifications-controller.yaml` - Create: `bootstrap/argocd-offline/patches/redis.yaml` - Create: `bootstrap/argocd-offline/patches/repo-server.yaml` - Create: `bootstrap/argocd-offline/patches/server.yaml` **Step 1: Vendor the exact upstream release** Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff: ```bash curl -fsSL \ https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \ -o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml git diff --stat ``` Expected: a single vendored upstream manifest, with no generated or unrelated files. **Step 2: Add the Kustomize image rewrites** Create `kustomization.yaml` with the vendored manifest as its only resource and these exact replacements: ```yaml images: - name: quay.io/argoproj/argocd newName: 192.168.1.119:8888/argoproj/argocd newTag: v3.4.4 - name: ghcr.io/dexidp/dex newName: 192.168.1.119:8888/dexidp/dex newTag: v2.45.0 - name: public.ecr.aws/docker/library/redis newName: 192.168.1.119:8888/library/redis newTag: 8.2.3-alpine ``` Reference all seven strategic-merge patches from the same file. **Step 3: Patch every main and init container explicitly** Each patch must target one exact Deployment or StatefulSet in namespace `argocd`. Include the expected container names from Task 1 and set only: ```yaml imagePullPolicy: IfNotPresent ``` Do not alter replicas, commands, probes, security contexts, volumes, or application resources. **Step 4: Render and make the verification contract pass** ```bash tmp="$(mktemp)" kubectl kustomize bootstrap/argocd-offline > "$tmp" bootstrap/argocd-offline/verify-render.sh "$tmp" rm -f "$tmp" ``` Expected: verifier exits zero. Also inspect the render directly: ```bash kubectl kustomize bootstrap/argocd-offline \ | grep -E 'image:|imagePullPolicy:' \ | sort -u ``` Expected: only the three Harbor images and `imagePullPolicy: IfNotPresent`; no `Always`. **Step 5: Commit the vendored source and overlay** ```bash git add bootstrap/argocd-offline git commit -m "feat(gitops): add offline argocd bootstrap overlay" ``` ### Task 3: Add the safe, idempotent server apply workflow **Files:** - Create: `bootstrap/argocd-offline/apply.sh` - Create: `bootstrap/argocd-offline/README.md` - Modify: `bootstrap/argocd-offline/verify-render.sh` **Step 1: Add a testable cache-check mode** Extend the verifier or add a private function in `apply.sh` that compares the three exact expected references with: ```bash k3s ctr images list -q ``` The comparison must be exact, print every missing image, and return non-zero before any `kubectl apply`. It must not contact a registry. **Step 2: Implement `apply.sh`** The script must: 1. Use `set -euo pipefail` and resolve all paths relative to its own directory. 2. Require UID 0, `k3s`, and `k3s kubectl kustomize`. 3. Render into a temporary file and remove it with `trap`. 4. Run `verify-render.sh` against that file. 5. Verify all three images exist in k3s containerd. 6. Snapshot current Argo CD Application names to a temporary file. 7. Apply only the rendered file with `k3s kubectl apply -f`. 8. Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1. 9. Fail if any Argo CD pod reports `ErrImagePull` or `ImagePullBackOff`. 10. Confirm the pre-existing Application names are still present, then print pod state. Use a bounded rollout timeout such as `300s`; do not loop indefinitely. **Step 3: Document the only supported install/upgrade path** README must include: - Purpose and offline limitations: only exact cached tags can start without Harbor. - Pinned upstream URL and SHA-256. - Required cached image list. - Install/recovery command: `sudo ./apply.sh`. - Verification commands for pods, pull policies, and Applications. - Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply. - Warning not to run `kubectl apply` directly against upstream `install.yaml`. **Step 4: Test failure paths without changing the cluster** Run locally: ```bash bootstrap/argocd-offline/apply.sh ``` Expected: non-root or missing-k3s failure before apply. Run the render verification again and shell syntax checks: ```bash bash -n bootstrap/argocd-offline/apply.sh bash -n bootstrap/argocd-offline/verify-render.sh tmp="$(mktemp)" kubectl kustomize bootstrap/argocd-offline > "$tmp" bootstrap/argocd-offline/verify-render.sh "$tmp" rm -f "$tmp" ``` Expected: all syntax/render checks pass. **Step 5: Commit the operator workflow** ```bash git add bootstrap/argocd-offline/apply.sh \ bootstrap/argocd-offline/verify-render.sh \ bootstrap/argocd-offline/README.md git commit -m "feat(gitops): add safe argocd offline apply workflow" ``` ### Task 4: Publish the deploy source of truth and Hub pointer **Files:** - Modify: Hub submodule pointer `mixingx-deploy` **Step 1: Rebase and re-run verification before publishing deploy** ```bash git fetch origin git rebase origin/master bash -n bootstrap/argocd-offline/apply.sh bash -n bootstrap/argocd-offline/verify-render.sh tmp="$(mktemp)" kubectl kustomize bootstrap/argocd-offline > "$tmp" bootstrap/argocd-offline/verify-render.sh "$tmp" rm -f "$tmp" git status --short ``` Expected: clean deploy worktree and all verification passes. **Step 2: Push deploy master** ```bash git push origin master ``` Expected: remote `master` advances to the verified offline-bootstrap commits. **Step 3: Update only the Hub submodule pointer** From the Hub root: ```bash git fetch origin git rebase origin/develop/v.1.0.1 git add mixingx-deploy git diff --cached --submodule=log ``` Expected: the staged diff contains only the `mixingx-deploy` gitlink. The existing untracked `.zcode/plans/` files remain unstaged. **Step 4: Commit and push the Hub pointer** ```bash git commit -m "chore(gitops): update deploy offline argocd bootstrap" git push origin develop/v.1.0.1 ``` ### Task 5: Recover and verify Argo CD on `192.168.1.173` **Files:** - No repository changes expected. **Step 1: Capture the pre-change cluster inventory** Over SSH, save or print: ```bash sudo k3s kubectl -n argocd get applications.argoproj.io -o name sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide sudo k3s ctr images list -q ``` Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying. **Step 2: Transfer the committed bootstrap directory** From the Hub machine: ```bash scp -r mixingx-deploy/bootstrap/argocd-offline \ mx@192.168.1.173:/tmp/argocd-offline ``` Use the committed directory that was just pushed; do not assemble a manifest manually on the server. **Step 3: Apply through the checked entry point** ```bash ssh mx@192.168.1.173 \ 'sudo /tmp/argocd-offline/apply.sh' ``` Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images. **Step 4: Verify the permanent live state** Run: ```bash sudo k3s kubectl -n argocd get pods sudo k3s kubectl -n argocd get deploy,statefulset \ -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}' sudo k3s kubectl -n argocd get applications.argoproj.io -o name ``` Expected: - Every Argo CD pod is Ready. - Every listed container and init container reports `IfNotPresent`. - No pod is in `ErrImagePull` or `ImagePullBackOff`. - The Application inventory matches the pre-change snapshot. **Step 5: Prove reconciliation and reboot persistence without widening scope** Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification. ### Task 6: Final repository and remote verification **Files:** - No changes expected. **Step 1: Verify branch alignment and clean deploy state** ```bash git -C mixingx-deploy status --short git -C mixingx-deploy rev-parse HEAD git -C mixingx-deploy rev-parse origin/master git status --short git rev-parse HEAD git rev-parse origin/develop/v.1.0.1 git submodule status mixingx-deploy ``` Expected: - Deploy `HEAD` equals `origin/master`. - Hub `HEAD` equals `origin/develop/v.1.0.1`. - Hub points at the same deploy commit. - Only the user's pre-existing untracked `.zcode/plans/` files remain; no implementation files are unstaged. **Step 2: Record the handoff facts** Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.