Files
mixingx.deploy/docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md
T

13 KiB

Argo CD Offline Bootstrap Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan.

Goal: Make Argo CD on 192.168.1.173 restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades.

Architecture: Vendor the exact upstream Argo CD v3.4.4 install manifest in mixingx-deploy, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to IfNotPresent. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures.

Tech Stack: Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules.

Global constraints: Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub .zcode/plans/ files. Stop before applying if any required image is absent from the local k3s image store.


Task 1: Add a failing render-verification contract

Files:

  • Create: bootstrap/argocd-offline/verify-render.sh
  • Create: bootstrap/argocd-offline/tests/expected-workloads.txt

Step 1: Define the exact supported workload/container inventory

Add the seven workload names and their expected main/init container names to tests/expected-workloads.txt:

Deployment/argocd-applicationset-controller argocd-applicationset-controller
Deployment/argocd-dex-server dex copyutil
Deployment/argocd-notifications-controller argocd-notifications-controller
Deployment/argocd-redis redis secret-init
Deployment/argocd-repo-server argocd-repo-server copyutil
Deployment/argocd-server argocd-server
StatefulSet/argocd-application-controller argocd-application-controller

Step 2: Write the render verifier before the overlay exists

Implement verify-render.sh <rendered-manifest> with set -euo pipefail. It must fail unless all of these are true:

  • The render contains only the expected Harbor image references for Argo CD, Dex, and Redis.
  • No upstream quay.io, ghcr.io, or public.ecr.aws image reference remains in a workload.
  • No imagePullPolicy: Always remains.
  • Each inventory entry is present and each named container/init container has an explicit imagePullPolicy: IfNotPresent in its workload block.
  • The expected images are exactly:
192.168.1.119:8888/argoproj/argocd:v3.4.4
192.168.1.119:8888/dexidp/dex:v2.45.0
192.168.1.119:8888/library/redis:8.2.3-alpine

Keep this verifier dependency-free: Bash plus standard awk, grep, and sed only.

Step 3: Run it and confirm the contract fails

Run:

bootstrap/argocd-offline/verify-render.sh /dev/null

Expected: non-zero exit with a clear missing-workload or missing-image error.

Step 4: Commit the verification contract

git add bootstrap/argocd-offline/verify-render.sh \
  bootstrap/argocd-offline/tests/expected-workloads.txt
git commit -m "test(gitops): define argocd offline render contract"

Task 2: Vendor Argo CD and build the offline Kustomize overlay

Files:

  • Create: bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
  • Create: bootstrap/argocd-offline/kustomization.yaml
  • Create: bootstrap/argocd-offline/patches/application-controller.yaml
  • Create: bootstrap/argocd-offline/patches/applicationset-controller.yaml
  • Create: bootstrap/argocd-offline/patches/dex-server.yaml
  • Create: bootstrap/argocd-offline/patches/notifications-controller.yaml
  • Create: bootstrap/argocd-offline/patches/redis.yaml
  • Create: bootstrap/argocd-offline/patches/repo-server.yaml
  • Create: bootstrap/argocd-offline/patches/server.yaml

Step 1: Vendor the exact upstream release

Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff:

curl -fsSL \
  https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \
  -o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
git diff --stat

Expected: a single vendored upstream manifest, with no generated or unrelated files.

Step 2: Add the Kustomize image rewrites

Create kustomization.yaml with the vendored manifest as its only resource and these exact replacements:

images:
  - name: quay.io/argoproj/argocd
    newName: 192.168.1.119:8888/argoproj/argocd
    newTag: v3.4.4
  - name: ghcr.io/dexidp/dex
    newName: 192.168.1.119:8888/dexidp/dex
    newTag: v2.45.0
  - name: public.ecr.aws/docker/library/redis
    newName: 192.168.1.119:8888/library/redis
    newTag: 8.2.3-alpine

Reference all seven strategic-merge patches from the same file.

Step 3: Patch every main and init container explicitly

Each patch must target one exact Deployment or StatefulSet in namespace argocd. Include the expected container names from Task 1 and set only:

imagePullPolicy: IfNotPresent

Do not alter replicas, commands, probes, security contexts, volumes, or application resources.

Step 4: Render and make the verification contract pass

tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"

Expected: verifier exits zero. Also inspect the render directly:

kubectl kustomize bootstrap/argocd-offline \
  | grep -E 'image:|imagePullPolicy:' \
  | sort -u

Expected: only the three Harbor images and imagePullPolicy: IfNotPresent; no Always.

Step 5: Commit the vendored source and overlay

git add bootstrap/argocd-offline
git commit -m "feat(gitops): add offline argocd bootstrap overlay"

Task 3: Add the safe, idempotent server apply workflow

Files:

  • Create: bootstrap/argocd-offline/apply.sh
  • Create: bootstrap/argocd-offline/README.md
  • Modify: bootstrap/argocd-offline/verify-render.sh

Step 1: Add a testable cache-check mode

Extend the verifier or add a private function in apply.sh that compares the three exact expected references with:

k3s ctr images list -q

The comparison must be exact, print every missing image, and return non-zero before any kubectl apply. It must not contact a registry.

Step 2: Implement apply.sh

The script must:

  1. Use set -euo pipefail and resolve all paths relative to its own directory.
  2. Require UID 0, k3s, and k3s kubectl kustomize.
  3. Render into a temporary file and remove it with trap.
  4. Run verify-render.sh against that file.
  5. Verify all three images exist in k3s containerd.
  6. Snapshot current Argo CD Application names to a temporary file.
  7. Apply only the rendered file with k3s kubectl apply -f.
  8. Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1.
  9. Fail if any Argo CD pod reports ErrImagePull or ImagePullBackOff.
  10. Confirm the pre-existing Application names are still present, then print pod state.

Use a bounded rollout timeout such as 300s; do not loop indefinitely.

Step 3: Document the only supported install/upgrade path

README must include:

  • Purpose and offline limitations: only exact cached tags can start without Harbor.
  • Pinned upstream URL and SHA-256.
  • Required cached image list.
  • Install/recovery command: sudo ./apply.sh.
  • Verification commands for pods, pull policies, and Applications.
  • Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply.
  • Warning not to run kubectl apply directly against upstream install.yaml.

Step 4: Test failure paths without changing the cluster

Run locally:

bootstrap/argocd-offline/apply.sh

Expected: non-root or missing-k3s failure before apply.

Run the render verification again and shell syntax checks:

bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"

Expected: all syntax/render checks pass.

Step 5: Commit the operator workflow

git add bootstrap/argocd-offline/apply.sh \
  bootstrap/argocd-offline/verify-render.sh \
  bootstrap/argocd-offline/README.md
git commit -m "feat(gitops): add safe argocd offline apply workflow"

Task 4: Publish the deploy source of truth and Hub pointer

Files:

  • Modify: Hub submodule pointer mixingx-deploy

Step 1: Rebase and re-run verification before publishing deploy

git fetch origin
git rebase origin/master
bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
git status --short

Expected: clean deploy worktree and all verification passes.

Step 2: Push deploy master

git push origin master

Expected: remote master advances to the verified offline-bootstrap commits.

Step 3: Update only the Hub submodule pointer

From the Hub root:

git fetch origin
git rebase origin/develop/v.1.0.1
git add mixingx-deploy
git diff --cached --submodule=log

Expected: the staged diff contains only the mixingx-deploy gitlink. The existing untracked .zcode/plans/ files remain unstaged.

Step 4: Commit and push the Hub pointer

git commit -m "chore(gitops): update deploy offline argocd bootstrap"
git push origin develop/v.1.0.1

Task 5: Recover and verify Argo CD on 192.168.1.173

Files:

  • No repository changes expected.

Step 1: Capture the pre-change cluster inventory

Over SSH, save or print:

sudo k3s kubectl -n argocd get applications.argoproj.io -o name
sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide
sudo k3s ctr images list -q

Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying.

Step 2: Transfer the committed bootstrap directory

From the Hub machine:

scp -r mixingx-deploy/bootstrap/argocd-offline \
  mx@192.168.1.173:/tmp/argocd-offline

Use the committed directory that was just pushed; do not assemble a manifest manually on the server.

Step 3: Apply through the checked entry point

ssh mx@192.168.1.173 \
  'sudo /tmp/argocd-offline/apply.sh'

Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images.

Step 4: Verify the permanent live state

Run:

sudo k3s kubectl -n argocd get pods
sudo k3s kubectl -n argocd get deploy,statefulset \
  -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]}  init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]}  container:{.name}={.imagePullPolicy}{"\n"}{end}{end}'
sudo k3s kubectl -n argocd get applications.argoproj.io -o name

Expected:

  • Every Argo CD pod is Ready.
  • Every listed container and init container reports IfNotPresent.
  • No pod is in ErrImagePull or ImagePullBackOff.
  • The Application inventory matches the pre-change snapshot.

Step 5: Prove reconciliation and reboot persistence without widening scope

Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification.

Task 6: Final repository and remote verification

Files:

  • No changes expected.

Step 1: Verify branch alignment and clean deploy state

git -C mixingx-deploy status --short
git -C mixingx-deploy rev-parse HEAD
git -C mixingx-deploy rev-parse origin/master
git status --short
git rev-parse HEAD
git rev-parse origin/develop/v.1.0.1
git submodule status mixingx-deploy

Expected:

  • Deploy HEAD equals origin/master.
  • Hub HEAD equals origin/develop/v.1.0.1.
  • Hub points at the same deploy commit.
  • Only the user's pre-existing untracked .zcode/plans/ files remain; no implementation files are unstaged.

Step 2: Record the handoff facts

Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.