13 KiB
Argo CD Offline Bootstrap Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan.
Goal: Make Argo CD on 192.168.1.173 restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades.
Architecture: Vendor the exact upstream Argo CD v3.4.4 install manifest in mixingx-deploy, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to IfNotPresent. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures.
Tech Stack: Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules.
Global constraints: Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub .zcode/plans/ files. Stop before applying if any required image is absent from the local k3s image store.
Task 1: Add a failing render-verification contract
Files:
- Create:
bootstrap/argocd-offline/verify-render.sh - Create:
bootstrap/argocd-offline/tests/expected-workloads.txt
Step 1: Define the exact supported workload/container inventory
Add the seven workload names and their expected main/init container names to tests/expected-workloads.txt:
Deployment/argocd-applicationset-controller argocd-applicationset-controller
Deployment/argocd-dex-server dex copyutil
Deployment/argocd-notifications-controller argocd-notifications-controller
Deployment/argocd-redis redis secret-init
Deployment/argocd-repo-server argocd-repo-server copyutil
Deployment/argocd-server argocd-server
StatefulSet/argocd-application-controller argocd-application-controller
Step 2: Write the render verifier before the overlay exists
Implement verify-render.sh <rendered-manifest> with set -euo pipefail. It must fail unless all of these are true:
- The render contains only the expected Harbor image references for Argo CD, Dex, and Redis.
- No upstream
quay.io,ghcr.io, orpublic.ecr.awsimage reference remains in a workload. - No
imagePullPolicy: Alwaysremains. - Each inventory entry is present and each named container/init container has an explicit
imagePullPolicy: IfNotPresentin its workload block. - The expected images are exactly:
192.168.1.119:8888/argoproj/argocd:v3.4.4
192.168.1.119:8888/dexidp/dex:v2.45.0
192.168.1.119:8888/library/redis:8.2.3-alpine
Keep this verifier dependency-free: Bash plus standard awk, grep, and sed only.
Step 3: Run it and confirm the contract fails
Run:
bootstrap/argocd-offline/verify-render.sh /dev/null
Expected: non-zero exit with a clear missing-workload or missing-image error.
Step 4: Commit the verification contract
git add bootstrap/argocd-offline/verify-render.sh \
bootstrap/argocd-offline/tests/expected-workloads.txt
git commit -m "test(gitops): define argocd offline render contract"
Task 2: Vendor Argo CD and build the offline Kustomize overlay
Files:
- Create:
bootstrap/argocd-offline/vendor/install-v3.4.4.yaml - Create:
bootstrap/argocd-offline/kustomization.yaml - Create:
bootstrap/argocd-offline/patches/application-controller.yaml - Create:
bootstrap/argocd-offline/patches/applicationset-controller.yaml - Create:
bootstrap/argocd-offline/patches/dex-server.yaml - Create:
bootstrap/argocd-offline/patches/notifications-controller.yaml - Create:
bootstrap/argocd-offline/patches/redis.yaml - Create:
bootstrap/argocd-offline/patches/repo-server.yaml - Create:
bootstrap/argocd-offline/patches/server.yaml
Step 1: Vendor the exact upstream release
Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff:
curl -fsSL \
https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \
-o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
git diff --stat
Expected: a single vendored upstream manifest, with no generated or unrelated files.
Step 2: Add the Kustomize image rewrites
Create kustomization.yaml with the vendored manifest as its only resource and these exact replacements:
images:
- name: quay.io/argoproj/argocd
newName: 192.168.1.119:8888/argoproj/argocd
newTag: v3.4.4
- name: ghcr.io/dexidp/dex
newName: 192.168.1.119:8888/dexidp/dex
newTag: v2.45.0
- name: public.ecr.aws/docker/library/redis
newName: 192.168.1.119:8888/library/redis
newTag: 8.2.3-alpine
Reference all seven strategic-merge patches from the same file.
Step 3: Patch every main and init container explicitly
Each patch must target one exact Deployment or StatefulSet in namespace argocd. Include the expected container names from Task 1 and set only:
imagePullPolicy: IfNotPresent
Do not alter replicas, commands, probes, security contexts, volumes, or application resources.
Step 4: Render and make the verification contract pass
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
Expected: verifier exits zero. Also inspect the render directly:
kubectl kustomize bootstrap/argocd-offline \
| grep -E 'image:|imagePullPolicy:' \
| sort -u
Expected: only the three Harbor images and imagePullPolicy: IfNotPresent; no Always.
Step 5: Commit the vendored source and overlay
git add bootstrap/argocd-offline
git commit -m "feat(gitops): add offline argocd bootstrap overlay"
Task 3: Add the safe, idempotent server apply workflow
Files:
- Create:
bootstrap/argocd-offline/apply.sh - Create:
bootstrap/argocd-offline/README.md - Modify:
bootstrap/argocd-offline/verify-render.sh
Step 1: Add a testable cache-check mode
Extend the verifier or add a private function in apply.sh that compares the three exact expected references with:
k3s ctr images list -q
The comparison must be exact, print every missing image, and return non-zero before any kubectl apply. It must not contact a registry.
Step 2: Implement apply.sh
The script must:
- Use
set -euo pipefailand resolve all paths relative to its own directory. - Require UID 0,
k3s, andk3s kubectl kustomize. - Render into a temporary file and remove it with
trap. - Run
verify-render.shagainst that file. - Verify all three images exist in k3s containerd.
- Snapshot current Argo CD Application names to a temporary file.
- Apply only the rendered file with
k3s kubectl apply -f. - Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1.
- Fail if any Argo CD pod reports
ErrImagePullorImagePullBackOff. - Confirm the pre-existing Application names are still present, then print pod state.
Use a bounded rollout timeout such as 300s; do not loop indefinitely.
Step 3: Document the only supported install/upgrade path
README must include:
- Purpose and offline limitations: only exact cached tags can start without Harbor.
- Pinned upstream URL and SHA-256.
- Required cached image list.
- Install/recovery command:
sudo ./apply.sh. - Verification commands for pods, pull policies, and Applications.
- Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply.
- Warning not to run
kubectl applydirectly against upstreaminstall.yaml.
Step 4: Test failure paths without changing the cluster
Run locally:
bootstrap/argocd-offline/apply.sh
Expected: non-root or missing-k3s failure before apply.
Run the render verification again and shell syntax checks:
bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
Expected: all syntax/render checks pass.
Step 5: Commit the operator workflow
git add bootstrap/argocd-offline/apply.sh \
bootstrap/argocd-offline/verify-render.sh \
bootstrap/argocd-offline/README.md
git commit -m "feat(gitops): add safe argocd offline apply workflow"
Task 4: Publish the deploy source of truth and Hub pointer
Files:
- Modify: Hub submodule pointer
mixingx-deploy
Step 1: Rebase and re-run verification before publishing deploy
git fetch origin
git rebase origin/master
bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
git status --short
Expected: clean deploy worktree and all verification passes.
Step 2: Push deploy master
git push origin master
Expected: remote master advances to the verified offline-bootstrap commits.
Step 3: Update only the Hub submodule pointer
From the Hub root:
git fetch origin
git rebase origin/develop/v.1.0.1
git add mixingx-deploy
git diff --cached --submodule=log
Expected: the staged diff contains only the mixingx-deploy gitlink. The existing untracked .zcode/plans/ files remain unstaged.
Step 4: Commit and push the Hub pointer
git commit -m "chore(gitops): update deploy offline argocd bootstrap"
git push origin develop/v.1.0.1
Task 5: Recover and verify Argo CD on 192.168.1.173
Files:
- No repository changes expected.
Step 1: Capture the pre-change cluster inventory
Over SSH, save or print:
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide
sudo k3s ctr images list -q
Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying.
Step 2: Transfer the committed bootstrap directory
From the Hub machine:
scp -r mixingx-deploy/bootstrap/argocd-offline \
mx@192.168.1.173:/tmp/argocd-offline
Use the committed directory that was just pushed; do not assemble a manifest manually on the server.
Step 3: Apply through the checked entry point
ssh mx@192.168.1.173 \
'sudo /tmp/argocd-offline/apply.sh'
Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images.
Step 4: Verify the permanent live state
Run:
sudo k3s kubectl -n argocd get pods
sudo k3s kubectl -n argocd get deploy,statefulset \
-o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}'
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
Expected:
- Every Argo CD pod is Ready.
- Every listed container and init container reports
IfNotPresent. - No pod is in
ErrImagePullorImagePullBackOff. - The Application inventory matches the pre-change snapshot.
Step 5: Prove reconciliation and reboot persistence without widening scope
Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification.
Task 6: Final repository and remote verification
Files:
- No changes expected.
Step 1: Verify branch alignment and clean deploy state
git -C mixingx-deploy status --short
git -C mixingx-deploy rev-parse HEAD
git -C mixingx-deploy rev-parse origin/master
git status --short
git rev-parse HEAD
git rev-parse origin/develop/v.1.0.1
git submodule status mixingx-deploy
Expected:
- Deploy
HEADequalsorigin/master. - Hub
HEADequalsorigin/develop/v.1.0.1. - Hub points at the same deploy commit.
- Only the user's pre-existing untracked
.zcode/plans/files remain; no implementation files are unstaged.
Step 2: Record the handoff facts
Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.