Files
mixingx.deploy/docs/superpowers/plans/2026-07-15-argocd-offline-bootstrap.md

366 lines
13 KiB
Markdown

# Argo CD Offline Bootstrap Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan.
**Goal:** Make Argo CD on `192.168.1.173` restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades.
**Architecture:** Vendor the exact upstream Argo CD `v3.4.4` install manifest in `mixingx-deploy`, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to `IfNotPresent`. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures.
**Tech Stack:** Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules.
**Global constraints:** Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub `.zcode/plans/` files. Stop before applying if any required image is absent from the local k3s image store.
---
### Task 1: Add a failing render-verification contract
**Files:**
- Create: `bootstrap/argocd-offline/verify-render.sh`
- Create: `bootstrap/argocd-offline/tests/expected-workloads.txt`
**Step 1: Define the exact supported workload/container inventory**
Add the seven workload names and their expected main/init container names to `tests/expected-workloads.txt`:
```text
Deployment/argocd-applicationset-controller argocd-applicationset-controller
Deployment/argocd-dex-server dex copyutil
Deployment/argocd-notifications-controller argocd-notifications-controller
Deployment/argocd-redis redis secret-init
Deployment/argocd-repo-server argocd-repo-server copyutil
Deployment/argocd-server argocd-server
StatefulSet/argocd-application-controller argocd-application-controller
```
**Step 2: Write the render verifier before the overlay exists**
Implement `verify-render.sh <rendered-manifest>` with `set -euo pipefail`. It must fail unless all of these are true:
- The render contains only the expected Harbor image references for Argo CD, Dex, and Redis.
- No upstream `quay.io`, `ghcr.io`, or `public.ecr.aws` image reference remains in a workload.
- No `imagePullPolicy: Always` remains.
- Each inventory entry is present and each named container/init container has an explicit `imagePullPolicy: IfNotPresent` in its workload block.
- The expected images are exactly:
```text
192.168.1.119:8888/argoproj/argocd:v3.4.4
192.168.1.119:8888/dexidp/dex:v2.45.0
192.168.1.119:8888/library/redis:8.2.3-alpine
```
Keep this verifier dependency-free: Bash plus standard `awk`, `grep`, and `sed` only.
**Step 3: Run it and confirm the contract fails**
Run:
```bash
bootstrap/argocd-offline/verify-render.sh /dev/null
```
Expected: non-zero exit with a clear missing-workload or missing-image error.
**Step 4: Commit the verification contract**
```bash
git add bootstrap/argocd-offline/verify-render.sh \
bootstrap/argocd-offline/tests/expected-workloads.txt
git commit -m "test(gitops): define argocd offline render contract"
```
### Task 2: Vendor Argo CD and build the offline Kustomize overlay
**Files:**
- Create: `bootstrap/argocd-offline/vendor/install-v3.4.4.yaml`
- Create: `bootstrap/argocd-offline/kustomization.yaml`
- Create: `bootstrap/argocd-offline/patches/application-controller.yaml`
- Create: `bootstrap/argocd-offline/patches/applicationset-controller.yaml`
- Create: `bootstrap/argocd-offline/patches/dex-server.yaml`
- Create: `bootstrap/argocd-offline/patches/notifications-controller.yaml`
- Create: `bootstrap/argocd-offline/patches/redis.yaml`
- Create: `bootstrap/argocd-offline/patches/repo-server.yaml`
- Create: `bootstrap/argocd-offline/patches/server.yaml`
**Step 1: Vendor the exact upstream release**
Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff:
```bash
curl -fsSL \
https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \
-o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
git diff --stat
```
Expected: a single vendored upstream manifest, with no generated or unrelated files.
**Step 2: Add the Kustomize image rewrites**
Create `kustomization.yaml` with the vendored manifest as its only resource and these exact replacements:
```yaml
images:
- name: quay.io/argoproj/argocd
newName: 192.168.1.119:8888/argoproj/argocd
newTag: v3.4.4
- name: ghcr.io/dexidp/dex
newName: 192.168.1.119:8888/dexidp/dex
newTag: v2.45.0
- name: public.ecr.aws/docker/library/redis
newName: 192.168.1.119:8888/library/redis
newTag: 8.2.3-alpine
```
Reference all seven strategic-merge patches from the same file.
**Step 3: Patch every main and init container explicitly**
Each patch must target one exact Deployment or StatefulSet in namespace `argocd`. Include the expected container names from Task 1 and set only:
```yaml
imagePullPolicy: IfNotPresent
```
Do not alter replicas, commands, probes, security contexts, volumes, or application resources.
**Step 4: Render and make the verification contract pass**
```bash
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
```
Expected: verifier exits zero. Also inspect the render directly:
```bash
kubectl kustomize bootstrap/argocd-offline \
| grep -E 'image:|imagePullPolicy:' \
| sort -u
```
Expected: only the three Harbor images and `imagePullPolicy: IfNotPresent`; no `Always`.
**Step 5: Commit the vendored source and overlay**
```bash
git add bootstrap/argocd-offline
git commit -m "feat(gitops): add offline argocd bootstrap overlay"
```
### Task 3: Add the safe, idempotent server apply workflow
**Files:**
- Create: `bootstrap/argocd-offline/apply.sh`
- Create: `bootstrap/argocd-offline/README.md`
- Modify: `bootstrap/argocd-offline/verify-render.sh`
**Step 1: Add a testable cache-check mode**
Extend the verifier or add a private function in `apply.sh` that compares the three exact expected references with:
```bash
k3s ctr images list -q
```
The comparison must be exact, print every missing image, and return non-zero before any `kubectl apply`. It must not contact a registry.
**Step 2: Implement `apply.sh`**
The script must:
1. Use `set -euo pipefail` and resolve all paths relative to its own directory.
2. Require UID 0, `k3s`, and `k3s kubectl kustomize`.
3. Render into a temporary file and remove it with `trap`.
4. Run `verify-render.sh` against that file.
5. Verify all three images exist in k3s containerd.
6. Snapshot current Argo CD Application names to a temporary file.
7. Apply only the rendered file with `k3s kubectl apply -f`.
8. Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1.
9. Fail if any Argo CD pod reports `ErrImagePull` or `ImagePullBackOff`.
10. Confirm the pre-existing Application names are still present, then print pod state.
Use a bounded rollout timeout such as `300s`; do not loop indefinitely.
**Step 3: Document the only supported install/upgrade path**
README must include:
- Purpose and offline limitations: only exact cached tags can start without Harbor.
- Pinned upstream URL and SHA-256.
- Required cached image list.
- Install/recovery command: `sudo ./apply.sh`.
- Verification commands for pods, pull policies, and Applications.
- Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply.
- Warning not to run `kubectl apply` directly against upstream `install.yaml`.
**Step 4: Test failure paths without changing the cluster**
Run locally:
```bash
bootstrap/argocd-offline/apply.sh
```
Expected: non-root or missing-k3s failure before apply.
Run the render verification again and shell syntax checks:
```bash
bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
```
Expected: all syntax/render checks pass.
**Step 5: Commit the operator workflow**
```bash
git add bootstrap/argocd-offline/apply.sh \
bootstrap/argocd-offline/verify-render.sh \
bootstrap/argocd-offline/README.md
git commit -m "feat(gitops): add safe argocd offline apply workflow"
```
### Task 4: Publish the deploy source of truth and Hub pointer
**Files:**
- Modify: Hub submodule pointer `mixingx-deploy`
**Step 1: Rebase and re-run verification before publishing deploy**
```bash
git fetch origin
git rebase origin/master
bash -n bootstrap/argocd-offline/apply.sh
bash -n bootstrap/argocd-offline/verify-render.sh
tmp="$(mktemp)"
kubectl kustomize bootstrap/argocd-offline > "$tmp"
bootstrap/argocd-offline/verify-render.sh "$tmp"
rm -f "$tmp"
git status --short
```
Expected: clean deploy worktree and all verification passes.
**Step 2: Push deploy master**
```bash
git push origin master
```
Expected: remote `master` advances to the verified offline-bootstrap commits.
**Step 3: Update only the Hub submodule pointer**
From the Hub root:
```bash
git fetch origin
git rebase origin/develop/v.1.0.1
git add mixingx-deploy
git diff --cached --submodule=log
```
Expected: the staged diff contains only the `mixingx-deploy` gitlink. The existing untracked `.zcode/plans/` files remain unstaged.
**Step 4: Commit and push the Hub pointer**
```bash
git commit -m "chore(gitops): update deploy offline argocd bootstrap"
git push origin develop/v.1.0.1
```
### Task 5: Recover and verify Argo CD on `192.168.1.173`
**Files:**
- No repository changes expected.
**Step 1: Capture the pre-change cluster inventory**
Over SSH, save or print:
```bash
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide
sudo k3s ctr images list -q
```
Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying.
**Step 2: Transfer the committed bootstrap directory**
From the Hub machine:
```bash
scp -r mixingx-deploy/bootstrap/argocd-offline \
mx@192.168.1.173:/tmp/argocd-offline
```
Use the committed directory that was just pushed; do not assemble a manifest manually on the server.
**Step 3: Apply through the checked entry point**
```bash
ssh mx@192.168.1.173 \
'sudo /tmp/argocd-offline/apply.sh'
```
Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images.
**Step 4: Verify the permanent live state**
Run:
```bash
sudo k3s kubectl -n argocd get pods
sudo k3s kubectl -n argocd get deploy,statefulset \
-o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}'
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
```
Expected:
- Every Argo CD pod is Ready.
- Every listed container and init container reports `IfNotPresent`.
- No pod is in `ErrImagePull` or `ImagePullBackOff`.
- The Application inventory matches the pre-change snapshot.
**Step 5: Prove reconciliation and reboot persistence without widening scope**
Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification.
### Task 6: Final repository and remote verification
**Files:**
- No changes expected.
**Step 1: Verify branch alignment and clean deploy state**
```bash
git -C mixingx-deploy status --short
git -C mixingx-deploy rev-parse HEAD
git -C mixingx-deploy rev-parse origin/master
git status --short
git rev-parse HEAD
git rev-parse origin/develop/v.1.0.1
git submodule status mixingx-deploy
```
Expected:
- Deploy `HEAD` equals `origin/master`.
- Hub `HEAD` equals `origin/develop/v.1.0.1`.
- Hub points at the same deploy commit.
- Only the user's pre-existing untracked `.zcode/plans/` files remain; no implementation files are unstaged.
**Step 2: Record the handoff facts**
Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.