docs(gitops): plan argocd offline bootstrap
This commit is contained in:
1 parent
0fbe0dcb69
commit
ff761a3e90
1 file changed
+365
@@ -0,0 +1,365 @@
|
||||
# Argo CD Offline Bootstrap Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan.
|
||||
|
||||
**Goal:** Make Argo CD on `192.168.1.173` restart from its local k3s image cache when Harbor is offline, and preserve that behavior through server restarts, reinstalls, and controlled upgrades.
|
||||
|
||||
**Architecture:** Vendor the exact upstream Argo CD `v3.4.4` install manifest in `mixingx-deploy`, then wrap it with a Kustomize overlay that rewrites its three upstream images to the existing Harbor references and explicitly changes every Argo CD container and init container to `IfNotPresent`. A checked-in verification script validates the rendered manifest, while an idempotent root-only apply script checks the k3s image cache, applies the render, waits for all Argo CD rollouts, and rejects image-pull failures.
|
||||
|
||||
**Tech Stack:** Argo CD v3.4.4, Kubernetes/Kustomize, k3s embedded containerd and kubectl, Bash, Git submodules.
|
||||
|
||||
**Global constraints:** Do not delete Argo CD Applications or managed workloads. Do not pull, retag, or delete images. Do not apply an unmodified upstream manifest. Do not stage Hub `.zcode/plans/` files. Stop before applying if any required image is absent from the local k3s image store.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add a failing render-verification contract
|
||||
|
||||
**Files:**
|
||||
- Create: `bootstrap/argocd-offline/verify-render.sh`
|
||||
- Create: `bootstrap/argocd-offline/tests/expected-workloads.txt`
|
||||
|
||||
**Step 1: Define the exact supported workload/container inventory**
|
||||
|
||||
Add the seven workload names and their expected main/init container names to `tests/expected-workloads.txt`:
|
||||
|
||||
```text
|
||||
Deployment/argocd-applicationset-controller argocd-applicationset-controller
|
||||
Deployment/argocd-dex-server dex copyutil
|
||||
Deployment/argocd-notifications-controller argocd-notifications-controller
|
||||
Deployment/argocd-redis redis secret-init
|
||||
Deployment/argocd-repo-server argocd-repo-server copyutil
|
||||
Deployment/argocd-server argocd-server
|
||||
StatefulSet/argocd-application-controller argocd-application-controller
|
||||
```
|
||||
|
||||
**Step 2: Write the render verifier before the overlay exists**
|
||||
|
||||
Implement `verify-render.sh <rendered-manifest>` with `set -euo pipefail`. It must fail unless all of these are true:
|
||||
|
||||
- The render contains only the expected Harbor image references for Argo CD, Dex, and Redis.
|
||||
- No upstream `quay.io`, `ghcr.io`, or `public.ecr.aws` image reference remains in a workload.
|
||||
- No `imagePullPolicy: Always` remains.
|
||||
- Each inventory entry is present and each named container/init container has an explicit `imagePullPolicy: IfNotPresent` in its workload block.
|
||||
- The expected images are exactly:
|
||||
|
||||
```text
|
||||
192.168.1.119:8888/argoproj/argocd:v3.4.4
|
||||
192.168.1.119:8888/dexidp/dex:v2.45.0
|
||||
192.168.1.119:8888/library/redis:8.2.3-alpine
|
||||
```
|
||||
|
||||
Keep this verifier dependency-free: Bash plus standard `awk`, `grep`, and `sed` only.
|
||||
|
||||
**Step 3: Run it and confirm the contract fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bootstrap/argocd-offline/verify-render.sh /dev/null
|
||||
```
|
||||
|
||||
Expected: non-zero exit with a clear missing-workload or missing-image error.
|
||||
|
||||
**Step 4: Commit the verification contract**
|
||||
|
||||
```bash
|
||||
git add bootstrap/argocd-offline/verify-render.sh \
|
||||
bootstrap/argocd-offline/tests/expected-workloads.txt
|
||||
git commit -m "test(gitops): define argocd offline render contract"
|
||||
```
|
||||
|
||||
### Task 2: Vendor Argo CD and build the offline Kustomize overlay
|
||||
|
||||
**Files:**
|
||||
- Create: `bootstrap/argocd-offline/vendor/install-v3.4.4.yaml`
|
||||
- Create: `bootstrap/argocd-offline/kustomization.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/application-controller.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/applicationset-controller.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/dex-server.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/notifications-controller.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/redis.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/repo-server.yaml`
|
||||
- Create: `bootstrap/argocd-offline/patches/server.yaml`
|
||||
|
||||
**Step 1: Vendor the exact upstream release**
|
||||
|
||||
Download only the pinned release URL, record its SHA-256 for the README, and inspect the diff:
|
||||
|
||||
```bash
|
||||
curl -fsSL \
|
||||
https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml \
|
||||
-o bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
|
||||
shasum -a 256 bootstrap/argocd-offline/vendor/install-v3.4.4.yaml
|
||||
git diff --stat
|
||||
```
|
||||
|
||||
Expected: a single vendored upstream manifest, with no generated or unrelated files.
|
||||
|
||||
**Step 2: Add the Kustomize image rewrites**
|
||||
|
||||
Create `kustomization.yaml` with the vendored manifest as its only resource and these exact replacements:
|
||||
|
||||
```yaml
|
||||
images:
|
||||
- name: quay.io/argoproj/argocd
|
||||
newName: 192.168.1.119:8888/argoproj/argocd
|
||||
newTag: v3.4.4
|
||||
- name: ghcr.io/dexidp/dex
|
||||
newName: 192.168.1.119:8888/dexidp/dex
|
||||
newTag: v2.45.0
|
||||
- name: public.ecr.aws/docker/library/redis
|
||||
newName: 192.168.1.119:8888/library/redis
|
||||
newTag: 8.2.3-alpine
|
||||
```
|
||||
|
||||
Reference all seven strategic-merge patches from the same file.
|
||||
|
||||
**Step 3: Patch every main and init container explicitly**
|
||||
|
||||
Each patch must target one exact Deployment or StatefulSet in namespace `argocd`. Include the expected container names from Task 1 and set only:
|
||||
|
||||
```yaml
|
||||
imagePullPolicy: IfNotPresent
|
||||
```
|
||||
|
||||
Do not alter replicas, commands, probes, security contexts, volumes, or application resources.
|
||||
|
||||
**Step 4: Render and make the verification contract pass**
|
||||
|
||||
```bash
|
||||
tmp="$(mktemp)"
|
||||
kubectl kustomize bootstrap/argocd-offline > "$tmp"
|
||||
bootstrap/argocd-offline/verify-render.sh "$tmp"
|
||||
rm -f "$tmp"
|
||||
```
|
||||
|
||||
Expected: verifier exits zero. Also inspect the render directly:
|
||||
|
||||
```bash
|
||||
kubectl kustomize bootstrap/argocd-offline \
|
||||
| grep -E 'image:|imagePullPolicy:' \
|
||||
| sort -u
|
||||
```
|
||||
|
||||
Expected: only the three Harbor images and `imagePullPolicy: IfNotPresent`; no `Always`.
|
||||
|
||||
**Step 5: Commit the vendored source and overlay**
|
||||
|
||||
```bash
|
||||
git add bootstrap/argocd-offline
|
||||
git commit -m "feat(gitops): add offline argocd bootstrap overlay"
|
||||
```
|
||||
|
||||
### Task 3: Add the safe, idempotent server apply workflow
|
||||
|
||||
**Files:**
|
||||
- Create: `bootstrap/argocd-offline/apply.sh`
|
||||
- Create: `bootstrap/argocd-offline/README.md`
|
||||
- Modify: `bootstrap/argocd-offline/verify-render.sh`
|
||||
|
||||
**Step 1: Add a testable cache-check mode**
|
||||
|
||||
Extend the verifier or add a private function in `apply.sh` that compares the three exact expected references with:
|
||||
|
||||
```bash
|
||||
k3s ctr images list -q
|
||||
```
|
||||
|
||||
The comparison must be exact, print every missing image, and return non-zero before any `kubectl apply`. It must not contact a registry.
|
||||
|
||||
**Step 2: Implement `apply.sh`**
|
||||
|
||||
The script must:
|
||||
|
||||
1. Use `set -euo pipefail` and resolve all paths relative to its own directory.
|
||||
2. Require UID 0, `k3s`, and `k3s kubectl kustomize`.
|
||||
3. Render into a temporary file and remove it with `trap`.
|
||||
4. Run `verify-render.sh` against that file.
|
||||
5. Verify all three images exist in k3s containerd.
|
||||
6. Snapshot current Argo CD Application names to a temporary file.
|
||||
7. Apply only the rendered file with `k3s kubectl apply -f`.
|
||||
8. Wait with explicit timeouts for the six Deployments and one StatefulSet in Task 1.
|
||||
9. Fail if any Argo CD pod reports `ErrImagePull` or `ImagePullBackOff`.
|
||||
10. Confirm the pre-existing Application names are still present, then print pod state.
|
||||
|
||||
Use a bounded rollout timeout such as `300s`; do not loop indefinitely.
|
||||
|
||||
**Step 3: Document the only supported install/upgrade path**
|
||||
|
||||
README must include:
|
||||
|
||||
- Purpose and offline limitations: only exact cached tags can start without Harbor.
|
||||
- Pinned upstream URL and SHA-256.
|
||||
- Required cached image list.
|
||||
- Install/recovery command: `sudo ./apply.sh`.
|
||||
- Verification commands for pods, pull policies, and Applications.
|
||||
- Upgrade checklist: update vendored exact release, image rewrites, patches/inventory if upstream workloads change, checksum, render verification, cache images first, then apply.
|
||||
- Warning not to run `kubectl apply` directly against upstream `install.yaml`.
|
||||
|
||||
**Step 4: Test failure paths without changing the cluster**
|
||||
|
||||
Run locally:
|
||||
|
||||
```bash
|
||||
bootstrap/argocd-offline/apply.sh
|
||||
```
|
||||
|
||||
Expected: non-root or missing-k3s failure before apply.
|
||||
|
||||
Run the render verification again and shell syntax checks:
|
||||
|
||||
```bash
|
||||
bash -n bootstrap/argocd-offline/apply.sh
|
||||
bash -n bootstrap/argocd-offline/verify-render.sh
|
||||
tmp="$(mktemp)"
|
||||
kubectl kustomize bootstrap/argocd-offline > "$tmp"
|
||||
bootstrap/argocd-offline/verify-render.sh "$tmp"
|
||||
rm -f "$tmp"
|
||||
```
|
||||
|
||||
Expected: all syntax/render checks pass.
|
||||
|
||||
**Step 5: Commit the operator workflow**
|
||||
|
||||
```bash
|
||||
git add bootstrap/argocd-offline/apply.sh \
|
||||
bootstrap/argocd-offline/verify-render.sh \
|
||||
bootstrap/argocd-offline/README.md
|
||||
git commit -m "feat(gitops): add safe argocd offline apply workflow"
|
||||
```
|
||||
|
||||
### Task 4: Publish the deploy source of truth and Hub pointer
|
||||
|
||||
**Files:**
|
||||
- Modify: Hub submodule pointer `mixingx-deploy`
|
||||
|
||||
**Step 1: Rebase and re-run verification before publishing deploy**
|
||||
|
||||
```bash
|
||||
git fetch origin
|
||||
git rebase origin/master
|
||||
bash -n bootstrap/argocd-offline/apply.sh
|
||||
bash -n bootstrap/argocd-offline/verify-render.sh
|
||||
tmp="$(mktemp)"
|
||||
kubectl kustomize bootstrap/argocd-offline > "$tmp"
|
||||
bootstrap/argocd-offline/verify-render.sh "$tmp"
|
||||
rm -f "$tmp"
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected: clean deploy worktree and all verification passes.
|
||||
|
||||
**Step 2: Push deploy master**
|
||||
|
||||
```bash
|
||||
git push origin master
|
||||
```
|
||||
|
||||
Expected: remote `master` advances to the verified offline-bootstrap commits.
|
||||
|
||||
**Step 3: Update only the Hub submodule pointer**
|
||||
|
||||
From the Hub root:
|
||||
|
||||
```bash
|
||||
git fetch origin
|
||||
git rebase origin/develop/v.1.0.1
|
||||
git add mixingx-deploy
|
||||
git diff --cached --submodule=log
|
||||
```
|
||||
|
||||
Expected: the staged diff contains only the `mixingx-deploy` gitlink. The existing untracked `.zcode/plans/` files remain unstaged.
|
||||
|
||||
**Step 4: Commit and push the Hub pointer**
|
||||
|
||||
```bash
|
||||
git commit -m "chore(gitops): update deploy offline argocd bootstrap"
|
||||
git push origin develop/v.1.0.1
|
||||
```
|
||||
|
||||
### Task 5: Recover and verify Argo CD on `192.168.1.173`
|
||||
|
||||
**Files:**
|
||||
- No repository changes expected.
|
||||
|
||||
**Step 1: Capture the pre-change cluster inventory**
|
||||
|
||||
Over SSH, save or print:
|
||||
|
||||
```bash
|
||||
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
|
||||
sudo k3s kubectl -n argocd get deploy,statefulset,pods -o wide
|
||||
sudo k3s ctr images list -q
|
||||
```
|
||||
|
||||
Expected: the existing Applications are visible and all three exact Harbor images exist. If any image is missing, stop without applying.
|
||||
|
||||
**Step 2: Transfer the committed bootstrap directory**
|
||||
|
||||
From the Hub machine:
|
||||
|
||||
```bash
|
||||
scp -r mixingx-deploy/bootstrap/argocd-offline \
|
||||
mx@192.168.1.173:/tmp/argocd-offline
|
||||
```
|
||||
|
||||
Use the committed directory that was just pushed; do not assemble a manifest manually on the server.
|
||||
|
||||
**Step 3: Apply through the checked entry point**
|
||||
|
||||
```bash
|
||||
ssh mx@192.168.1.173 \
|
||||
'sudo /tmp/argocd-offline/apply.sh'
|
||||
```
|
||||
|
||||
Expected: render and cache checks pass; all six Deployments and the application-controller StatefulSet complete rollout using cached images.
|
||||
|
||||
**Step 4: Verify the permanent live state**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
sudo k3s kubectl -n argocd get pods
|
||||
sudo k3s kubectl -n argocd get deploy,statefulset \
|
||||
-o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}'
|
||||
sudo k3s kubectl -n argocd get applications.argoproj.io -o name
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- Every Argo CD pod is Ready.
|
||||
- Every listed container and init container reports `IfNotPresent`.
|
||||
- No pod is in `ErrImagePull` or `ImagePullBackOff`.
|
||||
- The Application inventory matches the pre-change snapshot.
|
||||
|
||||
**Step 5: Prove reconciliation and reboot persistence without widening scope**
|
||||
|
||||
Confirm Argo CD Applications can refresh from their existing Git sources. Then hand control back to the operator for the planned server reboot. After reboot, repeat Step 4; do not modify application workloads as part of this bootstrap verification.
|
||||
|
||||
### Task 6: Final repository and remote verification
|
||||
|
||||
**Files:**
|
||||
- No changes expected.
|
||||
|
||||
**Step 1: Verify branch alignment and clean deploy state**
|
||||
|
||||
```bash
|
||||
git -C mixingx-deploy status --short
|
||||
git -C mixingx-deploy rev-parse HEAD
|
||||
git -C mixingx-deploy rev-parse origin/master
|
||||
git status --short
|
||||
git rev-parse HEAD
|
||||
git rev-parse origin/develop/v.1.0.1
|
||||
git submodule status mixingx-deploy
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- Deploy `HEAD` equals `origin/master`.
|
||||
- Hub `HEAD` equals `origin/develop/v.1.0.1`.
|
||||
- Hub points at the same deploy commit.
|
||||
- Only the user's pre-existing untracked `.zcode/plans/` files remain; no implementation files are unstaged.
|
||||
|
||||
**Step 2: Record the handoff facts**
|
||||
|
||||
Report the deploy commit, Hub commit, live Argo CD pod readiness, pull-policy verification, retained Application count, and that the next user action is the planned reboot test.
|
||||
Reference in new issue
Block a user