Files
mixingx.deploy/docs/superpowers/specs/2026-07-15-argocd-offline-bootstrap-design.md
T

79 lines
4.3 KiB
Markdown

# Argo CD Offline Bootstrap
## Goal
Make Argo CD on the demo k3s server `192.168.1.173` able to start and reconcile from locally cached images while Harbor `192.168.1.119:8888` is offline, and provide a version-controlled installation path that preserves this behavior across server restarts, Argo CD reinstalls, and upgrades.
## Current State and Root Cause
Argo CD was installed with `kubectl apply`, then its images were redirected to Harbor. Most Argo CD containers still use `imagePullPolicy: Always`. When their pod sandboxes are recreated, kubelet contacts Harbor even though the exact images are cached locally. With Harbor offline, image resolution fails with `no route to host`, leaving the Argo CD control plane in `ErrImagePull` or `ImagePullBackOff`.
The following exact images are already cached on `192.168.1.173`:
- `192.168.1.119:8888/argoproj/argocd:v3.4.4`
- `192.168.1.119:8888/dexidp/dex:v2.45.0`
- `192.168.1.119:8888/library/redis:8.2.3-alpine`
## Repository Design
Create `bootstrap/argocd-offline/` in `mixingx.deploy` with:
- A vendored copy of the official Argo CD `v3.4.4` install manifest, pinned to that release rather than `stable`.
- A `kustomization.yaml` that rewrites the Argo CD, Dex, and Redis images to the Harbor references above.
- Kustomize patches that set every Argo CD workload container and init container to `imagePullPolicy: IfNotPresent`.
- An idempotent `apply.sh` entry point for the demo k3s server.
- A README describing installation, verification, offline recovery, and upgrade rules.
Vendoring the upstream manifest makes the bootstrap independent of GitHub availability. The supported install and upgrade path becomes:
```bash
sudo ./apply.sh
```
Operators must not apply an unmodified upstream Argo CD manifest directly, because doing so can restore `Always` policies.
## Apply Script Behavior
The script will:
1. Require root privileges and verify `k3s` and its embedded `kubectl` are available.
2. Render the local Kustomize overlay before making changes.
3. Reject the render if any workload container or init container still uses `imagePullPolicy: Always`.
4. Verify all three required image references exist in the local k3s containerd image store.
5. Apply the rendered resources to namespace `argocd`.
6. Wait for the Argo CD Deployments and application-controller StatefulSet to complete rollout.
7. Print pod state and fail if any Argo CD pod remains in an image-pull error.
The script must not pull, retag, delete, or garbage-collect images. Missing cached images are a hard stop so the operator can restore Harbor or import the required image deliberately.
## Live Recovery
After repository validation, run the versioned bootstrap against `192.168.1.173`. Applying the overlay changes the pod templates from `Always` to `IfNotPresent`, causing Argo CD pods to restart using the exact locally cached images. No Harbor access is required for this recovery.
Verify all Argo CD components become Ready and the existing MixingX Applications remain present. The bootstrap must not delete Applications or application workloads.
## GitOps Publication
1. Commit and push the `mixingx.deploy` `master` branch.
2. Update the `mixingx.hub` `mixingx-deploy` submodule pointer on `develop/v.1.0.1`.
3. Commit and push the Hub branch without staging existing `.zcode/plans/` files.
## Upgrade Procedure
For an Argo CD upgrade:
1. Replace the vendored manifest with the official manifest for an exact release tag.
2. Update the three expected image versions and Harbor rewrites.
3. Render locally and verify there are no `Always` policies or unexpected external image references.
4. Ensure the new images are present in Harbor and cached or explicitly imported on `192.168.1.173` before application.
5. Apply only through the versioned bootstrap directory and verify rollouts.
## Verification
- The Kustomize render contains the expected Argo CD `v3.4.4`, Dex `v2.45.0`, and Redis `8.2.3-alpine` Harbor references.
- Every Deployment and StatefulSet container/initContainer has `imagePullPolicy: IfNotPresent`.
- No application container image is rewritten or modified.
- All Argo CD pods on `192.168.1.173` become Ready while Harbor remains unavailable.
- Existing Argo CD Applications remain intact.
- Repository branches and the Hub submodule pointer are synchronized with their remotes.