4.3 KiB
Argo CD Offline Bootstrap
Goal
Make Argo CD on the demo k3s server 192.168.1.173 able to start and reconcile from locally cached images while Harbor 192.168.1.119:8888 is offline, and provide a version-controlled installation path that preserves this behavior across server restarts, Argo CD reinstalls, and upgrades.
Current State and Root Cause
Argo CD was installed with kubectl apply, then its images were redirected to Harbor. Most Argo CD containers still use imagePullPolicy: Always. When their pod sandboxes are recreated, kubelet contacts Harbor even though the exact images are cached locally. With Harbor offline, image resolution fails with no route to host, leaving the Argo CD control plane in ErrImagePull or ImagePullBackOff.
The following exact images are already cached on 192.168.1.173:
192.168.1.119:8888/argoproj/argocd:v3.4.4192.168.1.119:8888/dexidp/dex:v2.45.0192.168.1.119:8888/library/redis:8.2.3-alpine
Repository Design
Create bootstrap/argocd-offline/ in mixingx.deploy with:
- A vendored copy of the official Argo CD
v3.4.4install manifest, pinned to that release rather thanstable. - A
kustomization.yamlthat rewrites the Argo CD, Dex, and Redis images to the Harbor references above. - Kustomize patches that set every Argo CD workload container and init container to
imagePullPolicy: IfNotPresent. - An idempotent
apply.shentry point for the demo k3s server. - A README describing installation, verification, offline recovery, and upgrade rules.
Vendoring the upstream manifest makes the bootstrap independent of GitHub availability. The supported install and upgrade path becomes:
sudo ./apply.sh
Operators must not apply an unmodified upstream Argo CD manifest directly, because doing so can restore Always policies.
Apply Script Behavior
The script will:
- Require root privileges and verify
k3sand its embeddedkubectlare available. - Render the local Kustomize overlay before making changes.
- Reject the render if any workload container or init container still uses
imagePullPolicy: Always. - Verify all three required image references exist in the local k3s containerd image store.
- Apply the rendered resources to namespace
argocd. - Wait for the Argo CD Deployments and application-controller StatefulSet to complete rollout.
- Print pod state and fail if any Argo CD pod remains in an image-pull error.
The script must not pull, retag, delete, or garbage-collect images. Missing cached images are a hard stop so the operator can restore Harbor or import the required image deliberately.
Live Recovery
After repository validation, run the versioned bootstrap against 192.168.1.173. Applying the overlay changes the pod templates from Always to IfNotPresent, causing Argo CD pods to restart using the exact locally cached images. No Harbor access is required for this recovery.
Verify all Argo CD components become Ready and the existing MixingX Applications remain present. The bootstrap must not delete Applications or application workloads.
GitOps Publication
- Commit and push the
mixingx.deploymasterbranch. - Update the
mixingx.hubmixingx-deploysubmodule pointer ondevelop/v.1.0.1. - Commit and push the Hub branch without staging existing
.zcode/plans/files.
Upgrade Procedure
For an Argo CD upgrade:
- Replace the vendored manifest with the official manifest for an exact release tag.
- Update the three expected image versions and Harbor rewrites.
- Render locally and verify there are no
Alwayspolicies or unexpected external image references. - Ensure the new images are present in Harbor and cached or explicitly imported on
192.168.1.173before application. - Apply only through the versioned bootstrap directory and verify rollouts.
Verification
- The Kustomize render contains the expected Argo CD
v3.4.4, Dexv2.45.0, and Redis8.2.3-alpineHarbor references. - Every Deployment and StatefulSet container/initContainer has
imagePullPolicy: IfNotPresent. - No application container image is rewritten or modified.
- All Argo CD pods on
192.168.1.173become Ready while Harbor remains unavailable. - Existing Argo CD Applications remain intact.
- Repository branches and the Hub submodule pointer are synchronized with their remotes.