# Argo CD Offline Bootstrap This directory is the supported installation and recovery path for Argo CD on the demo k3s server `192.168.1.173`. It allows Argo CD to restart while Harbor `192.168.1.119:8888` is unavailable, provided the exact pinned images are already present in the server's k3s containerd cache. It does not make new or uncached image versions available offline. ## Pinned sources - Argo CD release: `v3.4.4` - Upstream manifest: `https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml` - Vendored manifest: `vendor/install-v3.4.4.yaml` - SHA-256: `b0f9119821f2e19b852c842b9cb235eb9c3ef1549554fbda6aa5904e8d440eae` The Kustomize overlay rewrites the upstream images to: ```text 192.168.1.119:8888/argoproj/argocd:v3.4.4 192.168.1.119:8888/dexidp/dex:v2.45.0 192.168.1.119:8888/library/redis:8.2.3-alpine ``` All Argo CD Deployment and StatefulSet containers, including init containers, explicitly use `imagePullPolicy: IfNotPresent`. ## Install or recover Copy this entire directory to the k3s server, then run: ```bash sudo ./apply.sh ``` The script is idempotent. Before applying anything, it: 1. Renders and verifies the local vendored manifest. 2. Rejects upstream images and any `imagePullPolicy: Always`. 3. Verifies all three exact Harbor image references are in the local k3s image cache. 4. Records the existing Argo CD Applications. It then applies the render with Kubernetes Server-Side Apply and conflict resolution, waits up to 300 seconds per workload, verifies the live pull policies, rejects image-pull failures, and confirms that pre-existing Applications remain present. Server-Side Apply is required for Argo CD 3.3 and later because the ApplicationSet CRD exceeds Kubernetes' 262144-byte annotation limit for client-side apply. The flags match the official Argo CD install and upgrade procedure: ```text --server-side --force-conflicts ``` The script never pulls, retags, deletes, or garbage-collects images. If an image is missing, restore Harbor or deliberately import that exact image before retrying. Do not apply the upstream `install.yaml` directly. Its `Always` policies will make Argo CD depend on Harbor again after pods or the server restart. ## Verify ```bash sudo k3s kubectl -n argocd get pods sudo k3s kubectl -n argocd get deploy,statefulset \ -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]} init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]} container:{.name}={.imagePullPolicy}{"\n"}{end}{end}' sudo k3s kubectl -n argocd get applications.argoproj.io -o name ``` Every listed container and init container must report `IfNotPresent`. No pod may report `ErrImagePull` or `ImagePullBackOff`. ## Upgrade For each Argo CD upgrade: 1. Download `manifests/install.yaml` from an exact Argo CD release tag, never `stable`. 2. Replace the vendored manifest and record its new SHA-256 here. 3. Update the Argo CD, Dex, and Redis versions in `kustomization.yaml`, `verify-render.sh`, and `apply.sh`. 4. Compare upstream workload and container names with `tests/expected-workloads.txt`; update the inventory and patches if upstream changed them. 5. Run `kubectl kustomize .` and `./verify-render.sh ` locally. 6. Push the new images to Harbor and ensure the exact Harbor references are cached or explicitly imported on `192.168.1.173`. 7. Run only `sudo ./apply.sh`, then verify rollouts, pull policies, pods, and Applications. Never apply an upgrade until the cache precheck passes.