Files
mixingx.deploy/bootstrap/argocd-offline/README.md
T

3.5 KiB

Argo CD Offline Bootstrap

This directory is the supported installation and recovery path for Argo CD on the demo k3s server 192.168.1.173.

It allows Argo CD to restart while Harbor 192.168.1.119:8888 is unavailable, provided the exact pinned images are already present in the server's k3s containerd cache. It does not make new or uncached image versions available offline.

Pinned sources

  • Argo CD release: v3.4.4
  • Upstream manifest: https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml
  • Vendored manifest: vendor/install-v3.4.4.yaml
  • SHA-256: b0f9119821f2e19b852c842b9cb235eb9c3ef1549554fbda6aa5904e8d440eae

The Kustomize overlay rewrites the upstream images to:

192.168.1.119:8888/argoproj/argocd:v3.4.4
192.168.1.119:8888/dexidp/dex:v2.45.0
192.168.1.119:8888/library/redis:8.2.3-alpine

All Argo CD Deployment and StatefulSet containers, including init containers, explicitly use imagePullPolicy: IfNotPresent.

Install or recover

Copy this entire directory to the k3s server, then run:

sudo ./apply.sh

The script is idempotent. Before applying anything, it:

  1. Renders and verifies the local vendored manifest.
  2. Rejects upstream images and any imagePullPolicy: Always.
  3. Verifies all three exact Harbor image references are in the local k3s image cache.
  4. Records the existing Argo CD Applications.

It then applies the render with Kubernetes Server-Side Apply and conflict resolution, waits up to 300 seconds per workload, verifies the live pull policies, rejects image-pull failures, and confirms that pre-existing Applications remain present.

Server-Side Apply is required for Argo CD 3.3 and later because the ApplicationSet CRD exceeds Kubernetes' 262144-byte annotation limit for client-side apply. The flags match the official Argo CD install and upgrade procedure:

--server-side --force-conflicts

The script never pulls, retags, deletes, or garbage-collects images. If an image is missing, restore Harbor or deliberately import that exact image before retrying.

Do not apply the upstream install.yaml directly. Its Always policies will make Argo CD depend on Harbor again after pods or the server restart.

Verify

sudo k3s kubectl -n argocd get pods

sudo k3s kubectl -n argocd get deploy,statefulset \
  -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\n"}{range .spec.template.spec.initContainers[*]}  init:{.name}={.imagePullPolicy}{"\n"}{end}{range .spec.template.spec.containers[*]}  container:{.name}={.imagePullPolicy}{"\n"}{end}{end}'

sudo k3s kubectl -n argocd get applications.argoproj.io -o name

Every listed container and init container must report IfNotPresent. No pod may report ErrImagePull or ImagePullBackOff.

Upgrade

For each Argo CD upgrade:

  1. Download manifests/install.yaml from an exact Argo CD release tag, never stable.
  2. Replace the vendored manifest and record its new SHA-256 here.
  3. Update the Argo CD, Dex, and Redis versions in kustomization.yaml, verify-render.sh, and apply.sh.
  4. Compare upstream workload and container names with tests/expected-workloads.txt; update the inventory and patches if upstream changed them.
  5. Run kubectl kustomize . and ./verify-render.sh <rendered-file> locally.
  6. Push the new images to Harbor and ensure the exact Harbor references are cached or explicitly imported on 192.168.1.173.
  7. Run only sudo ./apply.sh, then verify rollouts, pull policies, pods, and Applications.

Never apply an upgrade until the cache precheck passes.