Files
mixingx.deploy/docs/superpowers/specs/2026-07-15-argocd-offline-bootstrap-design.md
T

4.3 KiB

Argo CD Offline Bootstrap

Goal

Make Argo CD on the demo k3s server 192.168.1.173 able to start and reconcile from locally cached images while Harbor 192.168.1.119:8888 is offline, and provide a version-controlled installation path that preserves this behavior across server restarts, Argo CD reinstalls, and upgrades.

Current State and Root Cause

Argo CD was installed with kubectl apply, then its images were redirected to Harbor. Most Argo CD containers still use imagePullPolicy: Always. When their pod sandboxes are recreated, kubelet contacts Harbor even though the exact images are cached locally. With Harbor offline, image resolution fails with no route to host, leaving the Argo CD control plane in ErrImagePull or ImagePullBackOff.

The following exact images are already cached on 192.168.1.173:

  • 192.168.1.119:8888/argoproj/argocd:v3.4.4
  • 192.168.1.119:8888/dexidp/dex:v2.45.0
  • 192.168.1.119:8888/library/redis:8.2.3-alpine

Repository Design

Create bootstrap/argocd-offline/ in mixingx.deploy with:

  • A vendored copy of the official Argo CD v3.4.4 install manifest, pinned to that release rather than stable.
  • A kustomization.yaml that rewrites the Argo CD, Dex, and Redis images to the Harbor references above.
  • Kustomize patches that set every Argo CD workload container and init container to imagePullPolicy: IfNotPresent.
  • An idempotent apply.sh entry point for the demo k3s server.
  • A README describing installation, verification, offline recovery, and upgrade rules.

Vendoring the upstream manifest makes the bootstrap independent of GitHub availability. The supported install and upgrade path becomes:

sudo ./apply.sh

Operators must not apply an unmodified upstream Argo CD manifest directly, because doing so can restore Always policies.

Apply Script Behavior

The script will:

  1. Require root privileges and verify k3s and its embedded kubectl are available.
  2. Render the local Kustomize overlay before making changes.
  3. Reject the render if any workload container or init container still uses imagePullPolicy: Always.
  4. Verify all three required image references exist in the local k3s containerd image store.
  5. Apply the rendered resources to namespace argocd.
  6. Wait for the Argo CD Deployments and application-controller StatefulSet to complete rollout.
  7. Print pod state and fail if any Argo CD pod remains in an image-pull error.

The script must not pull, retag, delete, or garbage-collect images. Missing cached images are a hard stop so the operator can restore Harbor or import the required image deliberately.

Live Recovery

After repository validation, run the versioned bootstrap against 192.168.1.173. Applying the overlay changes the pod templates from Always to IfNotPresent, causing Argo CD pods to restart using the exact locally cached images. No Harbor access is required for this recovery.

Verify all Argo CD components become Ready and the existing MixingX Applications remain present. The bootstrap must not delete Applications or application workloads.

GitOps Publication

  1. Commit and push the mixingx.deploy master branch.
  2. Update the mixingx.hub mixingx-deploy submodule pointer on develop/v.1.0.1.
  3. Commit and push the Hub branch without staging existing .zcode/plans/ files.

Upgrade Procedure

For an Argo CD upgrade:

  1. Replace the vendored manifest with the official manifest for an exact release tag.
  2. Update the three expected image versions and Harbor rewrites.
  3. Render locally and verify there are no Always policies or unexpected external image references.
  4. Ensure the new images are present in Harbor and cached or explicitly imported on 192.168.1.173 before application.
  5. Apply only through the versioned bootstrap directory and verify rollouts.

Verification

  • The Kustomize render contains the expected Argo CD v3.4.4, Dex v2.45.0, and Redis 8.2.3-alpine Harbor references.
  • Every Deployment and StatefulSet container/initContainer has imagePullPolicy: IfNotPresent.
  • No application container image is rewritten or modified.
  • All Argo CD pods on 192.168.1.173 become Ready while Harbor remains unavailable.
  • Existing Argo CD Applications remain intact.
  • Repository branches and the Hub submodule pointer are synchronized with their remotes.