79 lines
4.3 KiB
Markdown
79 lines
4.3 KiB
Markdown
# Argo CD Offline Bootstrap
|
|
|
|
## Goal
|
|
|
|
Make Argo CD on the demo k3s server `192.168.1.173` able to start and reconcile from locally cached images while Harbor `192.168.1.119:8888` is offline, and provide a version-controlled installation path that preserves this behavior across server restarts, Argo CD reinstalls, and upgrades.
|
|
|
|
## Current State and Root Cause
|
|
|
|
Argo CD was installed with `kubectl apply`, then its images were redirected to Harbor. Most Argo CD containers still use `imagePullPolicy: Always`. When their pod sandboxes are recreated, kubelet contacts Harbor even though the exact images are cached locally. With Harbor offline, image resolution fails with `no route to host`, leaving the Argo CD control plane in `ErrImagePull` or `ImagePullBackOff`.
|
|
|
|
The following exact images are already cached on `192.168.1.173`:
|
|
|
|
- `192.168.1.119:8888/argoproj/argocd:v3.4.4`
|
|
- `192.168.1.119:8888/dexidp/dex:v2.45.0`
|
|
- `192.168.1.119:8888/library/redis:8.2.3-alpine`
|
|
|
|
## Repository Design
|
|
|
|
Create `bootstrap/argocd-offline/` in `mixingx.deploy` with:
|
|
|
|
- A vendored copy of the official Argo CD `v3.4.4` install manifest, pinned to that release rather than `stable`.
|
|
- A `kustomization.yaml` that rewrites the Argo CD, Dex, and Redis images to the Harbor references above.
|
|
- Kustomize patches that set every Argo CD workload container and init container to `imagePullPolicy: IfNotPresent`.
|
|
- An idempotent `apply.sh` entry point for the demo k3s server.
|
|
- A README describing installation, verification, offline recovery, and upgrade rules.
|
|
|
|
Vendoring the upstream manifest makes the bootstrap independent of GitHub availability. The supported install and upgrade path becomes:
|
|
|
|
```bash
|
|
sudo ./apply.sh
|
|
```
|
|
|
|
Operators must not apply an unmodified upstream Argo CD manifest directly, because doing so can restore `Always` policies.
|
|
|
|
## Apply Script Behavior
|
|
|
|
The script will:
|
|
|
|
1. Require root privileges and verify `k3s` and its embedded `kubectl` are available.
|
|
2. Render the local Kustomize overlay before making changes.
|
|
3. Reject the render if any workload container or init container still uses `imagePullPolicy: Always`.
|
|
4. Verify all three required image references exist in the local k3s containerd image store.
|
|
5. Apply the rendered resources to namespace `argocd`.
|
|
6. Wait for the Argo CD Deployments and application-controller StatefulSet to complete rollout.
|
|
7. Print pod state and fail if any Argo CD pod remains in an image-pull error.
|
|
|
|
The script must not pull, retag, delete, or garbage-collect images. Missing cached images are a hard stop so the operator can restore Harbor or import the required image deliberately.
|
|
|
|
## Live Recovery
|
|
|
|
After repository validation, run the versioned bootstrap against `192.168.1.173`. Applying the overlay changes the pod templates from `Always` to `IfNotPresent`, causing Argo CD pods to restart using the exact locally cached images. No Harbor access is required for this recovery.
|
|
|
|
Verify all Argo CD components become Ready and the existing MixingX Applications remain present. The bootstrap must not delete Applications or application workloads.
|
|
|
|
## GitOps Publication
|
|
|
|
1. Commit and push the `mixingx.deploy` `master` branch.
|
|
2. Update the `mixingx.hub` `mixingx-deploy` submodule pointer on `develop/v.1.0.1`.
|
|
3. Commit and push the Hub branch without staging existing `.zcode/plans/` files.
|
|
|
|
## Upgrade Procedure
|
|
|
|
For an Argo CD upgrade:
|
|
|
|
1. Replace the vendored manifest with the official manifest for an exact release tag.
|
|
2. Update the three expected image versions and Harbor rewrites.
|
|
3. Render locally and verify there are no `Always` policies or unexpected external image references.
|
|
4. Ensure the new images are present in Harbor and cached or explicitly imported on `192.168.1.173` before application.
|
|
5. Apply only through the versioned bootstrap directory and verify rollouts.
|
|
|
|
## Verification
|
|
|
|
- The Kustomize render contains the expected Argo CD `v3.4.4`, Dex `v2.45.0`, and Redis `8.2.3-alpine` Harbor references.
|
|
- Every Deployment and StatefulSet container/initContainer has `imagePullPolicy: IfNotPresent`.
|
|
- No application container image is rewritten or modified.
|
|
- All Argo CD pods on `192.168.1.173` become Ready while Harbor remains unavailable.
|
|
- Existing Argo CD Applications remain intact.
|
|
- Repository branches and the Hub submodule pointer are synchronized with their remotes.
|