Kubernetes GitOps Checklist: ArgoCD, DNS, Secrets, Rollouts

App-of-apps layout

gitops/
  apps/                  # one Application manifest per workload
    billing-api.yaml
    temporal-workers.yaml
  platform/              # cluster-level: ingress, cert-manager, external-dns
  root.yaml              # the app-of-apps Application

• The root app points at apps/ and platform/; it is the only thing installed by hand.
• One Application per deployable unit, not per team and not one giant app.
• Pin images by tag or digest. latest makes the repo a liar: ArgoCD sees no diff when the image moves.
• selfHeal: true, prune: true on everything. Drift that heals itself stops being an incident category.

Sync waves

Wave	What
-2	CRDs, namespaces
-1	Secrets stores, cert-manager, platform controllers
0	Databases, schemas, migrations (PreSync jobs)
1	Services, workers
2	Ingress, monitors, dashboards

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

external-dns

• One deployment per cluster, scoped to the zones it owns (--domain-filter).
• Records come from annotations, never hand-created in the console:

metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: api.internal.example.com

• policy: upsert-only until you trust it; sync only after an audit of existing records.
• Set a per-cluster TXT owner ID (--txt-owner-id), or two clusters will fight over records.
• Check: delete a record by hand; it must reappear within one sync interval. If it does not, external-dns is decoration.

Secrets

Two patterns that work:

	External Secrets Operator	Sealed Secrets
Source of truth	Cloud secret manager (AWS SM, Vault, GCP SM)	Encrypted blobs in git
Rotation	Provider-side, synced on interval	Re-seal and commit
Best for	Most platforms	Small clusters, no external manager

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: billing-api-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: billing-api-secrets
  data:
    - secretKey: database-url
      remoteRef:
        key: prod/billing-api/database-url

Rules either way: no plaintext secrets in git, ever; no secrets baked into images; rotation tested before the incident, not during.

Progressive rollout gates

Argo Rollouts (or Flagger) for anything user-facing:

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - setWeight: 50
      - pause: { duration: 10m }

Gates that catch real failures, in order of value:

1. Error rate from the service's own metrics (5xx ratio above baseline).
2. p95 latency against the previous version, not a fixed number.
3. Dependency health: downstream error rates, queue depths.
4. A business metric where one exists (checkout success, task completion).

Checks before merge, not after deploy: argocd app diff reviewed in CI, manifest validation (kubeconform or policy checks), and no kubectl apply paths outside ArgoCD.

Rollback drills (quarterly)

• Revert a merge and watch ArgoCD converge. Time it. Minutes, not tens of minutes.
• argocd app rollback for a config-only disaster; confirm you know which history entry is good before you need it.
• Roll back a change that included a CRD bump. ArgoCD will not downgrade CRDs; have the answer written down.
• Roll back with a database migration in flight. Forward-only migrations or expand-contract; pick one and document it.
• Rotate a secret and confirm pods pick it up without a manual restart.
• Break external-dns in a sandbox and confirm sync recovers the records.

A rollback that has never been rehearsed is a hope, not a capability.