infra: add kustomize HA deployment tree

mhahl-a commented

2026-08-15 08:28:24 +00:00

Owner

Full HA kustomize composition for Kubernetes:

base/ — stateless tier (Flask app + PostgREST) with HPA, PDB,
topology-spread, anti-affinity, hardened securityContext;
Postgres via CloudNativePG (3 instances, sync-replication quorum,
fencing, S3 WAL backups); Redis HA via Sentinel (3 redis + 3
sentinels, AOF everysec) with a kustomize-native failover
controller; External Secrets Operator wiring for MASTER_KEY /
JWT_SECRET / SECRET_KEY / DB / Redis / backup credentials;
cert-manager Ingress + ClusterIssuer; default-deny NetworkPolicy
with per-component allow-lists; Pod Security Standards
'restricted' namespace label.
overlays/prod — production sizing (5 app / 3 pg / 3 redis,
SSD storage, ResourceQuota + LimitRange, Let's Encrypt,
pinned image tags).
overlays/staging — single-replica cost-tuned variant, self-signed
TLS, dev-mode Vault.

README documents operator prerequisites, Vault policy/KV layout,
build/apply/dry-run commands, admin promotion, MASTER_KEY rotation,
pg failover + PITR, redis failover verification, upgrade procedure,
and monitoring alerts.

kustomize build verified clean on base, prod, and staging.

Full HA kustomize composition for Kubernetes: - base/ — stateless tier (Flask app + PostgREST) with HPA, PDB, topology-spread, anti-affinity, hardened securityContext; Postgres via CloudNativePG (3 instances, sync-replication quorum, fencing, S3 WAL backups); Redis HA via Sentinel (3 redis + 3 sentinels, AOF everysec) with a kustomize-native failover controller; External Secrets Operator wiring for MASTER_KEY / JWT_SECRET / SECRET_KEY / DB / Redis / backup credentials; cert-manager Ingress + ClusterIssuer; default-deny NetworkPolicy with per-component allow-lists; Pod Security Standards 'restricted' namespace label. - overlays/prod — production sizing (5 app / 3 pg / 3 redis, SSD storage, ResourceQuota + LimitRange, Let's Encrypt, pinned image tags). - overlays/staging — single-replica cost-tuned variant, self-signed TLS, dev-mode Vault. README documents operator prerequisites, Vault policy/KV layout, build/apply/dry-run commands, admin promotion, MASTER_KEY rotation, pg failover + PITR, redis failover verification, upgrade procedure, and monitoring alerts. kustomize build verified clean on base, prod, and staging.

mhahl-a added 1 commit

2026-08-15 08:28:24 +00:00

infra: add kustomize HA deployment tree

CI / tox py + lint (pull_request) Has been cancelled

Details

adf9d3c8b5

Full HA kustomize composition for Kubernetes:

- base/ — stateless tier (Flask app + PostgREST) with HPA, PDB,
  topology-spread, anti-affinity, hardened securityContext;
  Postgres via CloudNativePG (3 instances, sync-replication quorum,
  fencing, S3 WAL backups); Redis HA via Sentinel (3 redis + 3
  sentinels, AOF everysec) with a kustomize-native failover
  controller; External Secrets Operator wiring for MASTER_KEY /
  JWT_SECRET / SECRET_KEY / DB / Redis / backup credentials;
  cert-manager Ingress + ClusterIssuer; default-deny NetworkPolicy
  with per-component allow-lists; Pod Security Standards
  'restricted' namespace label.
- overlays/prod — production sizing (5 app / 3 pg / 3 redis,
  SSD storage, ResourceQuota + LimitRange, Let's Encrypt,
  pinned image tags).
- overlays/staging — single-replica cost-tuned variant, self-signed
  TLS, dev-mode Vault.

README documents operator prerequisites, Vault policy/KV layout,
build/apply/dry-run commands, admin promotion, MASTER_KEY rotation,
pg failover + PITR, redis failover verification, upgrade procedure,
and monitoring alerts.

kustomize build verified clean on base, prod, and staging.

mhahl-a added 1 commit

2026-08-15 08:37:50 +00:00

infra: drop Vault/ESO — operator-managed Secrets instead

CI / tox py + lint (pull_request) Has been cancelled

Details

d1b3cb89da

This application IS the external secrets store for other systems, so
its own bootstrap credentials do not need to be fetched from one.
Removed:

- deploy/base/external-secrets/ — ClusterSecretStore + 6 ExternalSecrets
- deploy/base/rbac/{external-secrets-reader SA, Role, RoleBinding}
- overlays/{prod,staging}/vault-server-patch.yaml

Replaced with:

- deploy/base/secrets/ — README + secrets.example.yaml (structural
  templates with empty stringData; commit-safe reference only)
- scripts/bootstrap-secrets.sh — generates random passwords for each
  required Secret and kubectl applies them (chmod 600 env dump with
  60s auto-delete; MASTER_KEY reminder)

Updated:

- base/kustomization.yaml — drop external-secrets, add secrets
- overlays/{prod,staging}/kustomization.yaml — drop vault-server-patch
- deploy/README.md — rewrite: remove Vault prerequisites + KV layout,
  add 'Secrets bootstrap' section, document rotation procedures
  against kubectl patch + kubectl rollout instead of ESO annotations

Required Secret names: secretserver-app-secrets,
secretserver-postgrest-secrets, secretserver-postgres-superuser,
secretserver-postgres-authenticator, secretserver-redis-secrets,
secretserver-postgres-backup (opt). kustomize build verified clean
on base, prod, staging.

mhahl-a added 1 commit

2026-08-15 08:46:25 +00:00

fix(deploy): address Kubernetes manifest review findings

CI / tox py + lint (pull_request) Has been cancelled

Details

2c32fcd35d

- Fix sentinel & failover-controller password expansion ($ vs $$)
- Drop unmounted --redis.password-file arg from redis-exporter sidecar
- Add NetworkPolicy allowing failover controller egress (DNS, sentinel, K8s API)
- Point app REDIS_URL to redis-master:6379/0 and document controller patching
- Remove empty storageClassName from base Redis PVC and Postgres cluster
- Remove invalid SET ROLE GUC from Postgres roleSettings
- Align MAX_CONTENT_LENGTH (16MB) in ConfigMap with Ingress proxy-body-size
- Remove arbitrary /29 except from probe CIDRs across all NetworkPolicies
- Restrict Postgres NetworkPolicy cnpg-system ingress with podSelector
- Set Postgres wal_level to replica and turn off connection logging
- Set Postgres podAntiAffinity to required for strict HA
- Add readOnlyRootFilesystem to Redis init container and exporter sidecar
- Remove deprecated kubernetes.io/ingress.class annotation from Ingress
- Drop non-functional PostgREST PodMonitor and app /healthz prometheus annotations
- Add missing optional UI env vars to app ConfigMap
- Add prominent staging-only warning in enable-insecure-defaults.yaml
- Update deploy/README.md layout, monitoring, and runbook

mhahl-a added 1 commit

2026-08-15 09:50:31 +00:00

feat: harden secrets UX and token policy

CI / tox py + lint (pull_request) Has been cancelled

Details

bcdefa562e

mhahl-a added 1 commit

2026-08-15 09:52:50 +00:00

feat(deploy): add operational cronjobs

CI / tox py + lint (pull_request) Has been cancelled

Details

bc67366310