Operations

Operations

Day-to-day Helm release commands, environment overlay strategy, chart version pinning, and the OCI publish flow.

Helm release commands

All commands assume the GHCR registry login is active and helm dep update has been run.

Install / upgrade

# Dev (Kind or staging cluster)
helm upgrade --install agent-manager helm/agent-manager \
  -f helm/agent-manager/values-dev.yaml \
  --namespace paper-board --create-namespace \
  --wait --timeout 10m

# Production
helm upgrade --install agent-manager helm/agent-manager \
  -f helm/agent-manager/values-prod.yaml \
  --namespace paper-board --create-namespace \
  --wait --timeout 10m

--wait blocks until all Deployments, StatefulSets, and Jobs (including migrator hook Jobs) report ready.

Diff before upgrade

helm diff upgrade agent-manager helm/agent-manager \
  -f helm/agent-manager/values-prod.yaml

Requires the helm-diff plugin (helm plugin install https://github.com/databus23/helm-diff).

Rollback

helm rollback agent-manager 1   # roll back to revision 1
helm history agent-manager      # list revisions

Note: Helm rollback does not run down migrations. Schema rollback requires a manual migrator down N invocation against the affected service’s MIGRATION_DB_URL.

Uninstall

helm uninstall agent-manager --namespace paper-board

This does not delete PersistentVolumeClaims. Delete them manually if a full teardown is needed.

Environment overlays

The umbrella chart uses a three-file layering strategy:

File	Purpose
values.yaml	Base defaults. Every configurable key must appear here.
values-dev.yaml	Overrides for local Kind / staging: smaller PVCs, Always pull policy, dev KEK, subcharts enabled.
values-prod.yaml	Overrides for production: larger PVCs, IfNotPresent pull policy, secrets.create: false (ESO owns Secrets).

Always pass exactly one overlay file with -f. Never merge ad-hoc --set flags into production releases; encode them as a new values file or a values-.yaml change committed to this repo.

Secret management

In dev, secrets.create: true lets the umbrella render Secret objects from values (acceptable for local development with non-sensitive placeholder keys).

In production, secrets.create: false on every service block. Secrets are owned by External Secrets Operator (ESO) or Vault — Phase 5+ infra. The umbrella chart still declares the Secret name expected by each Deployment; ESO must provision it before helm upgrade runs.

Chart version pinning

The umbrella’s Chart.yaml pins each OCI subchart to an exact SemVer:

dependencies:
  - name: identity
    version: 0.2.1
    repository: oci://ghcr.io/paper-board/helm
    condition: identity.enabled

Chart.lock records the resolved digest. Both files are committed. helm dep update is the only command that regenerates Chart.lock.

Bumping a subchart version

1. Update the version field in Chart.yaml for the relevant dependency.
2. Run helm dep update helm/agent-manager — this pulls the new OCI artifact and updates Chart.lock.
3. Commit both Chart.yaml and Chart.lock together.
4. Open a PR; CI lints and renders the new version before merge.

Renovate is configured to auto-open PRs when a new OCI chart tag appears at ghcr.io/paper-board/helm/<svc>.

OCI chart publish flow

Each service repo publishes its Helm chart as an OCI artifact to GHCR on every release tag. The pattern (identical across all services):

# .github/workflows/release.yml (inside each service repo)
- name: Package chart
  run: helm package helm/<svc> --destination /tmp/charts

- name: Push chart to GHCR
  run: |
    helm push /tmp/charts/<svc>-*.tgz oci://ghcr.io/paper-board/helm

GHCR package visibility is set to Internal so sibling-repo CI can pull with GITHUB_TOKEN (permissions: packages: read). Each service package must list the paper-board/infra repo under “Manage Actions access” → Read, otherwise helm dep update in infra CI fails with a 403.

Version alignment

Service chart versions track the service’s application version (appVersion in Chart.yaml). Infra pins to the exact chart version, not a range. Patch bumps in the service are independent of umbrella releases; only the pinned version line in helm/agent-manager/Chart.yaml determines what the umbrella deploys.

Postgres schema operations

The migrator Job runs automatically on every helm upgrade --install. For manual operations:

# Check current migration version for a service
kubectl run -it --rm migrator-check \
  --image=ghcr.io/paper-board/<svc>-migrator:<tag> \
  --restart=Never \
  --env="MIGRATION_DB_URL=<port-5432-dsn>" \
  -- version

# Apply pending migrations manually
kubectl run -it --rm migrator-up \
  --image=ghcr.io/paper-board/<svc>-migrator:<tag> \
  --restart=Never \
  --env="MIGRATION_DB_URL=<port-5432-dsn>" \
  -- up

# Roll back one migration (dev only)
kubectl run -it --rm migrator-down \
  --image=ghcr.io/paper-board/<svc>-migrator:<tag> \
  --restart=Never \
  --env="MIGRATION_DB_URL=<port-5432-dsn>" \
  -- down 1

Always use MIGRATION_DB_URL (port 5432, direct Postgres) — never DATABASE_URL (port 6432, PgBouncer) for migrator commands.