The release runners cannot resolve the in-cluster Service name, so the
binary uploads died on DNS. s3.rhema.rs fronts the same Garage through
Caddy and Traefik, with its own certificate, matching the web face's
shape (Caddy terminates, Traefik routes, bucket by path).
The longhorn instance-manager and engine-image pods carry
longhorn.io/component labels, not app, so the exclusions never matched.
Kyverno stripped their cpu requests at admission, longhorn-manager saw a
spec mismatch and deleted the pod in a loop, and after any node reboot
all longhorn volumes stayed detached with every workload stuck in
ContainerCreating. Match the real labels instead.
Garage's hourly full-sync cycles were the InvalidAccessKeyId source:
while a key-table sync runs, auth lookups transiently fail, and the
edge job's prune step kept landing inside the window (one sync ran
87 minutes). Both embedded stores get a weekly interval.
The saggar-dist chart: a single-node Garage hosting the distribution
bucket — public read-only website on get.saggar.dev (the host label
is what Garage's vhost routing resolves the bucket from), S3 write
surface staying cluster-internal for the publish workflow. Same edge
path as the other saggar hosts: dns A record, cert-manager
certificate, caddy route. Chart repo and packages auth carry over
from the saggar release.
The chart's docs book (static nginx) was up in-cluster only: give its
ingress a host, with the same edge path as the api host — dns A
record, cert-manager certificate and caddy route.
index-pack on ~12M objects OOMKilled the 1Gi container mid push, and
500m of CPU stretched the receive for many minutes before that.
Bump to 6Gi / 4 cores; the node has ~40Gi free.
Traefik v3 defaults respondingTimeouts.readTimeout to 60s counted
over the whole request body, so large HTTP git pushes get a 504 the
moment they pass one minute. writeTimeout stays unset: a cap there
would cut long-lived SSE streams mid-flight.
Flux HelmRelease of the saggar hub (chart saggar-hub >=0.2.0-0,
CI-published on the gitea saggar org, appVersion pinned to the image
commit) with the web UI. Engine on api.saggar.dev for the CLI and the
docker/apt registries, UI on saggar.dev — new cert, caddy route and
dns A record for the api host, TLS terminating at Caddy as usual.
Login via the home Gitea as an OIDC provider; private registry and
helm repo auth share one packages token.
Flux HelmRelease tracking the CI-published chart (>=0.4.0-0, timestamped
versions, appVersion pinned to the image commit). Ingress host
pkgatlas.vhaudiquet.fr via traefik; TLS terminates at Caddy with the
*.vhaudiquet.fr wildcard, so the new @pkgatlas route proxies to Traefik
like the other public vhosts.
The operator's managed policy is default-deny with egress limited to DNS
and TCP 443, so reaching a LAN service on any other port is dropped.
Measured against 10.4.1.222 from inside the pod: 443 returns RST while
7020 times out, so routing is fine and the port is what gets refused.
Add a supplementary egress policy admitting 10.0.0.0/8, excluding the
service (10.96.0.0/12) and pod (10.244.0.0/16) CIDRs so this does not
grant every pod a path out. No port list: the point is arbitrary LAN
services. Registered in the directory kustomization alongside the
existing mail and radicale egress policies.
- Pin gateway image to immutable tag 2026.9.3: the stale node-cached
'latest' (old build, state schema v1) crash-looped against the PVC's
DB already migrated to schema v15/16.
- Add fix-data-owner init container: the operator mounts the PVC root
at /home/openclaw/.openclaw with fsGroup only, leaving the mount root
owned by root; OpenClaw >= 2026.9 fchmods its config dir on write and
fails with EPERM (paperclipinc/openclaw-operator#607).
- Ran one-time 'openclaw doctor --fix --non-interactive' against the
PVC to complete the 2.0 workspace-state migration.
vhaudiquet
renovate[bot]andvhaudiquet
kosmosandvhaudiquet