Docker — rolling updates¶
With two or more instances behind a load balancer a core update needs no downtime.
Sequence¶
- Runner first (if you have one): its plugin routes are pinned there anyway, so a short
absence is visible only in the IaC Orchestrator UI.
docker compose pull && docker compose up -don the runner host; wait forGET /api/ready→ 200. - API instances one by one: pull,
up -d, wait for/api/readyon that host, then the next. During the window the preflight showsversions: warn(mixed versions) — expected. - Verify in Settings → Cluster: every instance alive, one version, preflight without
FAIL.
What happens on each docker stop:
- SIGTERM → at once
/api/readyanswers 503, the peers see the instance asdrainingand its leases are handed over (a peer takes the leader lease and plugin leases within about a second) → the balancer removes the instance after its next check (3–5 s) → afterLYNDRIX_DRAIN_SECONDSthe instance closes SSE streams withevent: reconnect(browsers reconnect through the balancer), leaves its tombstone → uvicorn shuts down. - While the container restarts, peers show it as
restarting(not gone) forLYNDRIX_RESTART_GRACE_SECONDS(90 s); pinned plugins answer 503 withRetry-After: 5. stop_grace_periodmust be larger than drain + lease hand-over + graceful shutdown or Docker kills the process: 30 s for the default 10 s drain (10 s drain + up to 5 s forlostcallbacks + uvicorn's 5 s + margin). Docker's default is 10 s — exactly the drain — so without it the process is killed before it can leave its tombstone (the peers then still treat it asrestarting, because it had announced the drain).
Schema changes¶
DDL runs at boot under GET_LOCK('lyndrix_bootstrap'): the first new instance applies additive
changes while the old ones keep running. Only additive migrations (new tables, new nullable
columns) are safe for a rolling update — that is what the core and the shipped plugins do.
Rolling restart without an update¶
POST /api/system/restart (Settings → System Info → Restart) in cluster mode starts a
sequential restart driven by the leader: it tells one instance to drain and exit (the restart
policy brings it back), waits until that instance is ready again as a new incarnation, then
continues with the next; the leader restarts itself last. At most one instance is away at any
time. If an instance is not back within LYNDRIX_RESTART_INSTANCE_TIMEOUT_SECONDS (300 s) the run
is aborted and the remaining instances are left alone. Instances with the runner role are
skipped (a restart orphans their running jobs) unless the request says
?include_runners=true; instances still on core ≤ 1.11 are skipped as well. The request is
answered with 409 when no leader can coordinate it (none elected, a run already active, or the
leader itself still on an old core). Progress: GET /api/cluster → restart on the leader. No
Docker socket is needed on the API instances. (Up to core 1.11 this endpoint made every instance
exit 5 s apart — effectively all at once.)
Rollback¶
Re-deploy the previous image tag with the same sequence. Database changes are additive, so an older core runs against a newer schema.