Operations — cluster status and preflight¶
The Cluster page¶
Settings → Cluster (needs admin:read) shows, for the instance that answered the request:
- This instance: id, host, roles, leader marker, ready/not ready with reasons, draining, uptime, coordinator driver. In single mode a hint explains how to enable cluster mode.
- Instances: every row of
cluster_instanceswith roles, core/API version, last heartbeat, alive/stale, ready/draining; the leader carries a crown. The API additionally reportsstate(alive,draining,restarting,dead),incarnationandstopped_atper instance and the totals instates. - Leases: name, holder, time to expiry; Release (
admin:write) hands a lease over — the peers acquire it within a second. Use it to move the leader or a plugin's background work. - Plugin placement: placement and status per plugin, where its lease is held,
remotemarkers for pinned plugins. - Preflight: every check with status, message and hint; Re-run checks evaluates them now.
- Coordinator: TTL, heartbeat, poll interval, bridge counters (published/relayed; the API
also has
outbox,failures,dropped,expired,oversized), drain, trusted proxies, database version.
The page refreshes on cluster:changed / cluster:leader_changed over SSE and every 15 s.
The API¶
curl -s -H "Authorization: Bearer $TOKEN" https://lyndrix.example.com/api/cluster | jq
curl -s -H "Authorization: Bearer $TOKEN" https://lyndrix.example.com/api/cluster/preflight | jq '.checks[] | select(.status != "ok")'
curl -s -X POST -H "Authorization: Bearer $TOKEN" https://lyndrix.example.com/api/cluster/leases/core:leader/release
/api/cluster is answered by whichever instance the balancer picks; instance.id tells you
which. Hit an instance directly (host:8081) to inspect that one.
Boot screen¶
In cluster mode the boot screen (React and NiceGUI) lists four extra rows: coordinator (driver,
connecting, failed), leader election (this instance / <id> / electing), cluster peers (how
many others are alive) and shared storage (not available in this version).
Logs to know¶
| Line | Meaning |
|---|---|
CLUSTER: mode=cluster driver=db instance=… roles=… |
startup summary |
COORDINATOR: db driver started … |
tables ready, event polling from id N |
LEASE: acquired 'core:leader' / LEASE: lost '…' to a peer |
lease transitions |
LEASE: '…' was not renewed for 22.5s (ttl 30s) — treating it as lost before a peer can take it |
coordinator unreachable → the holder fenced itself ahead of the TTL |
LEASE: '…' came back under a new token — the holding was interrupted |
the row was force-released or expired while this instance believed it held it; lost fired, the next tick re-acquires |
PREFLIGHT [storage_secret] FAIL: … |
contract violation; instance stays out of rotation |
CLUSTER: heartbeat failed (N×, …) / heartbeat recovered after … |
the registry row could not be written; after a full TTL /api/ready answers 503 heartbeat |
COORDINATOR: database connection pool refreshed (heartbeat timed out) |
a call hung on a dead database socket; the next call opens new connections |
CLUSTER: storage job 'marker' has not returned for 120s … |
the STORAGE_DIR mount hangs; marker writes are skipped until it answers |
DRAIN: SIGTERM received — /api/ready → 503, draining for 10s |
graceful stop in progress |
RECONCILE (peer dev-a: installed): {'loaded': […]} |
plugin artifacts converged after a peer's install |