/api/ready → 503 with reasons: ["coordinator"] on one instance |
the coordinator could not start (first-boot DDL race, DB unreachable); it retries with back-off |
check CLUSTER: coordinator start failed in the logs; it recovers by itself once the DB answers |
reasons: ["plugins:converging"] for minutes on a new instance |
it is still downloading/installing the cluster's plugins (pip into vendor/); a repository may be unreachable |
log lines RECONCILE: installing …; it turns ready when done or after LYNDRIX_READY_PLUGIN_WAIT_SECONDS (then preflight plugins: WARN) |
A plugin is missing on one instance after it started (404 on its routes), log RECONCILE: install of '<id>' failed … |
the download or the dependency install (pip) failed on that instance. A failure of the network or the package index (… the package index or the network did not answer — retry n/4 in … s) is retried after 30, 60, 120 and 300 s, and the instance stays plugins:converging while such a retry is outstanding (at most LYNDRIX_READY_PLUGIN_WAIT_SECONDS). Any other failure — and a transient one after those four retries — backs off for 10 min, doubling up to 24 h (failed n× — next attempt in … min) |
transient: nothing, it retries by itself. Permanent: read the PIP_ERROR: / INSTALL_ERROR: line before it, fix the cause; a restart of the instance retries at once. An internal mirror (LYNDRIX_PLUGIN_PIP_INDEX_URL) removes the dependency on PyPI |
reasons: ["heartbeat"] |
no heartbeat reached the database for a full lease TTL (dead pooled sockets after a VIP move, a starved event loop/executor, the DB unreachable from this host only) |
GET /api/cluster on that instance → coordinator.heartbeat (failures, last_error, last_duration_s); it clears with the next successful beat. Peers showed the instance as stale during that time |
Preflight storage: WARN — /data/storage is NOT shared with <instance> (api) |
that instance mounts a different directory (another host without the shared volume) |
true finding: files uploaded through it are missing on the others. Mount the shared volume there or do not route users to it; a runner-only instance is exempt (LYNDRIX_STORAGE_SHARED_ROLES) |
Preflight storage: WARN — … is not responding — a storage job has been stuck for N s |
the STORAGE_DIR mount hangs (NFS server / share-manager restart) |
fix the mount; the core keeps one thread waiting on it and skips further marker writes meanwhile (GET /api/cluster → storage). Requests that touch files block until it is back |
reasons: ["preflight:storage_secret"] on one instance |
its STORAGE_SECRET differs from the value most instances share |
set the cluster's value on that instance; the others stay ready and only show storage_secret: WARN naming it. To rotate the secret restart all instances together |
Preflight roles: FAIL — no live instance has role 'runner' |
the IaC Orchestrator is pinned to runner and no live instance has it |
start the runner instance / add runner to one instance's LYNDRIX_INSTANCE_ROLES |
| IaC Orchestrator: "No runner instance online" right after a core update |
core ≤ 1.11: the runner's graceful stop deleted its registry row at once |
fixed in 1.12 (the runner shows as restarting); if it still appears, the runner took longer than LYNDRIX_RESTART_GRACE_SECONDS to come back or was killed without a drain (stop_grace_period too short) |
reasons: ["instance_id_conflict"], log CRITICAL … instance id '…' is also used by a process that started later |
two running processes share one LYNDRIX_INSTANCE_ID (copied compose file, a Deployment with a fixed id and maxSurge) |
give every instance its own id (Kubernetes: the pod name); the older process stays out of rotation and holds no leases until the other is gone |
| Plugin page shows a plugin as Remote |
it is pinned:<role> and this instance lacks the role |
expected; route its paths to the pinned instance at the balancer |
LEASE: '…' was not renewed for 22.5s (ttl 30s) — treating it as lost … |
the instance could not renew the lease (database unreachable or too slow for it) |
fencing worked as designed — it stops 7.5 s before a peer can take over; the work moves to a peer after the TTL; check DB connectivity of that host |
| Two instances probe monitors / duplicate state-change events |
both hold work that should be leased |
a plugin that is not cluster-aware yet, a lost callback that does not stop the work, or an instance that was paused past the TTL (the plugin should fence its writes with ctx.lease_token); check Settings → Cluster → Leases |
(1020, "Record has changed since last read") in plugin logs |
MariaDB ≥ 11.6 snapshot isolation + concurrent writers |
the core coordinator uses READ COMMITTED + retry; plugins must do the same for maintenance jobs (see the plugin development page) |
(1050, "Table 'cluster_instances' already exists") once at first boot |
two instances created the tables together |
tolerated; the second instance logs it and continues |
| Browser keeps the old leader's SSE stream after a stop |
the drain closed it with event: reconnect; EventSource reconnects in 1–3 s |
nothing — if streams never reconnect, the balancer buffers responses (fix flushInterval/proxy_buffering) |
| Login limiter blocks everyone after a few failures |
the core sees the balancer's IP |
set LYNDRIX_TRUSTED_PROXIES (preflight trusted_proxies warns) |
| Mixed versions after an update |
an instance did not restart |
finish the rolling update; preflight versions is warn until then |
| Boot screen shows Cluster coordinator: Starting… forever |
DB connected but the coordinator keeps failing |
logs of that instance; typically credentials or max_statement_time too low for GET_LOCK |
UI answers 502 on every /api/* (Kubernetes), nginx log "could not be resolved" |
the UI's nginx resolves the backend via LYNDRIX_DNS_RESOLVER (default: Docker's 127.0.0.11) and ignores search domains |
set the kube-dns IP and the Service FQDN in LYNDRIX_BACKEND_URL (manifests) |
| Vault pod crash-loops with "Failed to lock memory" (Kubernetes) |
non-root container without ambient IPC_LOCK |
disable_mlock: true in the Vault config |
First-boot plugin installs abort with DetachedInstanceError |
core < 1.4.1 |
upgrade; the plugins were still activated, only the post-install verification was skipped |
| One plugin install shows N× "installed" toasts and N² bell entries |
core < 1.4.5: every instance's reconcile install notified, and reactions to relayed events were bridged again |
upgrade — reconcile installs are silent and derived events stay local |