🚀 Get to pre-production in weeks, not months, with private training direct from Jube’s developer — real sovereignty, zero vendor lock-in.

Jube Cluster Monitoring Reference

Host-level CPU, RAM, and disk metrics are assumed to be collected at the node level and are not covered here. This document covers the checks you run inside the stack itself:

  1. Per-container resource checks — CPU and memory via docker stats.
  2. Port availability — TCP-level probes confirming services are bound and accepting connections.
  3. Application health endpoints — HTTP-level probes confirming services are operational.
  4. HAProxy traffic monitoring — frontend byte-rate and status checks.
  5. PostgreSQL internals — connections, replication lag, long-running queries.
  6. Redis internals — memory pressure, replication offset, keyspace hit rate.
  7. Patroni internals — cluster state, leader presence, timeline divergence.
  8. Sentinel internals — master reachability, quorum, address consistency.
  9. Jube application monitors — queue saturation, engine liveness, response time, elevation rate.
  10. Container log monitoring — STDERR for Jube, STDOUT for infrastructure, restart detection.

All checks are bare shell commands. Wire them into whatever collection or alerting mechanism you use.

Superseded in Large Part by Jube’s Own Observability Suite

This page predates Jube’s Infrastructure Health Metrics suite and the Jube.Monitoring sidecar, both of which now collect most of sections 2, 3.2, 6, 7, 8, 9, 11 and 12 automatically, once a minute, into queryable Postgres tables – no docker exec, no manual psql/redis-cli/etcdctl invocation, no bespoke cron job. Every table is browsable via its own GET /api/<Table> endpoint (see Infrastructure Health Metrics and HTTP Asynchronous Model Invocation), and the same signal is additionally exported as OpenTelemetry metrics (jube.engine.stage.duration and friends) whenever EnableOpenTelemetry is set, so it lands in whatever Grafana/Prometheus-compatible backend you already point the rest of your stack at rather than requiring a separate collection path just for Jube.

This page is kept, deliberately, not deprecated. The shell commands below remain the right tool in two cases this automation doesn’t reach: a zero-dependency pre-flight check you can run before trusting Jube’s own database-backed instrumentation is even up, and anything Jube’s tables don’t carry (table bloat, section 6.5, is the one gap of substance below). HAProxy’s own frontend/traffic stats (section 5) and frontend port reachability (section 3.3) – previously the two gaps of substance here – are now covered too, see the table below.

Section Now automated by Still worth running manually for
§2 Per-container resource checks DockerContainerMetric / DockerHostMetric (Jube.Monitoring sidecar, one instance per host) An immediate point-in-time number without waiting a minute
§3.2 etcd deep health (etcdctl endpoint health) EtcdMemberStatus (raft/proposal detail etcdctl doesn’t surface) + EtcdClusterEvent Pre-flight check before etcd is discoverable by Jube
§3.3 HAProxy frontend ports HAProxyReachabilityProbe – a real GET /api/ready through HAProxy for jube_ui/jube_api, a bare TCP connect for Postgres, against every individual HAProxy replica (tasks.haproxy, bypassing the VIP) An immediate point-in-time check without waiting a minute
§5 HAProxy traffic monitoring HAProxyServerStatus – one row per backend server slot per minute (status, resolved address, check failures/down-transitions, sessions, HTTP response codes), scraped from HAProxy’s own stats CSV §5.5 the HAProxy Runtime API stats socket specifically – not scraped, this sidecar only reads the stats CSV over HTTP
§6 PostgreSQL internals PostgresMetric (connections, cache hit ratio, WAL, longest-running query) + PostgresReplicationStatus (per-replica lag) §6.5 table bloat – not collected anywhere, still a manual weekly job
§7 Redis internals RedisMetric (memory, hit rate, evictions) + RedisConnectionMultiplexerMetric/RedisConnectionEvent (the application’s own connection health) —
§8 Patroni internals PatroniMemberStatus (topology, lag, timeline) + PatroniClusterEvent (role/state/timeline diffs, plus an authoritative Failover log read from Patroni’s own DCS history – something no polling script can reconstruct after the fact) —
§9 Sentinel internals RedisSentinelStatus (topology, per-replica lag) + RedisSentinelEvent (every Sentinel pub/sub event captured the instant it happens, not just whatever state polling catches between samples) —
§11 Jube application monitors Already Jube’s own counters before this suite existed; now additionally exported as OpenTelemetry metrics, plus finer per-stage/per-task breakdowns (EntityAnalysisModelStagePerformanceCounter, TaskPerformanceCounter, ResponseTimePipelineCounter) than the five queries below show —
§12.1 STDERR — Jube containers ApplicationLogEntry – every WARN/ERROR/FATAL logged anywhere in the process, captured centrally and queryable, not grepped per-container —
§12.3 Restart detection (Jube/Docker containers) DockerContainerMetric.RestartCount/OomKilled/ExitCode Non-Docker or non-Jube-monitored hosts

Sections 1, 3.1, 3.4 (as a live check; superseded as data), 3.5 (ports only), 4, 10, and 12.2 (HAProxy’s own log lines specifically) have no Jube-side equivalent and are unchanged by any of this – keep running them as written.

New ground with no prior section to supersede: OverlayNetworkTaskDrift checks something this page never covered at all – whether Docker’s embedded DNS (what server-template/tasks.<service> resolution actually returns) agrees with what the Swarm API says is really running, for every service in the topology. A non-empty AddressesOnlyInDns/AddressesOnlyInSwarm is a stale DNS answer or a not-yet-caught-up new task, decoupled entirely from whether any health check has noticed yet – see Infrastructure Health Metrics.

More new ground, added since, with no shell-probe equivalent below either:

  • PostgresStatementStatistics – per-query-shape aggregate statistics (pg_stat_statements: calls, total/mean/ min/max/stddev execution time, cache-hit vs disk-read bytes, temp-file spill, WAL) accumulated across all history, not just the currently-running queries §6 already covers – answers “which query shape is worst overall” rather than “what is running right now”. See Infrastructure Health Metrics.
  • RedisCallCounter – per-minute call counts/timings by Jube.Cache repository method, pinpointing exactly which area of the system is generating Redis load, rather than only Redis’ own aggregate memory/hit-rate figures in §7. See Infrastructure Health Metrics.
  • OtlpDispatchCounter – per-minute success/failure/dropped counts for this instance’s own outbound OTLP export attempts, so a dead or overwhelmed OpenTelemetry backend is visible from a table rather than only from logs. See OTLP Dispatch Counter.
  • ArchiverStagePerformanceCounter/ArchiverWarning, CaseCreationStagePerformanceCounter/CaseCreationWarning, ModelInvokeWarning, CaptureQueueHealth – finer-grained stage breakdowns and a database-backed warn-threshold record for the two background pipelines §11 only measures in aggregate (Archive/case-creation queue depth), plus a health check on the bounded in-memory queues that capture those warnings. See Background Processing Performance.

All of the above are also summarised, with a brief purpose statement each, on Performance Counter Balance.

1. Host-Level Resource Note

Suggested baseline thresholds for node-level collection:

Metric Warning Critical
CPU utilisation (5-min avg) > 70% > 90%
Memory utilisation > 75% > 90%
Disk utilisation > 75% > 90%

2. Per-Container Resource Checks

Collected automatically, every minute, by DockerContainerMetric/DockerHostMetric (the Jube.Monitoring sidecar, one instance per Docker host) – see Superseded in Large Part by Jube’s Own Observability Suite above. docker stats remains useful for an immediate point-in-time number without waiting for the next sample.

Run from the Swarm manager:

# All containers
docker stats --no-stream --format \
  "container= cpu= mem= mem_usage="

# Specific container
docker stats --no-stream --format \
  "container= cpu= mem=" \
  $(docker ps -qf "name=<container-name>")

Suggested thresholds:

Container type CPU alert Memory alert
Database (Patroni/PostgreSQL) > 75% sustained > 85% of limit
Cache (Redis master) > 60% sustained > 80% of limit
Consensus (etcd) > 50% sustained > 70% of limit
Application (API / engine) > 80% sustained > 85% of limit
Application (UI / frontend) > 60% sustained > 75% of limit
Background jobs / workers > 70% sustained > 80% of limit

3. Port Availability Checks

TCP probes run from inside a container on the same overlay network. If nothing else, monitor port connectivity — it is the fastest indicator of a dead service.

3.1 Generic TCP Port Probe 

docker exec <container> \
  sh -c 'nc -z -w 3 <hostname> <port> && echo "OK:<hostname>:<port>" || echo "FAIL:<hostname>:<port>"'

3.2 etcd Client and Peer Ports 

# Repeat for each etcd node
docker exec $(docker ps -qf "name=etcd1") \
  sh -c 'nc -z -w 3 etcd1 2379 && echo "OK:etcd1:2379" || echo "FAIL:etcd1:2379"'

docker exec $(docker ps -qf "name=etcd1") \
  sh -c 'nc -z -w 3 etcd1 2380 && echo "OK:etcd1:2380" || echo "FAIL:etcd1:2380"'

Collected automatically, every minute, by EtcdMemberStatus (raft/proposal/alarm detail etcdctl endpoint health doesn’t surface) and EtcdClusterEvent – see Superseded in Large Part by Jube’s Own Observability Suite above. etcdctl remains the right pre-flight check before etcd is discoverable by Jube at all.

Deep cluster health (run from inside any etcd container):

docker exec $(docker ps -qf "name=etcd1") \
  etcdctl \
    --endpoints=http://etcd1:2379,http://etcd2:2379,http://etcd3:2379,http://etcd4:2379,http://etcd5:2379 \
    endpoint health --write-out=table

Alert if any member reports unhealthy or if fewer than 3 members are reachable.

3.3 HAProxy Frontend Ports 

# PostgreSQL primary (read-write)
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'nc -z -w 3 haproxy 5432 && echo "OK:haproxy:5432" || echo "FAIL:haproxy:5432"'

# PostgreSQL replica (read-only)
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'nc -z -w 3 haproxy 5433 && echo "OK:haproxy:5433" || echo "FAIL:haproxy:5433"'

# HAProxy stats / admin
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'nc -z -w 3 haproxy 7000 && echo "OK:haproxy:7000" || echo "FAIL:haproxy:7000"'

3.4 Patroni REST API

Each Patroni node exposes a health endpoint on port 8008. No database credentials required.

# Repeat for each Patroni node
docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -qO- --timeout=5 http://patroni1:8008/health 2>/dev/null || echo "FAIL:patroni1:8008"'

Expected: {"state":"running","role":"master"} for the primary; {"state":"running","role":"replica"} for standbys. Alert if no node reports master.

3.5 Redis and Sentinel Ports 

# Redis master
docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'nc -z -w 3 redis-master 6379 && echo "OK:redis-master:6379" || echo "FAIL:redis-master:6379"'

# Redis replicas — repeat for each
docker exec $(docker ps -qf "name=redis-replica1") \
  sh -c 'nc -z -w 3 redis-replica1 6379 && echo "OK:redis-replica1:6379" || echo "FAIL:redis-replica1:6379"'

# Redis PING — confirms the instance is serving commands, not just accepting TCP
docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" PING 2>/dev/null'
# Expected: PONG

# Sentinel ports — repeat for each
docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'nc -z -w 3 sentinel1 26379 && echo "OK:sentinel1:26379" || echo "FAIL:sentinel1:26379"'

# Sentinel master discovery
docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL masters 2>/dev/null | grep -c "name" && echo "OK:sentinel-has-master" || echo "FAIL:sentinel-no-master"'

Alert if fewer than 3 sentinels are reachable.

4. Application Health Endpoint Checks

A passing port check with a failing health endpoint means the process is bound but broken.

# Generic HTTP health check
docker exec <container> \
  sh -c 'wget -qO- --timeout=5 http://localhost:<port><path> 2>/dev/null \
    && echo "OK:<service>:ready" || echo "FAIL:<service>:ready"'

For multi-replica services, iterate over all instances:

for CONTAINER in $(docker ps -qf "name=<service-name>"); do
  RESULT=$(docker exec "$CONTAINER" \
    sh -c 'wget -qO- --timeout=5 http://localhost:<port><path> 2>/dev/null \
      && echo OK || echo FAIL')
  echo "$CONTAINER:<service-name>:$RESULT"
done

For services with no HTTP endpoint, fall back to process liveness:

docker exec $(docker ps -qf "name=<container>") \
  sh -c 'pgrep -f "<process-name>" > /dev/null \
    && echo "OK:<container>:process" || echo "FAIL:<container>:process"'

5. HAProxy Traffic Monitoring

5.1 Enable the Stats Endpoint

Ensure haproxy.cfg includes:

frontend stats
    bind *:7000
    stats enable
    stats uri /stats
    stats refresh 10s

Optionally add a Unix socket for socat-based queries:

global
    stats socket /var/run/haproxy/admin.sock mode 660 level admin

5.2 Pull Full Stats CSV

HAProxy exposes machine-readable stats at /stats;csv:

docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null'

Key CSV columns (0-indexed):

Index Field Meaning
0 pxname Proxy name
1 svname FRONTEND, BACKEND, or server name
4 scur Current sessions
7 stot Total sessions (cumulative)
8 bin Bytes in (cumulative)
9 bout Bytes out (cumulative)
17 status OPEN, UP, DOWN, MAINT, etc.
48 req_tot Total HTTP requests (frontends only)

5.3 Frontend Status 

# List all frontends and their status
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" {print $1, $18}"'

# Alert on any frontend not OPEN
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" && $18!=\"OPEN\" {print \"ALERT:frontend-not-open:\", $1, $18}"'

5.4 Frontend Byte Counters 

# Snapshot bytes-in, bytes-out, and status for all frontends
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" {printf \"frontend=%s bin=%s bout=%s status=%s\n\", $1, $9, $10, $18}"'

# Frontends with absolute zero bytes (useful at startup or after a reset)
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" && $9==\"0\" && $10==\"0\" {print \"WARN:zero-bytes:\", $1}"'

# Frontends with zero sessions
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" && $8==\"0\" {print \"WARN:zero-sessions:\", $1}"'

Cumulative counters that have not advanced between two scrapes indicate a frontend processing no traffic. Take two snapshots separated by your collection interval and compare.

5.5 Via Unix Socket 

# Full stats dump
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'echo "show stat" | socat stdio /var/run/haproxy/admin.sock'

# Runtime info (uptime, current connections)
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'echo "show info" | socat stdio /var/run/haproxy/admin.sock'

# Frontend rows only
docker exec $(docker ps -qf "name=haproxy") \
  sh -c 'echo "show stat" | socat stdio /var/run/haproxy/admin.sock \
    | awk -F, "$2==\"FRONTEND\" {print $1, $9, $10, $18}"'

5.6 Delta Check Script

Takes two snapshots and reports whether each frontend advanced its byte counter:

#!/usr/bin/env bash

INTERVAL=60  # seconds between samples
SNAP1=$(mktemp)
SNAP2=$(mktemp)

get_frontend_bytes() {
  docker exec $(docker ps -qf "name=haproxy") \
    sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
      | awk -F, "$2==\"FRONTEND\" {print $1, $9}"'
}

get_frontend_bytes > "$SNAP1"
sleep "$INTERVAL"
get_frontend_bytes > "$SNAP2"

while IFS=' ' read -r name bytes1; do
  bytes2=$(grep "^$name " "$SNAP2" | awk '{print $2}')
  if [ -n "$bytes2" ] && [ "$bytes2" -gt "$bytes1" ] 2>/dev/null; then
    echo "OK:$name:traffic_active"
  else
    echo "WARN:$name:no_bytes_in_window"
  fi
done < "$SNAP1"

rm -f "$SNAP1" "$SNAP2"

6. PostgreSQL Internal Monitors

Collected automatically, every minute, by PostgresMetric and PostgresReplicationStatus – see Superseded in Large Part by Jube’s Own Observability Suite above. Table bloat (§6.5) is the one query below with no automated equivalent.

Run from inside any Patroni container (which includes psql). All queries connect as the postgres superuser.

The principle: connection saturation, replication lag (especially), and runaway queries cover the majority of production incidents.

6.1 Connection Saturation

Connection exhaustion causes all new connections to fail immediately — instant total outage from the application’s perspective.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT
      count(*)                                                                     AS total_connections,
      (SELECT setting::int FROM pg_settings WHERE name = 'max_connections')       AS max_connections,
      round(100.0 * count(*) /
        (SELECT setting::int FROM pg_settings WHERE name = 'max_connections'), 1) AS pct_used
    FROM pg_stat_activity
    WHERE state IS NOT NULL;
  "

Thresholds: warn at 75%, critical at 90%.

Connections waiting on locks — non-zero over several minutes signals contention:

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT count(*) AS waiting_on_lock
    FROM pg_stat_activity
    WHERE wait_event_type = 'Lock';
  "

6.2 Replication Lag

Run on the primary. Standbys return no rows from pg_stat_replication.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT
      application_name,
      state,
      pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
      sync_state
    FROM pg_stat_replication
    ORDER BY lag_bytes DESC;
  "

Alert if lag_bytes exceeds your RPO tolerance (e.g. warn > 64 MB, critical > 256 MB). Alert if the query returns zero rows — no standbys connected.

6.3 Long-Running Queries

Queries running beyond a threshold are a lock pile-up precursor.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT
      pid,
      now() - query_start AS duration,
      state,
      left(query, 120) AS query_snippet
    FROM pg_stat_activity
    WHERE state != 'idle'
      AND query_start < now() - interval '5 minutes'
      AND query NOT ILIKE '%pg_stat_activity%'
    ORDER BY duration DESC;
  "

Alert if any query exceeds 10 minutes. Tune the interval to your workload.

6.4 Idle Transactions

Open idle-in-transaction connections hold locks and block autovacuum.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT count(*) AS idle_in_transaction
    FROM pg_stat_activity
    WHERE state = 'idle in transaction'
      AND state_change < now() - interval '2 minutes';
  "

Alert if count > 0 sustained for more than 5 minutes.

6.5 Table Bloat (Weekly Check)

Not a real-time alert — run as a scheduled job. High bloat means autovacuum is not keeping up.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT
      schemaname,
      tablename,
      n_dead_tup,
      n_live_tup,
      round(100.0 * n_dead_tup / nullif(n_live_tup + n_dead_tup, 0), 1) AS dead_pct,
      last_autovacuum
    FROM pg_stat_user_tables
    WHERE n_dead_tup > 10000
    ORDER BY dead_pct DESC
    LIMIT 20;
  "

Alert if any table exceeds 20% dead tuples and last_autovacuum is more than 24 hours ago.

7. Redis Internal Monitors

Collected automatically, every minute, by RedisMetric (plus RedisConnectionMultiplexerMetric/RedisConnectionEvent for the application’s own connection health) – see Superseded in Large Part by Jube’s Own Observability Suite above.

All redis-cli commands require the password from the secret store.

The key signals: memory headroom, replication offset divergence, and whether the eviction policy is silently discarding data.

7.1 Memory Pressure 

docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" INFO memory 2>/dev/null \
    | grep -E "used_memory_human|used_memory_peak_human|maxmemory_human|mem_fragmentation_ratio|maxmemory_policy"'
Field Alert condition
used_memory / maxmemory Warn > 75%, critical > 90%
mem_fragmentation_ratio Warn if > 1.5 (fragmentation) or < 1.0 (swapping)
maxmemory_policy Alert if allkeys-lru or allkeys-random on a persistence store — silent data loss under pressure

Numeric ratio for scripting:

docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" INFO memory 2>/dev/null \
    | awk -F: "/^used_memory:/{used=$2} /^maxmemory:/{max=$2} END {
        if (max > 0) printf \"%.1f%%\n\", used/max*100
        else print \"no_limit\"
      }"'

7.2 Replication Offset Divergence

Run on each replica.

docker exec $(docker ps -qf "name=redis-replica1") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" INFO replication 2>/dev/null \
    | grep -E "role|master_host|master_link_status|master_last_io_seconds_ago|slave_repl_offset|master_repl_offset"'

Alert conditions:

  • master_link_status: down — replica has lost contact with master.
  • master_last_io_seconds_ago > 30 — replica has not heard from master recently.
  • Offset delta exceeds your tolerance (e.g. > 1 MB).

7.3 Keyspace Hit Rate

A sustained low hit rate means data is missing from the cache or being evicted too aggressively.

docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" INFO stats 2>/dev/null \
    | awk -F: "/keyspace_hits/{hits=$2} /keyspace_misses/{misses=$2} END {
        total = hits + misses;
        if (total > 0) printf \"hit_rate=%.1f%%\n\", hits/total*100
        else print \"hit_rate=no_data\"
      }"'

Alert if hit rate drops below 80% sustained over 10 minutes. Cold starts will always show low — establish a baseline first.

7.4 Evictions and Rejected Connections 

docker exec $(docker ps -qf "name=redis-master") \
  sh -c 'redis-cli -a "$(cat /run/secrets/REDIS_PASSWORD)" INFO stats 2>/dev/null \
    | grep -E "evicted_keys|rejected_connections|total_commands_processed"'
  • evicted_keys incrementing means memory is full and data is being silently discarded. Any non-zero rate warrants investigation.
  • rejected_connections > 0 means maxclients has been hit.

8. Patroni Internal Monitors

Collected automatically, every minute, by PatroniMemberStatus and PatroniClusterEvent – the latter additionally reads Patroni’s own DCS history key for an authoritative failover log no polling script can reconstruct after the fact. See Superseded in Large Part by Jube’s Own Observability Suite above.

Patroni’s REST API requires no database credentials and works regardless of whether PostgreSQL itself is up — making it the most reliable interface for cluster state.

All endpoints on port 8008 of each Patroni container.

8.1 Cluster Topology

The /cluster endpoint returns all members, their roles, lag, and timeline in one call.

docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -qO- --timeout=5 http://patroni1:8008/cluster 2>/dev/null'

Alert conditions:

  • No member has "role":"leader" — no primary.
  • Any member "lag" exceeds your RPO.
  • A known member is absent from the response — unreachable node.

Largest lag value for scripting:

docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -qO- --timeout=5 http://patroni1:8008/cluster 2>/dev/null \
    | grep -o "\"lag\":[0-9]*" | sort -t: -k2 -n | tail -1'

8.2 Per-Node Health and Leader Check 

# Health — returns 200 for any running member
docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -qO- --timeout=5 http://patroni1:8008/health 2>/dev/null'

# Leader — returns 200 only from the primary, 503 from standbys
docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -S --timeout=5 http://patroni1:8008/leader 2>&1 | grep "HTTP/" | awk "{print \$2}"'

Confirm exactly one node returns 200 from /leader:

LEADER_COUNT=0
for i in 1 2 3 4; do
  CODE=$(docker exec $(docker ps -qf "name=patroni${i}") \
    sh -c "wget -S --timeout=5 http://patroni${i}:8008/leader 2>&1 \
      | grep 'HTTP/' | awk '{print \$2}'" 2>/dev/null)
  [ "$CODE" = "200" ] && LEADER_COUNT=$((LEADER_COUNT + 1))
done
echo "leader_count=$LEADER_COUNT"
# Alert if != 1

8.3 DCS Connectivity 

# /config returns 200 only when Patroni can reach etcd
docker exec $(docker ps -qf "name=patroni1") \
  sh -c 'wget -S --timeout=5 http://patroni1:8008/config 2>&1 | grep "HTTP/" | awk "{print \$2}"'
# Alert on anything other than 200

8.4 Timeline Divergence

A timeline mismatch across nodes indicates an incomplete failover or split-brain risk.

for i in 1 2 3 4; do
  TL=$(docker exec $(docker ps -qf "name=patroni${i}") \
    sh -c "wget -qO- --timeout=5 http://patroni${i}:8008/health 2>/dev/null \
      | grep -o '\"timeline\":[0-9]*' | cut -d: -f2")
  echo "patroni${i} timeline=${TL:-UNREACHABLE}"
done
# Alert if values differ across nodes

9. Sentinel Internal Monitors

Collected automatically by RedisSentinelStatus (per-minute topology) and RedisSentinelEvent (every Sentinel pub/sub event captured the instant it fires, not just whatever state a one-minute poll happens to catch). See Superseded in Large Part by Jube’s Own Observability Suite above.

Sentinel is healthy when all expected nodes are up, they agree on the same master, and that master is reachable.

9.1 Master Reachability 

docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL masters 2>/dev/null'

Key fields (returned as a flat key-value list):

Field Alert condition
status Alert if not ok
flags Alert if contains disconnected, o_down, or s_down
num-slaves Alert if below expected replica count
num-other-sentinels Alert if below (expected sentinels − 1)
quorum Alert if num-other-sentinels + 1 < quorum value — failover cannot proceed

Extract master flags for scripting:

docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL masters 2>/dev/null \
    | awk "/^flags$/{getline; print}"'
# Expected: "master" — anything else (s_down, o_down, disconnected) is an alert

9.2 Quorum Check 

# Run from every sentinel — if any returns NOQUORUM, failover is not possible from that node
docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL ckquorum redis-master 2>/dev/null'
# Expected: OK N usable Sentinels. Quorum and failover authorization can be reached
# Alert on: NOQUORUM or NOAUTH

9.3 Master Address Consistency

All sentinels must agree on the same master address. Divergence means a split has occurred.

for i in 1 2 3 4 5; do
  MASTER=$(docker exec $(docker ps -qf "name=sentinel${i}") \
    sh -c 'redis-cli -p 26379 SENTINEL get-master-addr-by-name redis-master 2>/dev/null \
      | tr "\n" ":"' 2>/dev/null)
  echo "sentinel${i} master=${MASTER:-UNREACHABLE}"
done
# Alert if any two sentinels report different addresses

9.4 Recent Failover Detection

A recent failover warrants investigation even if the cluster is now healthy.

docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL masters 2>/dev/null \
    | awk "/^last-ok-ping-reply/{getline; print \"last_ok_ping_ms=\"\$0}"'

docker exec $(docker ps -qf "name=sentinel1") \
  sh -c 'redis-cli -p 26379 SENTINEL masters 2>/dev/null \
    | awk "/^last-failover-attempt$/{getline; print \"last_failover_epoch=\"\$0}"'
# Non-zero epoch means a failover was attempted — cross-check with cluster logs

10. Composite Shell Script

Wraps port and health endpoint checks into a single script. Wire the output into your collection mechanism. The internal monitors in sections 6–9 are best run as separate scheduled jobs given their heavier queries.

#!/usr/bin/env bash
# stack_health_check.sh
# Adjust container names, hostnames, ports, and service names for your stack.
# Output: one line per check — OK or FAIL prefixed with a label.

probe_tcp() {
  local label=$1 container=$2 host=$3 port=$4
  local result
  result=$(docker exec "$container" sh -c "nc -z -w 3 $host $port && echo OK || echo FAIL" 2>/dev/null)
  echo "$label: $(echo "$result" | grep -q '^OK' && echo OK || echo FAIL)"
}

probe_http() {
  local label=$1 container=$2 url=$3
  local result
  result=$(docker exec "$container" sh -c "wget -qO- --timeout=5 $url 2>/dev/null && echo OK || echo FAIL" 2>/dev/null)
  echo "$label: $(echo "$result" | grep -q 'OK' && echo OK || echo FAIL)"
}

probe_process() {
  local label=$1 container=$2 pattern=$3
  local result
  result=$(docker exec "$container" sh -c "pgrep -f '$pattern' > /dev/null && echo OK || echo FAIL" 2>/dev/null)
  echo "$label: $(echo "$result" | grep -q '^OK' && echo OK || echo FAIL)"
}

# --- etcd ---
for i in 1 2 3 4 5; do
  probe_tcp "etcd${i}_client" "$(docker ps -qf "name=etcd${i}")" "etcd${i}" 2379
  probe_tcp "etcd${i}_peer"   "$(docker ps -qf "name=etcd${i}")" "etcd${i}" 2380
done

# --- HAProxy ---
probe_tcp "haproxy_pg_primary" "$(docker ps -qf "name=haproxy")" haproxy 5432
probe_tcp "haproxy_pg_replica" "$(docker ps -qf "name=haproxy")" haproxy 5433
probe_tcp "haproxy_stats"      "$(docker ps -qf "name=haproxy")" haproxy 7000

# --- Patroni ---
for i in 1 2 3 4; do
  probe_http "patroni${i}_health" "$(docker ps -qf "name=patroni${i}")" "http://patroni${i}:8008/health"
done

# --- Redis ---
probe_tcp "redis_master"   "$(docker ps -qf "name=redis-master")"   redis-master   6379
probe_tcp "redis_replica1" "$(docker ps -qf "name=redis-replica1")" redis-replica1 6379
probe_tcp "redis_replica2" "$(docker ps -qf "name=redis-replica2")" redis-replica2 6379
probe_tcp "redis_replica3" "$(docker ps -qf "name=redis-replica3")" redis-replica3 6379

# --- Sentinels ---
for i in 1 2 3 4 5; do
  probe_tcp "sentinel${i}" "$(docker ps -qf "name=sentinel${i}")" "sentinel${i}" 26379
done

# --- Application HTTP health endpoints ---
for CONTAINER in $(docker ps -qf "name=<ui-service>"); do
  probe_http "ui_ready_${CONTAINER:0:12}" "$CONTAINER" "http://localhost:<port>/api/ready"
done

for CONTAINER in $(docker ps -qf "name=<api-service>"); do
  probe_http "api_ready_${CONTAINER:0:12}" "$CONTAINER" "http://localhost:<port>/api/ready"
done

# --- Background worker (no HTTP endpoint) ---
probe_process "jobs_process" "$(docker ps -qf "name=<jobs-service>")" "<process-pattern>"

# --- HAProxy frontend byte snapshot ---
echo ""
echo "HAProxy frontend traffic:"
docker exec "$(docker ps -qf "name=haproxy")" \
  sh -c 'wget -qO- http://localhost:7000/stats;csv 2>/dev/null \
    | awk -F, "$2==\"FRONTEND\" {
        printf \"  %-30s  bin=%-12s  bout=%-12s  status=%s\n\", $1, $9, $10, $18
      }"'

Cron schedule (30-second approximate interval):

* * * * * /etc/scripts/stack_health_check.sh >> /var/log/stack_health.log 2>&1
* * * * * sleep 30 && /etc/scripts/stack_health_check.sh >> /var/log/stack_health.log 2>&1

11. Jube Application Monitors

These five queries were already Jube’s own counters before the wider Infrastructure Health Metrics suite existed. The same signal, plus a finer per-stage/per-task breakdown, is now also exported as OpenTelemetry metrics (jube.engine.stage.duration and friends) whenever EnableOpenTelemetry is set – see Superseded in Large Part by Jube’s Own Observability Suite above. The raw queries below remain valid either way.

Five SQL queries run directly against the Jube application database. Poll on a short interval (30–60 seconds). Run from inside any Patroni container or any host with psql access to the primary via HAProxy port 5432.

The queries read from Jube’s own instrumentation tables — no schema changes or agents required.

11.1 Archive Queue Saturation

Measures the backlog of transaction records waiting to be written to the archive. A rising value means the asynchronous writer is falling behind, typically due to database I/O pressure or a crashed worker.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT \"Archive\"
    FROM \"EntityAnalysisModelAsynchronousQueueBalance\"
    ORDER BY 1 DESC
    LIMIT 1;
  "

Alert if value > 1000.

11.2 Case Creation Queue Saturation

Measures the backlog of cases waiting to be created. Saturation here means compliance workflows are being delayed — alerts and cases are not reaching analysts in near-real-time.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT \"CaseCreation\"
    FROM \"EntityAnalysisAsynchronousQueueBalance\"
    ORDER BY 1 DESC
    LIMIT 1;
  "

Alert if value > 100.

11.3 HTTP Processing Counter (Engine Liveness)

Measures the volume of HTTP requests being processed by the engine. A value at or near zero means the engine is dead or not receiving traffic — the application-layer equivalent of the HAProxy frontend zero-byte check, and will typically fire together with it.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT \"Model\"
    FROM \"HttpProcessingCounter\"
    ORDER BY 1 DESC
    LIMIT 1;
  "

Alert if value is zero or not advancing between polls.

11.4 Model Response Time

Average response time in milliseconds per model invocation. Degradation indicates infrastructure pressure (database latency, Redis latency, CPU saturation) or a model that has grown too complex for its invocation volume.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT \"ModelTotalResponseTime\" / NULLIF(\"ModelInvoke\", 0)
    FROM \"EntityAnalysisModelProcessingCounter\"
    ORDER BY 1 DESC
    LIMIT 1;
  "

Alert if value > 5000 (5 milliseconds).

11.5 Response Elevation Rate

Average response elevation as a proportion of total invocations. A high elevation rate indicates the model is triggering alerts on an abnormally large fraction of transactions — usually a sign of a misconfigured or degraded model rather than a genuine fraud spike.

docker exec $(docker ps -qf "name=patroni1") \
  psql -U postgres -Atc "
    SELECT \"ResponseElevation\" / NULLIF(\"ModelInvoke\", 0)
    FROM \"EntityAnalysisModelProcessingCounter\"
    ORDER BY 1 DESC
    LIMIT 1;
  "

Alert if value > 0.20 (20%).

12. Container Log Monitoring

All containers write to STDOUT. Log4net (used by Jube) additionally writes errors and warnings to STDERR. Monitoring splits accordingly: STDERR for Jube containers, STDOUT patterns for infrastructure.

Pass --since to bound the scan to your collection interval and avoid re-reading the full log history on every run.

12.1 STDERR — Jube Containers

Collected automatically by ApplicationLogEntry – every WARN/ERROR/FATAL logged anywhere in the process lands there centrally and is queryable, rather than needing a per-container docker logs grep. See Superseded in Large Part by Jube’s Own Observability Suite above.

Under normal operation STDERR should be silent. Any output warrants inspection.

# Count of STDERR lines per Jube container in the last 60 seconds
for CONTAINER in $(docker ps -qf "name=jube"); do
  COUNT=$(docker logs --since 60s "$CONTAINER" 2>&1 1>/dev/null | wc -l)
  echo "$CONTAINER stderr_lines=$COUNT"
done

For a higher-signal check, filter by severity keyword:

for CONTAINER in $(docker ps -qf "name=jube"); do
  ERRORS=$(docker logs --since 60s "$CONTAINER" 2>&1 1>/dev/null \
    | grep -ciE "ERROR|FATAL|WARN" || true)
  echo "$CONTAINER stderr_errors=$ERRORS"
done

Alert if stderr_errors > 0.

12.2 STDOUT — Infrastructure Containers

Infrastructure containers log cluster-state transitions to STDOUT. The signal here is state changes and crash markers, not a raw error count.

# etcd — leader elections and member failures
for CONTAINER in $(docker ps -qf "name=etcd"); do
  docker logs --since 60s "$CONTAINER" 2>/dev/null \
    | grep -iE "elected|lost|failed|compaction error" \
    | sed "s/^/$CONTAINER: /"
done

# Patroni — failovers, demotions, DCS errors
for CONTAINER in $(docker ps -qf "name=patroni"); do
  docker logs --since 60s "$CONTAINER" 2>/dev/null \
    | grep -iE "demoted|promoted|failover|etcd|cannot|timeout" \
    | sed "s/^/$CONTAINER: /"
done

# Redis — replica disconnections, AOF/RDB errors
for CONTAINER in $(docker ps -qf "name=redis"); do
  docker logs --since 60s "$CONTAINER" 2>/dev/null \
    | grep -iE "replica|replication|error|warning|lost connection" \
    | sed "s/^/$CONTAINER: /"
done

# HAProxy — backend server state changes
docker logs --since 60s $(docker ps -qf "name=haproxy") 2>/dev/null \
  | grep -iE "DOWN|UP|no server|backend|timeout"

12.3 Restart Detection

For Docker-hosted containers, DockerContainerMetric.RestartCount/OomKilled/ExitCode (via the Jube.Monitoring sidecar) already carries this per-minute – see Superseded in Large Part by Jube’s Own Observability Suite above. The commands below remain the only option for non-Docker hosts.

A container that has restarted recently has likely crashed. Check restart counts across all containers:

docker ps --format "table \t" | grep -i "restarting\|unhealthy"

For a numeric restart count per container:

docker inspect --format ' restarts= status=' \
  $(docker ps -q) 2>/dev/null | sed 's|^/||'

Alert if restarts has increased since the last poll, or if status is anything other than running.

13. Quick Reference

Component Container pattern Port(s) Probe Internal monitor
etcd (×N) etcd<N> 2379 client, 2380 peer TCP + etcdctl endpoint health —
PostgreSQL primary haproxy 5432 TCP pg_stat_activity, pg_stat_replication
PostgreSQL replica haproxy 5433 TCP replication lag query
HAProxy haproxy 7000 TCP + CSV byte counters frontend status, delta byte check
Patroni (×N) patroni<N> 8008 HTTP /health, /leader, /cluster /config for DCS, timeline check
Redis master redis-master 6379 TCP + PING INFO memory, INFO stats
Redis replicas (×N) redis-replica<N> 6379 TCP INFO replication
Redis Sentinel (×N) sentinel<N> 26379 TCP + SENTINEL ckquorum SENTINEL masters flags, address check
Application (HTTP) per service varies wget /api/ready —
Application (no HTTP) per service — pgrep —
Jube archive queue patroni<N> 5432 (via HAProxy) SQL EntityAnalysisModelAsynchronousQueueBalance > 1000
Jube case queue patroni<N> 5432 (via HAProxy) SQL EntityAnalysisAsynchronousQueueBalance > 100
Jube engine liveness patroni<N> 5432 (via HAProxy) SQL HttpProcessingCounter at zero
Jube response time patroni<N> 5432 (via HAProxy) SQL avg ms per invocation > 5000
Jube elevation rate patroni<N> 5432 (via HAProxy) SQL avg elevation ratio > 20%
Jube containers jube* — STDERR line count Log4net ERROR / FATAL / WARN
All containers — — Restart count delta docker inspect RestartCount

Jube™. © Jube Holdings Limited 2022 to present.

This site uses Just the Docs, a documentation theme for Jekyll.