🚀 Get to pre-production in weeks, not months, with private training direct from Jube’s developer — real sovereignty, zero vendor lock-in.

Deploying with Jube Cluster

Deploying with Docker covers a single-node deployment, which is the right starting point for evaluation, proof of concept, and smaller production workloads. This page covers a load-balanced, highly available, clustered deployment on Docker Swarm - multiple Postgres nodes under Patroni, multiple Redis nodes under Sentinel, and multiple Jube application instances, all fronted by HAProxy - suitable for production workloads that need to keep running through the loss of a node.

This is a real, working runbook, not a conceptual overview - the commands here are what an operator actually runs, in order, to stand up and operate the cluster. Placeholders like <node1-ip>, <hostname1> and <deploy-user> stand in for values specific to your environment.

Architecture Overview

Five-node Docker Swarm (four plus a tiebreaker) cluster, quorum-based throughout so the cluster keeps making progress after losing any single node.

Layer	Components
Consensus	etcd1, etcd2, etcd3, etcd4, etcd5
Database	patroni1, patroni2, patroni3, patroni4 (PostgreSQL, under Patroni)
WAL Archive	pgBackRest, writing to a NAS-backed named Docker volume
Cache	redis-master, redis-replica1/2/3, sentinel1/2/3/4/5
Load Balancer	haproxy
Application	jube-jobs (singleton), jube-api (scaled), jube-ui (scaled)

Postgres traffic flows through HAProxy only - application containers connect to haproxy:5432 (read/write) or haproxy:5433 (read-only, routed to a replica), never directly to a patroniN node. HAProxy makes that primary/replica routing decision by polling each Patroni node’s own REST API (GET /primary / GET /replica on port 8008) rather than guessing from a raw TCP check - this is Patroni’s own recommended integration pattern, since only Patroni itself reliably knows which node currently holds the lease.

Redis traffic bypasses HAProxy entirely. Jube’s RedisConnectionString points directly at the five Sentinels (sentinel1:26379,...,serviceName=redis-master,...), using StackExchange.Redis’s native Sentinel-aware connection mode: the client asks the Sentinel quorum who the current master is, connects there, and re-asks automatically on a failover. This is the standard client-side pattern for Redis Sentinel high availability, and is arguably a better fit than routing Redis through a TCP load balancer would be, since the client library’s own failover detection is faster and more precise than a proxy re-running a periodic health check. The Redis nodes themselves also defer to Sentinel on their own startup, for the same reason - see Data Layer Configuration below.

Both the etcd (5) and Sentinel (5) counts are deliberately odd - both are quorum/majority-vote systems, and an odd node count avoids the tied vote a same-sized even count can produce during a partition. Patroni’s replica count (4) has no such constraint, since Postgres replication itself isn’t a voting system - Patroni’s leader election is delegated entirely to etcd.

Data Layer Configuration

The Patroni, pgBackRest and Redis configuration in this cluster isn’t left at defaults - most of it is a deliberate tradeoff worth understanding rather than copying blind.

Postgres / Patroni

synchronous_mode: true with synchronous_mode_strict: false, alongside Postgres’s own synchronous_commit: off, is a specific and intentional combination, not a contradiction. synchronous_mode controls failover safety: Patroni always designates a synchronous standby, and will only promote a standby that Patroni knows had fully received what the old primary had committed - so automatic failover cannot silently lose a transaction. It does not, by itself, mean every commit waits for that standby. That’s controlled separately by synchronous_commit: off, which lets a client’s COMMIT return as soon as the local WAL write completes, without waiting for the standby’s acknowledgement over the network. Together: failover promotion is safe, but everyday commit latency doesn’t pay a synchronous-replication network round trip. The strict: false half of that first setting means the primary keeps accepting writes even if the synchronous standby is temporarily unreachable, rather than the whole cluster refusing writes because one designated standby is briefly down.

use_pg_rewind: true lets a demoted former primary rejoin the cluster as a standby by rewinding just the diverged WAL, rather than requiring a full new basebackup - faster recovery after a failover. use_slots: true with max_slot_wal_keep_size: 4GB uses replication slots so a standby that briefly disconnects (a restart, a network blip) doesn’t need a full resync - the primary retains the WAL it needs - while the cap stops a permanently dead standby’s slot from filling the disk indefinitely. maximum_lag_on_failover: 33554432 (32MB) excludes a standby that’s fallen too far behind from being an automatic-failover candidate at all, favouring “wait” or “pick a better candidate” over promoting something meaningfully behind.

archive_mode/archive_command continuously ship WAL to pgBackRest for point-in-time recovery, bounded by archive_timeout: 60s - the maximum window of committed-but-not-yet-archived data if archiving somehow stalled. checkpoint_timeout: 15min and checkpoint_completion_target: 0.9 spread checkpoint I/O smoothly across most of that interval rather than bursting it; wal_compression: lz4 trades a small amount of CPU for meaningfully less WAL volume to write and ship. commit_delay: 1000 (microseconds) with commit_siblings: 10 briefly batches concurrent commits so their WAL flushes can be amortised together under load - a classic high-throughput OLTP tuning, appropriate for a continuous stream of transaction-monitoring writes rather than occasional writes. Autovacuum is tuned meaningfully more aggressive than Postgres’s defaults (autovacuum_vacuum_scale_factor: 0.01, autovacuum_analyze_scale_factor: 0.005, versus defaults an order of magnitude larger) for the same reason - a high-insert-rate workload accumulates dead tuples and stale statistics faster than the defaults assume, and letting either lag invites both bloat and poor query plans.

pg_hba scopes replication, superuser-over-network, and application traffic all to the 10.0.1.0/24 overlay subnet specifically, rather than a broader range - defence in depth underneath “all traffic must flow through HAProxy,” so even a misconfiguration elsewhere can’t accept a Postgres connection from outside the cluster’s own network. The one exception, local all all trust, only ever applies to the Unix socket inside a given container - unreachable without already having exec access to that specific container - which is exactly what makes the Total Postgres Password Wipeout Recovery procedure above both possible and safe: the trust boundary it relies on is equivalent to already having host/container access, not a network-open door.

pgBackRest and why the backup repository is on NAS

repo1-path (the pgBackRest backup repository - WAL archive plus periodic full backups) is mounted from NAS (/mnt/pgbackrest), not host-local disk. The reason is durability independence: if backups lived on the same host’s disk as the Patroni node producing them, losing that host would take out the live database and its own backup history in the same event. NAS decouples the two, and also means any node - not just the one that produced a given backup - can read the repository for a restore.

Within that, archive-async=y plus spool-path (mounted as tmpfs in the compose file, so it’s RAM-backed and non-persistent) decouples Postgres’s archive_command from actual NAS write latency: Postgres hands a completed WAL segment to a fast local spool and moves on immediately, while a background pgBackRest process drains that spool to the NAS repository asynchronously. A slow or momentarily congested NAS write can never stall the primary’s own WAL archiving as a result - the spool being non-durable is fine precisely because the durable copy of record is whatever has already landed in the NAS repository, not whatever’s transiently sitting in the spool. repo1-retention-full=2 (keep the last two full backups, with WAL retained to cover both) is a deliberate rollback safety margin - if the newest full backup turns out to be bad, there’s still a second, older one to fall back to rather than only ever having exactly one.

Redis

appendonly yes with appendfsync everysec accepts the same kind of throughput/durability tradeoff as Postgres’s synchronous_commit off above: an fsync once per second, rather than per write, bounds the worst-case data loss on a hard crash to roughly one second of writes, in exchange for not paying an fsync’s latency on every single command. no-appendfsync-on-rewrite yes avoids a further fsync stall while the AOF file is being compacted in the background. io-threads 4 with io-threads-do-reads yes parallelises Redis’s normally single-threaded network I/O across threads, worthwhile on the multi-core hosts this cluster runs on under concurrent load. protected-mode no is safe specifically because requirepass/masterauth are always set (via the wrapper command, from the REDIS_PASSWORD secret) and the network is already confined to the private overlay - protected mode exists to stop an accidentally-unauthenticated instance being reachable from anywhere, which doesn’t apply here since both of the conditions it guards against are independently already covered.

Every node defers to Sentinel on startup, rather than any one node hardcoding itself as master. Each of redis-master, redis-replica1, redis-replica2 and redis-replica3 runs the same shape of startup logic: query each Sentinel in turn for who it currently considers the master (SENTINEL get-master-addr-by-name redis-master), and if one answers with a master that isn’t itself, start as --replicaof that node - only falling back to the historical static topology (redis-master seeds as a plain master; the others target redis-master directly) if no Sentinel can be reached or none has an opinion yet, which is only ever true on a genuinely fresh cluster.

This matters because Sentinel’s failover promotion happens at the Redis protocol level, live, without restarting any container - but a container restart (a crash, a redeploy, docker service update --force) always re-runs whatever is written in docker-compose.yml. A static, unconditional redis-master command with no --replicaof at all would mean: after Sentinel has genuinely promoted a different node to master following a real failure, the original redis-master service could later restart and declare itself master all over again with its own old data - a second, independent master directly contradicting whatever Sentinel and the rest of the cluster now agree is authoritative. Querying Sentinel first closes that specific gap - every node’s actual role after any restart reflects what the cluster currently believes, not what the compose file assumed when it was first written.

redis-master’s volume is NAS-backed (/mnt/redis) while the three replicas use local, Swarm-managed volumes (redis-replicaN-data) - and this is purely a backup decision, not a role/failover one now that startup role resolution is Sentinel-driven for all four. Moving that one node’s AOF file off the host and onto NAS gives a further, independent layer of backup for the data beyond in-cluster replica redundancy itself - the same durability-independence reasoning as pgBackRest’s repository above, applied to Redis. The three replicas deliberately don’t need this: their entire purpose is live redundancy, and any one of them can always fully re-sync from whichever node Sentinel currently considers master, so paying NAS latency for volumes that exist to be disposable and rebuildable would be pure overhead with no corresponding benefit.

Why background processing runs on one instance

jube-jobs is the only service with EnableMigration, EnableReprocessing, EnableSanctionLoader, CachePruneServer, EnableTtlCounter, EnableSearchKeyCache, EnableCasesAutomation and EnableCallback all set to True; jube-api and jube-ui have every one of those False. This isn’t an oversight - several of these are documented as safe to run on only one instance at a time ( see Environment Variables), and letting every scaled jube-api/jube-ui replica also run background processing would mean N copies of the same job racing each other. jube-jobs is deployed as a singleton for exactly this reason; jube-api/jube-ui are the stateless, horizontally-scaled request handlers.

Building and Distributing Images

Jube.Cluster/build-images.sh automates the five builds below, tagging each image with the current git HEAD’s short SHA automatically (rather than you filling in <sha> by hand) and writing the tags straight into Jube.Cluster/.env (if it already exists). Run it from anywhere - it locates the repo root itself:

Jube.Cluster/build-images.sh

Pass --save to also docker save each of the five images straight after building, collected under Jube.Cluster/Images/ (gitignored - the tarballs are too large to push to git) as <image>:<sha>.tar, matching the tags just written to .env:

Jube.Cluster/build-images.sh --save

jube-otel-listener is one of the five despite being only a throwaway local OTLP receiver for manual testing, because docker-compose.yml deploys it unconditionally (replicas: 1, not gated behind any EnableOpenTelemetry flag) - without its image, that service has nothing to run and sits failing in docker service ls.

If you don’t need OpenTelemetry testing, the simpler fix is commenting out the jube-otel-listener service in docker-compose.yml entirely rather than building and distributing an image you’ll never point anything at - and then dropping it from the manual commands below.

jube.etcd wraps the stock quay.io/coreos/etcd binaries with an entrypoint that bootstraps etcd’s own username/password auth once, idempotently, on every node’s own startup - see Software Inventory. Its build context is Jube.Cluster/etcd, not the repo root, since it doesn’t need to COPY any Jube project files.

jube.patroni must be built from a Dockerfile pinning Patroni 4.0.11/4.1.1 or newer. Older Patroni cannot talk to an auth-enabled etcd at all, for the reason set out under Database & backup.

To build manually instead, from the root of the repository (Jube.Cluster/patroni/Dockerfile has no -f-implied default, so it needs one just like the others):

docker build --no-cache -f Jube.Cluster/etcd/Dockerfile -t jube.etcd:<sha> Jube.Cluster/etcd
docker build --no-cache -f Jube.Cluster/patroni/Dockerfile -t jube.patroni:<sha> .
docker build --no-cache -f Jube.App/Dockerfile -t jube.app:<sha> .
docker build --no-cache -f Jube.Monitoring/Dockerfile -t jube.monitoring:<sha> .
docker build --no-cache -f Jube.OpenTelemetryListener/Dockerfile -t jube.opentelemetrylistener:<sha> .

Save each image to a tar file, collected under Jube.Cluster/Images/ (the same directory build-images.sh --save uses, and gitignored for the same reason - the tarballs are too large to push to git):

docker save -o Jube.Cluster/Images/jube.etcd:<sha>.tar jube.etcd:<sha>
docker save -o Jube.Cluster/Images/jube.patroni:<sha>.tar jube.patroni:<sha>
docker save -o Jube.Cluster/Images/jube.app:<sha>.tar jube.app:<sha>
docker save -o Jube.Cluster/Images/jube.monitoring:<sha>.tar jube.monitoring:<sha>
docker save -o Jube.Cluster/Images/jube.opentelemetrylistener:<sha>.tar jube.opentelemetrylistener:<sha>
docker save -o Jube.Cluster/Images/redis.tar redis:7-alpine
docker save -o Jube.Cluster/Images/haproxy.tar haproxy:2.8

build-images.sh --save covers the five images it builds; redis and haproxy are still pulled, not built, so they always need saving manually like this regardless of which path you took above.

Zip the whole Jube.Cluster/Images/ directory (after resetting permissions - see File System below, and removing any .git/IDE directories) with a date, name or version in the archive name, and distribute it to every node.

This tar-file distribution, rather than pushing to a private registry, is the standard approach where the cluster’s network cannot reach an external registry (an air-gapped or tightly firewalled deployment, which is common for on-premises AML/fraud infrastructure at a regulated institution). The operational cost is real and worth understanding before choosing it: every image update is a manual build-save-copy-load cycle across every node, rather than a single docker service update --image registry/jube.app:tag, so version drift between nodes is a risk this process has to actively guard against (see the docker images verification step below). Where the network allows it, a private registry (Harbor, a cloud provider’s container registry, or similar) removes this manual step entirely and should be preferred.

First Time Setup

NAS Permissions

The mount needs to be accessible to containers under SELinux enforcing, which means the share needs container_file_t context. For example:

Share	Mount path	Owner (uid:gid)	Mode
pgBackRest repo	/var/lib/pgbackrest/repo	70:70	0750
pgBackRest spool	/var/spool/pgbackrest	70:70	0750
Redis data	/data	999:999	0750

# pgBackRest
<share_path>  (rw,all_squash,anonuid=70,anongid=70)

# Redis
<share_path>  (rw,all_squash,anonuid=999,anongid=999)

The Patroni entrypoint handles initialisation of its own directories automatically:

container starts as root
  └─ /var/lib/pgbackrest/repo/.initialized absent?
       └─ mkdir -p, chown postgres:postgres, chmod 750, touch .initialized
  └─ /var/spool/pgbackrest/.initialized absent?
       └─ mkdir -p, chown postgres:postgres, chmod 750, touch .initialized
  └─ su-exec postgres patroni ...

Redis handles its own /data directory internally; no init script is required for it.

The bind-mounted directory still needs to be owned/writable by the right user before the first container start - pre-create it on each host:

sudo mkdir -p /mnt/pgbackrest
sudo chown 70:70 /mnt/pgbackrest
sudo chcon -Rt container_file_t /mnt/pgbackrest

# Redis
sudo mkdir -p /mnt/redis
sudo chown 999:999 /mnt/redis
sudo chcon -Rt container_file_t /mnt/redis

SSL Offloading

SSL certificates should be issued by your organisation’s certificate authority and placed in:

etc/haproxy/ssl

referenced from the HAProxy config by a fixed naming convention:

• Frontend jube_api expects jube-api.pem in that directory.
• Frontend jube_ui expects jube-ui.pem in that directory.

Treat the HAProxy configuration files as production configuration under version control alongside the rest of this deployment, rather than something edited ad hoc on a host.

Creating Secrets

Every credential the cluster needs - Postgres passwords, the Redis password, JWTKey, PasswordHashingKey, ApiHmacKey, ElementSymmetricEncryptionKey, the RSA keypair for PasswordAsymmetricEncryption, and etcd’s own ETCD_ROOT_PASSWORD/PATRONI_ETCD3_PASSWORD - is generated locally and loaded into Docker Swarm’s own secret store (docker secret create), never written into docker-compose.yml or an Environment Variable directly. For the Jube application services, Jube’s own [@Key@] tokenisation ( see Environment Variables) resolves each Environment Variable’s [@SECRET_NAME@] placeholder against the mounted secret file at container start, so the secret value itself is never visible in docker service inspect, container environment listings, or this compose file. This list is just what this particular cluster deployment happens to need, not a fixed or hardcoded set - Jube’s tokenisation is entirely free-form (see Environment Variables), so wiring in an additional secret for a new Environment Variable is just adding it to the secrets: block below, mounting it on the relevant service, and referencing [@ITS_NAME@] from that variable’s value - no code change involved.

haproxy.cfg itself needs no secret at all - jube_ui and jube_api are both plain load-balanced backends with no session affinity, since the Watcher live feed (Server-Sent Events, fanned out to every node via Redis pub/sub) no longer needs a client pinned to one node the way its old SignalR hub connection did, so there is no dynamic-cookie-key or equivalent left to template a secret into.

ETCD_ROOT_PASSWORD and PATRONI_ETCD3_PASSWORD are a second pattern again, since neither etcd nor Patroni is a Jube .NET process and so neither can use [@Key@] tokenisation either. Both the jube.etcd and jube.patroni images bake in their own small bash load_secret function in their Dockerfile-generated entrypoint scripts, which reads /run/secrets/<NAME> directly and exports it as a plain environment variable before handing off to the real process - Patroni auto-maps PATRONI_ETCD3_PASSWORD onto its own etcd3.password config the same way it already does for PATRONI_ETCD3_HOSTS, and etcd’s entrypoint reads ETCD_ROOT_PASSWORD/PATRONI_ETCD3_PASSWORD directly to bootstrap its own auth (see Software Inventory).

That export only ever reaches the process the entrypoint hands off to, which is not where an operator works. docker exec gives a shell the container’s configured environment - PATRONI_ETCD3_USERNAME=patroni from the compose file below - and nothing the entrypoint exported at runtime, so a patronictl command typed into such a shell ran with a username and no password: python-etcd warns Username provided without password, both are required for authentication and Patroni then fails every one of the five endpoints, which an auth-enabled etcd reports as etcdserver: user name is empty. database.sh failed the same way, its patronictl list returning nothing to read a leader out of. The jube.patroni image therefore ships /usr/local/bin/patronictl, a wrapper ahead of the real /opt/patroni/bin/patronictl on PATH that sources the image’s single load_secret definition (/usr/local/lib/patroni-secrets.sh, shared with the entrypoint) and then hands off. Nothing you type changes - patronictl in a docker exec shell simply authenticates now, and so does every patronictl command in this runbook. Patroni patches its PATRONI_* environment variables over whatever the configuration file says, so the credential applies whether or not -c /etc/patroni.yml is passed; the image also sets PATRONICTL_CONFIG_FILE=/etc/patroni.yml, because patronictl reads that variable rather than the PATRONI_CONFIG_FILE the daemon is given, and without it a bare patronictl list looks for a ~/.config/patroni/patronictl.yaml that no container has. Jube.Tests/Cluster/ClusterPatroniImageTests.cs asserts that contract against the Dockerfile: the wrapper ahead of the real binary on PATH, both entry paths loading their secrets from one definition, and the patronictl default configuration file matching the one the container is actually given.

PATRONI_ETCD3_PASSWORD is the one secret that spans both patterns, because Patroni is not the only etcd client here. Every Jube service running the engine (EnableEngine=True - jube-api and jube-jobs) polls etcd for member and health status and reads Patroni’s DCS history key, for the Infrastructure Health Metrics pages. Those services therefore mount the same secret and reach it the ordinary [@Key@] way, through EtcdClientUsername=patroni and EtcdClientPassword=[@PATRONI_ETCD3_PASSWORD@]. Reusing Patroni’s own scoped credential rather than issuing a second one keeps the etcd role list to what Patroni already needs; the samplers only read. Leave either variable off a service that runs the engine and its etcd samplers fall back to unauthenticated calls, which an auth-enabled etcd rejects - the pages then stay empty with nothing logged, which is why Jube.Tests/Cluster/ClusterComposeSecretTests.cs asserts the pairing directly against this compose file.

Any secret created outside this process - for example, a client-provided key received over email or another channel that isn’t end-to-end secure - should still be rotated into docker secret create rather than left as a plain Environment Variable, for the same reason: once it’s a Swarm secret, it’s no longer readable from the running service’s inspected configuration.

On All Nodes

From the cluster directory (Jube.Cluster/, where the distributed archive’s Images/ folder lands), load images on every host:

docker load -i Images/jube.etcd:<sha>.tar
docker load -i Images/jube.patroni:<sha>.tar
docker load -i Images/jube.app:<sha>.tar
docker load -i Images/jube.monitoring:<sha>.tar
docker load -i Images/jube.opentelemetrylistener:<sha>.tar
docker load -i Images/redis.tar
docker load -i Images/haproxy.tar

Verify the images are present, and that no stale images/volumes remain from a previous version:

docker images
docker volume ls
docker volume rm <image>  # if a stale one exists

Open the firewall ports Swarm needs on every host, including its overlay-network encryption protocols:

sudo firewall-cmd --permanent --add-port=2377/tcp
sudo firewall-cmd --permanent --add-port=7946/tcp
sudo firewall-cmd --permanent --add-port=7946/udp
sudo firewall-cmd --permanent --add-port=4789/udp
sudo firewall-cmd --permanent --add-port=500/udp
sudo firewall-cmd --permanent --add-port=4500/udp
sudo firewall-cmd --reload
sudo systemctl restart docker

Run the permissions script on every host:

sudo chmod +x permissions-init.sh
./permissions-init.sh

On Node 1 Only

Initialise the swarm:

docker swarm init --advertise-addr <node1-ip>

Get the manager join token:

docker swarm join-token manager

On Nodes 2 and 3

Join as managers using the token from node 1:

docker swarm join --token <manager-join-token> <node1-ip>:2377

Back on Node 1

Verify all nodes have joined:

docker node ls

Add zone labels - these are what the placement constraints below pin each Patroni/etcd instance to a distinct host:

docker node update --label-add zone=host1 <hostname1>
docker node update --label-add zone=host2 <hostname2>
docker node update --label-add zone=host3 <hostname3>

Validate the labels landed correctly:

docker node ls -q | xargs docker node inspect -f ' []: = '

If running on a single host for evaluation, comment out the placement constraints in the compose file’s Jube services first - otherwise Swarm has nowhere to schedule the tiebreaker role and scaling will simply fail to find an eligible host.

      # placement:
      # constraints:
      # - node.labels.zone != $CRITICAL_HOST_TIEBREAKER

Create the .env file the compose file reads image tags and zone assignments from:

cat <<EOF > .env
JUBE_IMAGE=jube.app:<sha>
PATRONI_IMAGE=jube.patroni:<sha>
JUBE_MONITORING_IMAGE=jube.monitoring:<sha>
JUBE_OTEL_LISTENER_IMAGE=jube.opentelemetrylistener:<sha>
CRITICAL_HOST_1=host1
CRITICAL_HOST_2=host1
CRITICAL_HOST_3=host1
CRITICAL_HOST_4=host1
CRITICAL_HOST_TIEBREAKER=host1
EOF

Generate and load all secrets:

sudo chmod +x secrets-init.sh
./secrets-init.sh

secrets-init.sh is destructive by design - it prompts for confirmation, then recreates every secret from scratch (random passwords, a random ENCRYPTION_KEY for ElementSymmetricEncryptionKey, plus a fresh 4096-bit RSA keypair for PasswordAsymmetricEncryptionPrivateKey/PasswordAsymmetricEncryptionPublicKey) and loads them into Docker Swarm. Nothing it generates is written to disk in plaintext - every credential goes straight into a Docker secret, and database.sh (below) reads back the handful it needs from a running container’s /run/secrets/ mount rather than from a bootstrap record on disk. Because a Docker secret can never be read back out of the Swarm store once created, secrets-init.sh prints every generated value to the console in one labelled block at the end of the run - this is the only opportunity to capture them into a password manager or vault, so do that now, then clear the terminal’s scrollback so the values don’t linger in shell history either.

The public half of the RSA keypair is printed in that same block (it is not itself sensitive - only the private key needs to stay secret, and that travels through Swarm secrets/tokenisation automatically) and needs to be copied into the PasswordAsymmetricEncryptionPublicKey Environment Variable for jube-api/jube-ui in docker-compose.yml (the CHANGE-ME-paste-public-key-printed-by-secrets-init.sh placeholder) before the first deploy - this is the one manual step. Without it, PasswordAsymmetricEncryption=True (set by default in the compose file for jube-api/jube-ui) will fail to start, since Jube refuses to start with RSA password encryption enabled and either key missing. ENCRYPTION_KEY and PASSWORD_ASYMMETRIC_ENCRYPTION_PRIVATE_KEY require no such manual step - both flow into jube-api/jube-jobs/jube-ui automatically via the [@Key@] Docker Secrets tokenisation pattern (see Environment Variables), the same as the passwords and JWT_KEY/API_HMAC_KEY above.

Every Deploy (Fresh or After Full Reset)

Remove any existing stack first:

sudo chmod +x remove.sh
./remove.sh

Deploy the stack:

sudo chmod +x deploy.sh
./deploy.sh

Check stability:

docker service ls

Watch Patroni especially - wait until all four nodes show, a leader is elected, and lag is 0:

docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml list

Check logs if anything looks wrong:

docker service logs --follow --since 1m jube-cluster_patroni1

If a node doesn’t come up on its own - common on a brand new cluster - force it:

docker service update --force jube-cluster_patroni1

Then create the pgBackRest stanza (once, on a fresh cluster) and verify it:

docker exec -u postgres $(docker ps -q -f name=patroni1) \
    pgbackrest --config=/etc/pgbackrest/pgbackrest.conf \
               --lock-path=/var/lib/postgresql/tmp/pgbackrest-lock \
               --stanza=postgres-cluster \
               stanza-create

docker exec -u postgres $(docker ps -q -f name=patroni1) \
    pgbackrest --config=/etc/pgbackrest/pgbackrest.conf \
               --stanza=postgres-cluster \
               check

If a Patroni node shows shutdown purely due to startup timing, force it individually - never force two Patroni nodes at once, or you risk losing quorum:

docker service update --force jube-cluster_patroni2

Verify HAProxy shows exactly one green server in postgres_primary at http://<host-ip>:7000.

Getting patroni1 into the leader role

The deployment expects patroni1 specifically to hold the leader role. Check who currently holds it, and who the current Sync Standby is - you can only promote the current Sync Standby, so getting patroni1 into the lead sometimes takes two switchovers rather than one:

docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml list

If patroni1 is not the current Sync Standby, switch leadership to whichever node is (this rotates patroni1 into the Sync Standby slot):

docker exec -it $(docker ps -q -f name=patroni1) \
  patronictl -c /etc/patroni.yml switchover postgres-cluster \
  --leader <current-leader> --candidate <current-sync-standby> --force

Once patroni1 shows as the Sync Standby, perform the final switchover to make it leader:

docker exec -it $(docker ps -q -f name=patroni1) \
  patronictl -c /etc/patroni.yml switchover postgres-cluster \
  --leader <current-leader> --candidate patroni1 --force

Creating the database users

Run the least-privilege database user setup before scaling Jube up - jube-api/jube-jobs/jube-ui connect as jube_app/jube_reporting/jube_migration from the moment they start, and those users don’t exist until this script creates them (mirroring the same service/reporting least-privilege split as the single-node CreateUsers.sql referenced from Deploying with Docker - jube_migration gets DDL rights, jube_app gets DML only, jube_reporting gets read-only):

sudo chmod +x database.sh
./database.sh

Scale up jube-jobs first, since it runs the database migration on startup:

docker service scale jube-cluster_jube-jobs=1

Watch its logs until migration completes - a welcome message with no .NET exceptions:

docker service logs -f jube-cluster_jube-jobs

Only then scale up the application tier:

docker service scale jube-cluster_jube-api=4 jube-cluster_jube-ui=4

Watch both come up cleanly the same way:

docker service logs -f jube-cluster_jube-api
docker service logs -f jube-cluster_jube-ui

Navigate to http://<host-ip>:5001.

jube-api and jube-ui are both defined with replicas: 0 in the compose file, and scaled up manually only after migration succeeds - deliberately, so a fresh deployment can never race the application tier against an unmigrated schema.

Run a full backup to validate the NAS mount is actually working end to end, not just mounted:

# Find the primary
docker ps | grep patroni1

# Full backup, skipping WAL verification (busy cluster, WAL is archived separately anyway)
docker exec -it <container-id> pgbackrest --stanza=postgres-cluster --archive-check=n --archive-timeout=60 --type=full backup

# Watch progress
docker exec -it <container-id> ps aux | grep pgbackrest

Browse the NAS separately to confirm Redis AOF/RDB files are landing there too.

Nothing further to clean up here - secrets-init.sh never wrote credentials to disk, so there is no bootstrap record to shred.

Forcing a Compose Change to Apply

Swarm diffs the running service spec against the new compose file and only updates what changed - occasionally it decides nothing has changed when something has, and silently skips the update.

Check what the running service actually has:

docker service inspect jube-cluster_<service> --pretty

Force Swarm to re-evaluate the whole compose spec:

docker stack deploy --with-registry-auth -c docker-compose.yml jube-cluster

Force a specific service, optionally busting the image cache:

docker service update --force jube-cluster_<service>
docker service update --force --image <image>:latest jube-cluster_<service>

Changing a Postgres Parameter on an Already-Bootstrapped Cluster

Editing bootstrap.dcs.postgresql.parameters in patroni/patroni{1,2,3,4}.yml and redeploying the image does nothing on a cluster that already exists. Patroni’s container entrypoint just execs patroni /etc/patroni.yml - there is no reconciliation step anywhere in this repo that pushes file changes into the DCS. Patroni only ever reads bootstrap: once, the very first time it initialises a brand-new cluster in etcd; every later container start reads the config already stored in etcd instead and ignores the file’s bootstrap section entirely. Editing these files is still correct and worth keeping in sync - it is what a future from-scratch bootstrap (disaster recovery, a new environment) will use - it just isn’t how you change a parameter on the cluster you’re already running.

For that, push the change into the DCS directly with patronictl edit-config, from any running Patroni container:

# -p is shorthand for postgresql.parameters.<key>=<value>; repeat it per parameter.
# --force skips the interactive confirmation prompt.
docker exec -it $(docker ps -q -f name=patroni1) \
    patronictl -c /etc/patroni.yml edit-config \
    -p shared_preload_libraries=pg_stat_statements \
    -p pg_stat_statements.track=all \
    -p pg_stat_statements.max=10000 \
    --force postgres-cluster

Confirm it landed and see which members now need a restart to actually pick it up:

docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml list postgres-cluster

shared_preload_libraries is postmaster-context - Patroni marks every member “pending restart” but keeps serving traffic on the old value until each is actually restarted. Restart replicas first, one at a time, and the current leader last and deliberately - restarting the leader forces a failover:

docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml restart postgres-cluster <replica-member>
# ...repeat for every replica, then finally:
docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml restart postgres-cluster <leader-member>

Verify the live value once every member is back:

docker exec -it $(docker ps -q -f name=patroni1) su-exec postgres psql -U postgres -c "SHOW shared_preload_libraries;"

Ordering trap: CREATE EXTENSION IF NOT EXISTS pg_stat_statements; alone would not catch a missed restart - it succeeds either way, since it only registers catalog objects; the underlying shared-memory hook it actually needs is only wired up at postmaster start via shared_preload_libraries, so without it you’d get a silently dead extension and a confusing failure later, the first time something queries the view. The migration (AddImplicitAsyncRecallAndTimeoutFeatures) guards against exactly this: its very first statement checks current_setting('shared_preload_libraries') and raises an explicit error if pg_stat_statements isn’t in it, before anything else in that migration runs. So if jube-jobs reaches this migration before the rolling restart above has happened, the whole migration fails loudly and cleanly instead - do the edit-config + rolling restart first, and only then (re)deploy or let jube-jobs pick up the pending migration.

Full Reset of Corrupted Swarm State

docker stack rm jube-cluster
sleep 15
docker swarm leave --force
docker network prune -f
sudo rm -rf /var/lib/docker/swarm
sudo systemctl restart docker
docker swarm init --advertise-addr <node1-ip>
docker stack deploy -c docker-compose.yml jube-cluster

Port Map

Host-Exposed Ports

Port	Service	Reachable via	Exposed?	Reason
5432	PostgreSQL R/W	HAProxy → patroni1/2/3/4	⚠️ Dev only	Remove in production
5433	PostgreSQL R/O	HAProxy → patroni1/2/3/4	⚠️ Dev only	Remove in production
7000	HAProxy Stats	haproxy	✅ Always	Ops monitoring dashboard
5001	Jube UI	jube-ui	✅ Always	Web interface
5002	Jube API	jube-api	✅ Always	Public API (maps to 5001 internally)

In production, remove the two dev-only Postgres entries from the compose file’s ports: block entirely. Application services on the Swarm network reach Postgres via haproxy:5432/haproxy:5433 regardless.

Unlike Postgres, Redis has no host-published port and no HAProxy entry at all in this topology - clients (including Jube itself) reach it exclusively via the Sentinel ports below, from inside the overlay network, matching the Sentinel-aware connection pattern described in Architecture Overview.

Port 7000 (HAProxy stats) is an ops/diagnostic dashboard with no authentication layer of its own. This deployment assumes it is reachable only from a private management network or VPN, never from the open internet. If that doesn’t hold for your network, put a reverse proxy with authentication in front of HAProxy stats, or firewall it to known management IPs, before exposing the cluster more broadly.

Internal-Only Ports

Port	Service	Reason
2379	etcd client API	Cluster consensus - no external access needed; client auth is mandatory regardless (see above)
2380	etcd peer traffic	etcd-to-etcd peering only
5432	Patroni direct	Application traffic must flow through HAProxy
8008	Patroni REST API	HAProxy health checks only
6379	Redis direct	Not used - see Sentinel ports below
26379	Redis Sentinel	Sentinel-to-Sentinel coordination, and the entry point Jube’s Sentinel-aware Redis client connects to

Health Checks

PostgreSQL (Patroni) - HAProxy polls port 8008 every 3s:

• GET /primary → 200 OK → eligible for the postgres_primary backend.
• GET /replica → 200 OK → eligible for the postgres_replicas backend.
• Anything else → marked down after 3 consecutive failures; existing sessions on that server are terminated immediately on mark-down, rather than left to drain, since a demoted primary must stop taking writes at once.
• 2 consecutive successes → marked back up.

Patroni container health (Docker Swarm’s own healthcheck) - polls /liveness on port 8008 every 10s, 10 retries. /liveness returns 200 as long as the Patroni process itself is alive, regardless of replication state - it answers “is this container worth restarting”, not “is this node the primary”.

Redis - Sentinel monitors redis-master directly (not via HAProxy, per the architecture above): sentinel monitor redis-master redis-master 6379 3 (quorum of 3), down-after-milliseconds 5000, failover-timeout 60000.

Diagnostics — General

# All services and replica counts
docker service ls

# Why a service won't start
docker service ps --no-trunc jube-cluster_<service>

# Live logs for a service, or by task ID
docker service logs -f jube-cluster_<service>
docker service logs <task-id>

# Patroni cluster state
docker exec -it $(docker ps -q -f name=patroni1) patronictl -c /etc/patroni.yml list

# HAProxy stats
http://localhost:7000

# Shell into a container as root
docker exec -it -u root <container-id> /bin/sh

# Network tools on Alpine-based containers
apk add --no-cache busybox-extras

Diagnostics — SELinux

# Recent denials, optionally filtered to a service
sudo ausearch -m AVC -ts recent
sudo ausearch -m AVC -ts recent | grep sentinel

# Check a file's SELinux label
ls -laZ <cluster-directory>/redis/sentinel1/sentinel.conf

# Temporarily disable, to confirm SELinux is the cause - then re-enable
sudo setenforce 0
sudo setenforce 1

Always re-enable SELinux after testing. If disabling it fixes the issue, run the permissions script to apply the correct labels rather than leaving SELinux disabled - the permissions script exists specifically so this is never a live tradeoff.

Diagnostics — Volumes

docker volume ls
docker volume ls -f dangling=true
docker volume inspect jube-cluster_etcd1_data
docker volume rm jube-cluster_etcd1_data
docker volume rm $(docker volume ls -f name=jube-cluster -q)
docker volume prune

Diagnostics — Network

# Overlay networks
docker network ls --filter driver=overlay
docker network inspect jube-cluster_jube-cluster
docker network prune -f

# DNS resolution from inside a container
docker exec -it $(docker ps -q -f name=patroni1) nslookup etcd1
docker exec -it $(docker ps -q -f name=patroni1) cat /etc/resolv.conf

# Docker daemon errors, including network allocation failures
journalctl -u docker -n 50 --no-pager

# Firewall and host networking
sudo firewall-cmd --list-ports
ip addr show

# A container's internal overlay IP
docker inspect -f '' <container-id>

Containers are reachable via the published host IP and port. Internal 10.x.x.x overlay addresses are not routable from the host - always use the published host port, not the overlay IP, when connecting from outside a container.

Diagnostics — Swarm Cluster Membership

docker node ls
docker node inspect <hostname> --pretty
docker swarm join-token manager
docker swarm join-token worker
docker node update --availability drain <hostname>   # before maintenance
docker node update --availability active <hostname>  # after maintenance
docker node promote <hostname>

Diagnostics — Failover Testing

docker ps | grep patroni

# Simulate a hard failure
docker kill <container-id>

# Simulate a slow/unresponsive node, and recover from it
docker pause <container-id>
docker unpause <container-id>

# Manually trigger a Patroni failover
docker exec -it $(docker ps -q -f name=patroni1) \
    patronictl -c /etc/patroni.yml failover postgres-cluster

# Reinitialise a single bad node without cluster downtime
docker exec -it $(docker ps -q -f name=patroni1) \
    patronictl -c /etc/patroni.yml reinit postgres-cluster <member>

File System

Reset the deployment directory back to sane, predictable permissions after a chmod/chown mishap:

sudo chown -R <deploy-user>:<deploy-user> <cluster-directory>
sudo find <cluster-directory> -type d -exec chmod 755 {} \;
sudo find <cluster-directory> -type f -exec chmod 644 {} \;
sudo find <cluster-directory> -name "*.sh" -exec chmod 755 {} \;

Total Postgres Password Wipeout Recovery

If every Postgres credential is genuinely lost (not merely rotated), the local Unix socket accepts trust authentication from the container’s own postgres user - this is a last-resort local-access-only recovery path, not a routine operation:

docker exec -it $(docker ps -q -f name=patroni1) su-exec postgres psql -U postgres

Notes

• Never pre-create Docker networks manually - always let the stack manage them, since Swarm’s own network lifecycle handling assumes it owns creation and teardown.
• Patroni and Redis should be pinned to specific hosts via placement constraints - unpinned, Swarm is free to reschedule a stateful service onto whichever host has capacity, which defeats the zone-per-node design this cluster relies on for fault isolation.
• Mount Postgres data to remote/redundant disks in production, not host-local storage.
• Remember the pgBackRest stanza step on a fresh cluster - nothing else prompts for it, and backups silently have nowhere to go without it.
• patronictl inside a Patroni container is the image’s wrapper, not Patroni’s own binary - it loads /run/secrets before handing off, which is what makes it work at all under etcd client authentication. It lives in the image, so changing it means rebuilding (build-images.sh), reloading on every host, and rolling the containers one at a time, standbys before the primary.
• The Patroni entrypoint fixes pgBackRest volume ownership automatically on every container start - seeing the volume start with the wrong ownership (Fedora maps UID 999 to avahi) is expected and self-corrects.
• Sentinel conf files are rewritten by Redis Sentinel at runtime - reset them manually if they become corrupted.
• jube-api is mapped 5002:5001 externally - the application always listens on 5001 internally; 5002 is only the host-published port.
• EnableMigration=True is set only on jube-jobs - explicitly False on jube-api and jube-ui, so migration can never run from more than one place.
• On Fedora with SELinux, MCS category mismatches (c###,c### labels) are resolved with chcon -Rl s0, which strips the categories so any container can access the files.
• The /run/docker.sock SELinux label resets on every Docker restart - the permissions script re-applies it, so this is expected rather than a regression each time.
• jube-api’s ReportConnectionString points at the read replica pool (haproxy:5433), matching jube-ui and jube-jobs - all three route reporting queries away from the primary.