🚀 Get to pre-production in weeks, not months, with private training direct from Jube’s developer — real sovereignty, zero vendor lock-in.

Redis Schema

As introduced in the architecture section of this documentation, the real-time performance of the platform is in large part owing to the use of a Remote Key Value \ Data Structure Pair data store, in the form of Redis (although these days it implies any RESP protocol). A Key Value Pair data store at first pass might appear analogous to a Relational Database Management System (RDBMS). However, given the inability to query - as it can only access data via known keys—significant thought needs to be given to schema.

In the Redis Schema set out in Jube, there are three data structures in use:

Data Structure	Use Case
Sets	A structure that given a key maintains a list of values rolling up to a key.
Sorted Sets	A structure that given a key maintains a subset of keys preserving a prescribed order.
Hash Tables	A structure that given a key maintains a large subset of key values rolling up (a key value within a key value). Using Hash Tables over the persistence of this same data in a more traditional key value pair, which would work fine, is thought to be less memory intensive (https://instagram-engineering.com/storing-hundreds-of-millions-of-simple-key-value-pairs-in-redis-1091ae80f74c), while maintaining a slightly more elegant schema.

A visual example of the schema is as follows:

The key structure follows a commonplace Redis standard where components of the key are separated with a semicolon ( i.e., :). Key segments uniformly include a tenant identifier immediately following the prefix (e.g., ReferenceDate:1, where 1 is the tenant identifier). Guids are stripped of their hyphens, although this is the only place in the system where that is done.

The prefixes are described in the following table:

Prefix	Type	Key Structure	Description
PayloadJournal	Set	PayloadJournal:TenantRegistryId:EntityAnalysisModelGuid:EntityAnalysisModelInstanceEntryGuid	For a given EntityAnalysisModelInstanceEntryInstanceGuid a journal or payloads references. This set is important for deletion of payload items to ensure that the references to the payload data removed from ledgers referencing it.
Journal	Sorted Set	Journal:TenantRegistryId:EntityAnalysisModelGuid:EntityAnalysisRequestXpathSearchKeyName:EntityAnalysisRequestXpathSearchKeyNameValue	Jube aggregates for EntityAnalysisRequestXpathSearchKeyName, for example all transactions for a given account identifier or IP. This sorted set maintains a list of payload references for a given EntityAnalysisRequestXpathSearchKeyName and EntityAnalysisRequestXpathSearchKeyNameValue combination, preserving the order by reference date.
ReferenceDate	Sorted Set	ReferenceDate:TenantRegistryId:EntityAnalysisModelGuid	For each EntityAnalysisModelGuid, a record of its EntityAnalysisModelInstanceEntryGuid alongside its reference date in sorted order. This sorted set is important to identify the current reference date prevailing for an EntityAnalysisModelGuid, with that value being offset and used for delete processing.
ReferenceDateLatest	Sorted Set	ReferenceDateLatest:TenantRegistryId:EntityAnalysisModelGuid:EntityAnalysisRequestXpathSearchKeyName	For each EntityAnalysisRequestXpathSearchKeyNameValue (e.g., AccountId of 123456) an ordered list of the latest payload reference date available. This sorted set is used as an access path to the ledgers in conjunction with the ReferenceDateLatest Hash Table below as an alternative to a scan. The payload latest is important for EntityAnalysisRequestXpathSearchKeyNameValue evaluation as it helps navigate to the distinct values stored for the EntityAnalysisRequestXpathSearchKeyNameValue. The serialized object also stores important metadata such as the update time to avoid reevaluation of a EntityAnalysisRequestXpathSearchKeyNameValue given no beta.
ReferenceDateLatest	Hash Table	ReferenceDateLatest:TenantRegistryId:EntityAnalysisModelGuid	Renamed from the legacy LatestCount prefix (migration RenameLatestCountInRedisToReferenceDateLatest). For each EntityAnalysisRequestXpathSearchKeyName, maps to the timestamp it was most recently touched - not a count, despite the legacy name. Used as an access path - to identify which EntityAnalysisRequestXpathSearchKeyName values exist for a model - when pruning the per-search-key ReferenceDateLatest Sorted Sets above, as an alternative to a scan.
Payload	Hash Table	Payload:TenantRegistryId:EntityAnalysisModelGuid	One of the most material data structures containing the largest payload byte volume, representing the transaction data. For an EntityAnalysisModelInstanceEntryGuid a byte array value is stored, which is an optimized MessagePack binary serialisation representing the full transaction payload processed. Referenced in several other structures.
PayloadBytes	Hash Table	PayloadBytes:TenantRegistryId	Information only and relating to the total stored bytes for a model for the purpose of identifying usage in a multi tenancy environment without necessitating a scan. Can be disabled via the RedisStorePayloadCountsAndBytes environment variable.
PayloadCount	Hash Table	PayloadCount:TenantRegistryId	Information only and relating to the total stored payload entries for a model for the purpose of identifying usage in a multi tenancy environment without necessitating a scan. Can be disabled via the RedisStorePayloadCountsAndBytes environment variable.
PayloadLatest	Hash Table	PayloadLatest:TenantRegistryId:EntityAnalysisModelGuid:EntityAnalysisRequestXpathSearchKeyName	For a EntityAnalysisRequestXpathSearchKeyName a contractless serialized MessagePack object containing various information about the latest payload, via a reference to the payload data, including as its most recent updated date which is used in abstraction rule caching for calculating time to live and reclassification status (reserved for future use). In practice, only the updated date and this could arguably be a much more lightweight schema, however, it is envisaged that more advanced processing of latest transaction may happen in the future.
ReferenceDate	Hash Table	ReferenceDate:TenantRegistryId	For an EntityAnalysisModelGuid the current reference date being used for processing and offsetting, used as an access path for Payload deletions.
Sanction	Hash Table	Sanction:TenantRegistryId:EntityAnalysisModelGuid	For a given EntityAnalysisModelGuid, a cache of sanctions lookups to avoid burdensome Levenshtein Distance algorithm calculations, albeit subject to an expiration in online processing. Uses the search string (i.e. Robert Mugabe) in composite with the distance prescribed (i.e. 2). Returning a MessagePack contractless serialisation including the computed value from the Levenshtein Distance algorithm recall and the updated date for use expiry in online processing.
TtlCounter	Hash Table	TtlCounter:TenantRegistryId:EntityAnalysisModelGuid:TtlCounterGuid:TtlCounterGroupingFieldName	For a Time to Live Counter TtlCounterGroupingFieldName, the current counter value for a given TtlCounterGroupingFieldNameValue.
TtlCounterEntry	Hash Table	TtlCounterEntry:TenantRegistryId:EntityAnalysisModelGuid:TtlCounterGuid:TtlCounterGroupingFieldName:TtlCounterGroupingFieldNameValue	To facilitate the deprecation of Time to Live Counters, grouping by a given date resolution, the number of incrementation for the date resolution, for a given TtlCounterGroupingFieldName and TtlCounterGroupingFieldNameValue combination. Used for the deprecation of counters in the TTLCounter prefixed data structure on expiry.
LruJournal	Sorted Set	LruJournal:TenantRegistryId:EntityAnalysisModelGuid	On each recall of a PayloadJournal key it is stored alongside the system data in a sorted set for use in the warming of local Lru cache having the effect of warming most recently used payload data on the basis that for each PayloadJournal all Payload data is returned for the keys returned by the PayloadJournal. Entries older than LruJournalMaxAgeInterval/lruJournalMaxAgeValue are pruned by a dedicated background task on the WaitLruJournalPrune interval - see Local LRU Cache.
ActivationCaseIdempotency	Set	ActivationCaseIdempotency:TenantRegistryId:EntityAnalysisModelGuid:ActivationRuleGuid	Subject to the ActivationRuleIdempotency environment variable, the set of EntityAnalysisModelInstanceEntryGuid values for which this Activation Rule has already created a Case, so Case Creation is applied at most once per transaction even under reprocessing or retry.
ActivationNotificationIdempotency	Set	ActivationNotificationIdempotency:TenantRegistryId:EntityAnalysisModelGuid:ActivationRuleGuid	The same idempotency mechanism as ActivationCaseIdempotency, guarding Notification dispatch instead of Case Creation.
TtlCounterIdempotency	Set	TtlCounterIdempotency:TenantRegistryId:EntityAnalysisModelGuid:TtlCounterGuid	The same idempotency mechanism again, guarding a TTL Counter increment triggered from an Activation Rule so it is only ever applied once per transaction.
IdempotencyJournal	Set	IdempotencyJournal:TenantRegistryId:EntityAnalysisModelGuid	The set of idempotency Set keys (above) that exist for a model, used so that the idempotency records for a transaction can be found and removed when its payload is deleted, rather than needing a scan. Idempotency Set entries have no Redis TTL of their own - they are only ever removed via this cleanup path when the underlying payload is deleted, so a deployment that never expires payloads will accumulate idempotency keys for as long as the payload itself is retained.

In the current release of Jube, Latest values are maintained on a largely informational basis but are reserved for future use in payload reclassification.

The Payload prefix can be subject to compression based on the RedisMessagePackCompression environment variable, in which case LZ4 Block Array Compression will be used. While LZ4 compression is extremely fast and introduces only negligible overhead, for a small payload, there is scant compression benefit.

Optional Postgres offload for Hash Table writes

The Hash Table structures above are, by default, all backed by Redis. Setting the RedisHsetOffloadToPostgres environment variable to True redirects Hash Table reads and writes - across every Hash Table prefix in the table above, this is a global switch rather than a per-structure choice - to a Postgres table, CacheSetHash, instead of Redis.

CacheSetHash is hash-partitioned by Key into four partitions, with a composite primary key on (Key, Field) and a covering index over (Key, Field) including ByteValue, IntValue and FloatValue, so point lookups are satisfied entirely from the index without a heap fetch. Byte-array values (serialized payloads and similar) are stored in ByteValue; numeric increments/decrements - the common case for counters - are stored in IntValue or FloatValue and applied as a single atomic INSERT ... ON CONFLICT DO UPDATE SET value = COALESCE(value, 0) + @increment statement, avoiding the byte-array-to-number-and-back cast chain that a Redis-backed numeric HINCRBY equivalent would otherwise need when the value is stored as serialised bytes.

This is a genuine architectural choice between two backends for the same data, and the right default depends on the deployment’s relative Redis versus Postgres headroom - benchmark on representative production hardware before enabling for a production workload, since the branch’s own benchmarking here was on a development laptop rather than server-grade hardware. The most common scenario where offloading HSET to Postgres is where memory resources are highly constrained, especially in cloud environments, given an asymetry between compute provisioning (Jube does not need much) and memory availability (Jube needs a lot in the absence of offloading) in stock images. The tradeoff comes down to a reluctance to pay a premium for RAM, where compute is comparatively cheap. On decent quality SSD’s, while response time vary a bit more, the performance degradation is not mateirally different, and response times tend to be perfectly acceptable (i.e. there is not a deal of difference between a 10ms response and a 30ms response time in practical terms).