Environment Variables

The traditional method of applying application configuration settings is via a file that is embedded in the applications root directory, with key and values representing the application configuration.

In Jube, application settings, known as Environment Variables, can be applied in one of three ways, loaded in the following priority:

  • Hardcoded values loaded into the dictionary available for the whole engine, via dependency injection.
  • Jube.Environment configuration file, with the file available alongside the binary executable.
  • Operating System Environment Variables.

It follows that hardcoded application settings are overridden by Jube.Environment file key value pairs, which may be persistently overwritten by Operating System Environment Variables key value pairs. Passing Operating System Environment Variables updates the corresponding key value pair in the Jube.environment file, henceforth, setting Operating System Environment Variables can be more convenient than making the same update in the Jube.environment file (i.e. setting an Operating System Environment Variable is slightly easier easier than locating a file, then VI’ing it). It also follows tha the Jube.environment file is the definitive record of the Environment Variables in use by a Jube instance.

Throughout the documentation, Environment Variables should be taken to mean the aforementioned source of key value pairs.

The following hardcoded Environment Variables, this would be to say defaults, are available:

Variable Default Value Description
ModelSynchronisationWait 10000 The duration in milliseconds between model synchronisation. Many of the entities in model synchronisation require explicit instruction to synchronise, with that instruction being checked for each interval. Model entities which do not require explicit synchronisation will be included for each interval.
EnableNotification True A boolean value to indicate if Notifications are enabled in the instance, in which case background threads will be established to relay the notifications created by the model invocation, and that model invocation may be branched to dispatch notifications to the thread.
EnableTTLCounter True A boolean value indicating if this instance should start a thread to manage the decrement of TTL Counters.
ConnectionString null The connection string to the Postgres database.
CacheConnectionString null The connection string to the optional Cache Postgres database. In the absence of value will fall back to ConnectionString.
EnableSearchKeyCache True A boolean value indicating if this instance should start a thread to manage the computation of Abstraction Rules set to Cache, meaning the background computation of aggregation values. Only a single instance may exist.
EnableCasesAutomation True A boolean value indicating if this instance should start a thread to manage changing of status in the case records. Only a single instance may exist.
CasesAutomationWait 60000 Subject to EnableCasesAutomation being True, the duration in milliseconds between model case status change and automation.
EnableEntityModel True A boolean value to indicate if the models should be synchronised to this instance thus being available for invocation.
ArchiverPersistThreads 1 DO NOT USE UNTIL CASE MANAGEMENT CASE CREATION CONCURRENCY RESOLVED or Case Management Case Creation is not required. The number of threads preparing records for bulk insert to the Archive and ArchiveKey tables. In extremely high throughput implementations with thousands of records being inserted, balancing the number of threads inserting records with the amount of records batched to insert allows for write performance optimisation. Zero (0) has the effect of disabling.
ModelInvokeAsynchronousThreads 1 The number of threads allocated to processing model invocation switched to be asynchronous. Zero (0) has the effect of disabling.
BulkCopyThreshold 100 For each Archiver Persist Thread the number of records to be buffered before a bulk copy is committed. A maximum of ten seconds is allowed from the last message else bulk insert is performed regardless.
ActivationWatcherAllowPersist True A boolean value to indicate that the instance may persist activation records to the ActivationWatcher table and also allowing for the instantiation of threads to manage the asynchronous bulk insert.
ActivationWatcherPersistThreads 1 The number of threads preparing records for bulk insert into the ActivationWatcher table. Zero (0) has the effect of disabling.
ActivationWatcherBulkCopyThreshold 100 For each Activation Watcher Persist Thread the number of records to be buffered before a bulk copy is committed.
EnableReprocessing True A boolean value to indicate that the reprocessing of model invocation given Archive data, be allowed to create threads.
ReprocessingThreads 1 The number of threads allocated to reprocessing model invocation given Archive data. Zero (0) has the effect of disabling.
ReprocessingBulkLimit 10000 Reprocessing returns data from the Postgres Archive tables in batch, in the order the transactions took place by the date of creation, then invokes the model for each record. The value is the limit for the records to be returned for each batch so not to flood the instance memory.
ThreadPoolManualControl True A boolean value to override the .Net Thread Pool with MinThreadPoolThreads and MaxThreadPoolThreads values, else .Net self managed. Managing the thread pool manually can reduce apparent warm up time for the application and reduce risk of thread starvation through threads being closely sized to underlying compute resources. All requests to the Kestrel web server will place demand on the Thread Pool and optionally the Forking of Abstraction Rules for each grouping key. Model invocation can be decoupled from the thread pool partially via the use of asynchronous model invocation, otherwise it forms part of the request handled by the Kestrel thread.
MinThreadPoolThreads 1 Given ThreadPoolManualControl being True, the minimum threads in the Thread Pool.
MaxThreadPoolThreads 1000 Given ThreadPoolManualControl being True, the maximum threads in the Thread Pool.
ForkAbstractionRuleSearchKeys False Abstraction Rules often share common Search Keys and in such cases the Cache database is only selected from the once, with the rule logic being executed against an in memory dataset. In the event that ForkAbstractionRuleSearchKeys is True, the processing will be executed in parallel for each Grouping \ Search Key. The forking of Abstraction Rules will place additional load on the Thread Pool and database compute.
MaximumModelInvokeAsyncQueue 10000 The maximum number of pending asynchronous HTTP Model invocations allowed in queue before further requests are returned with an error.
SMTPHost null For notification functionality, the SMTP server host.
SMTPPort 587 The SMTP server port.
SMTPUser null Assuming authentication the SMTP server user name.
SMTPPassword null The SMTP server password given user name.
SMTPFrom null The email address to be sent from.
ClickatellAPIKey null For notification functionality, the API key provided by Clickatell for the SMS dispatch service.
HttpAdaptationUrl https://localhost:5001 As a security measure a prefix to the HTTP Endpoint value entered by the end user in HTTP Adaptation to recall scores.
HttpAdaptationTimeout 1000 The timeout value for HTTP Adaptation endpoint recall.
HttpAdaptationValidateSsl False A boolean value to ensure SSL validation for HTTP Adaptation endpoint recall.
EnableMigration True A boolean value to indicate if migrations should be attempted for the ConnectionString and CacheConnectionString for the instance.
EnableSanction True A boolean value to indicate if sanctions should be loaded from the database, cached to the instance memory and made available for searching in model invocation or directly.
EnableSanctionLoader False A boolean value to indicate that this instance should load the Sanctions tables from local or internet resources. Disabled by default for reasons of security and privacy given outbound request to external HTTP resources.
SanctionLoaderWait 60000 The duration in milliseconds between polling the file system or internet location for sanctions files.
NegotiateAuthentication False A boolean value to indicate that authentication should take place via Negotiated Authentication (commonly known as Active Directory in Windows networks) rather than JSON Web Token authentication. Very common in corporate environments.
EnableCacheIndex True A boolean value to indicate that this instance should ensure that the Cache Tables are correctly indexed given model configuration.
UseMockDataExhaustive True A boolean value to indicate that given an Exhaustive training instance having been instructed by the end user, whether the Mock data should be used for training. Mock data is only useful for Demonstration purposes and should routinely be set to False for production use.
AMQP False A boolean value to indicate that connections should be established via the AMQPUri connection string to the RabbitMQ server or cluster.
AMQPUri null The connection string to the RabbitMQ server or cluster.
JWTValidAudience http://localhost:5001 The server domain that is allowed for the token.
JWTValidIssuer http://localhost:5001 The server domain that issued the token.
JWTKey null The encryption key for the JWT. This key is created randomly on first startup of the Jube instance, but can be changed. The key value is stored in the Jube.environment file, having being randomly created on application first use.
PasswordHashingKey null Passwords are encrypted in an irreversible manner using the Argon2 hashing algorithm, alongside a salting value. The salt value is stored in the Jube.environment file, having being randomly created on application first use.
EnablePublicInvokeController True Most HTTP Endpoint require authentication, although an exception is the api/Invoke endpoint which is intended for integration. A boolean value needs to be set to allow the invocation API to be recalled on an open basis, and will in the event of False, return Not Found HTTP Status.
EnableEngine True A boolean value to indicate if the engine should be instantiated. The engine contains functionality such as Sanctions, Model Invocation, Archiving, Search Key Caching, Notifications, Case Automation, Counters, Tagging, Reprocessing and Exhaustive Training. In the absence of the engine, only functions required by the user interface are instantiated.
EnableExhaustiveTraining True A boolean value to indicate that a thread should be started for the purpose of Exhaustive training. Given the computation expense of Exhaustive training only a single thread is available per instance.
ExhaustiveTrialsLimit 1000 For an Exhaustive training instance, the maximum number of trials to be performed (which is a random selection of input variables. This variable does not govern the evolution of a Neural Networks topology for each trial.
ExhaustiveMinVariableCount 5 For an Exhaustive trial, the minimum number of variables to be trained and topology evolve.
ExhaustiveMaxVariableCount 30 For an Exhaustive trial, the maximum number of variables to be trained and topology evolve.
ExhaustiveTrainingDataSamplePercentage 0.6 To avoid over-fitting, data available is randomly split between training, cross validation and testing. The percentage of data to be allocated to training.
ExhaustiveCrossValidationDataSamplePercentage 0.2 The percentage of data to be allocated to Cross Validation.
ExhaustiveTestingDataSamplePercentage 0.2 The percentage of data to be allocated to Testing.
ExhaustiveValidationTestingActivationThreshold 0.5 The threshold at which a recalled example will be allocated to the positive class.
ExhaustiveTopologySinceImprovementLimit 10 During a trial the internal topology will have processing elements and hidden layers evolved. The evolution process will terminate at the point a number of additional evolutions have taken place without improvement. The value at which evolution should terminate upon no improvement. Lesser performance will always terminate and carry forward as best topology.
ExhaustiveLayerDepthLimit 4 The maximum number of hidden layers in a topology exploration.
ExhaustiveLayerWidthLimitInputLayerFactor 4 A value when multiplied by the number of inputs for the trial represents the maximum number of processing elements for a hidden layer.
ExhaustiveTopologyComplexityLimit 10000 For topology evolution the maximum number of weights allowed before termination and carrying forward of that topology as best.
ExhaustiveActivationFunctionExplorationEpochs 3 The number of epochs to be trained to evaluate then select an Activation Function.
ExhaustiveTopologyExplorationEpochs 3 The number of epochs to be trained to evaluate then evolve further a topology.
ExhaustiveTopologyFinalisationEpochs 20 With topology selected, the number of epochs to create a final model for testing.
ExhaustiveSimulationsCount 100 The number of random simulations to be processed through the final model for the purpose of testing and creation of description statistics for the positive class. Often known as Monte Carlo Model Simulation.
CallbackTimeout 1000 The time in milliseconds before callbacks ready for collection via HTTP are to be removed. Callbacks are only created if HTTP model invocation has been called using the async switch.
StreamingActivationWatcher True The default behaviour for the Activation Watcher streaming, in the absence of AMQP having been enabled, is via database streaming notifications. When false indicates that the database ActivationWatcher table, given also ActivationWatcherAllowPersist, should be polled for Watcher messages rather than subscribing to database streaming notifications. Given streaming notifications the message is dispatched to the notification service in the database synchronously.
WaitPollFromActivationWatcherTable 5000 In the event that of StreamingActivationWatcher and ActivationWatcherAllowPersist the polling interval to relay messaged from the ActivationWatcher database table to the watcher user interface page.
WaitTtlCounterDecrement 60000 In the event of TTLCounter being enabled to signify that the server should be responsible to decrementing TTL counter entries, the wait interval between decrement jobs in milliseconds
ShowWelcomeMessage True Display the welcome message in the console to help the user get started

The file containing the Environment Variables to override the default is in the binary executable directory, a file by the name Jube.environment, which is a new line = seperated key value file as follows:

Image

JSON is not used to allow for a more forgiving value component without the need for escaping.

The file is read line by line, ignoring lines which have a hash # in front. For each key value, the key available from the hardcoded section will be overwritten with the value from the file, unless it has the same value.

Operating System Environment variables are supported and are commonly used to remotely configure virtual machine or container images.

For each Operating System Environment Variable passed, the hardcoded key and value will be referenced, being overwritten if the values are not the same.

Environment Variables are written out as INFO in logging at each precedence step.


Jube™. © Jube Holdings Limited 2022 to present.