High availability, Scalability and Performance
- Home
- Peregrine Management Suite
- High Availability, Scalability and Performance
- High availability, Scalability and Performance
Monitoring, Logging and Auditing Peregrine MS is backed by RabbitMQ and ElasticSearch as shown in the Architecture section. Both technologies are very scalable and offer high performance with great reliability. Neuron Event Processor service acts as a gateway between RabbitMQ and ElasticSearch. It is also very scalable, and it can be configured for high availability, simply by installing and running the Event Processor on multiple machines.
We have observed that a single server hosting the Event Processor, RabbitMQ, ElasticSearch and Neuron ESB can sustain rates as high as 2 million audit or wiretap entries indexed per hour.
The Event Processor service has several settings that you can adjust depending on your needs.
| Setting | Description | Default Value |
| NeuronEventConsumerCount | Number of workers for each of the metrics and logs | 5 |
| NeuronEventConsumerCountTrace | Number of workers for wiretap messages. Separate because it can raised separately when wiretapping. See Neuron ESB 3.7.5 documentation for Wiretap functionality | 10 |
| NeuronEventCacheSize | Number of messages for each metric and log that can live in memory before the event processor starts throttling and let the messages remain with RabbitMQ until it frees up. | 50000 |
| NeuronEventServiceRateCacheSize | Separate entry for Service Rate cache. Since service rate processing requires high memory, it can slow down other parts, its value is set lower to throttle more aggressively. | 100 |
| NeuronEventTracePointCacheSize | This setting comes into play only when wiretapping. Can be decreased or increased depending on how fast the wiretap messages need to be searchable. Increasing this setting can increase memory consumption. | 5000 |
| NeuronEventPrefetchCount | Prefetching messages from RabbitMQ reduces overhead by consuming messages in batches. This setting only applies to audit messages | 50 |
| NeuronEventServiceRatePrefetchCount | Prefetching messages from RabbitMQ reduces overhead by consuming messages in batches. This setting only applies to service rate messages. This value is set lower because service rate has high memory consumption. | 2 |
| NeuronEventTracepointPrefetchCount | This setting comes into play only when wiretapping. Can be decreased or increased depending on how large the messages are and rate of logging. Increasing this setting can increase memory consumption but it can drain the queue faster and therefore not create back pressure on the publisher. | 100 |
| NeuronEventCacheFlushInterval | This is the interval when the records are written to Elastic Search. It can be increased if there can be some more latency permissible between generation and viewing the messages in MS. | 2 seconds |
| NeuronEventESBatchSize | This the number of records written to Elastic Search as the same time. Can be increased to higher value for high loads. | 1000 |
You may need dedicated servers or clusters for RabbitMQ and ElasticSearch depending on your scalability needs.
The ElasticSearch Service has two files that control the settings for ElasticSearch performance. ElasticSearch.yml file and java.options file. The Peregrine MS installers sets the desired values that are suitable for most scenarios but you may need to tweak them if ElasticSearch service is shared by other applications or your load is very high. For more details ElasticSearch documentation links in the reference section.
You do not need to adjust any RabbitMQ settings for performance. They are automatically set by Peregrine Management Suite when creating the underlying queues.