Itential Validated Designs

An Active/Standby Architecture (ASA) is an Itential architecture where all of the components are redundant and can gracefully tolerate at least 1 catastrophic failure and also provides a redundancy for the primary data center. This architecture is the recommended architecture for production environments that must adhere to strict business continuity and uptime demands. It builds on top of the HA2 architecture essentially using two HA2 installs in geographically redundant locations. It makes use of a larger MongoDB replica set that is also geographically redundant.

The IAP application performs many reads and writes against the database and is sensitive to great latencies. All active components must be running in the same datacenter. The MongoDB replication process will ensure that the data written to the primary node in the active data center will be replicated to the geographically redundant MongoDB nodes in the secondary data center. All components must have authentication enabled.

The minimum ASA architecture is comprised of 17 VMs broken down as follows:

• 4 IAP servers
• 5 MongoDB servers
• 6 Redis servers
• 2 IAG server

Highly Available Itential Enterprise (IAP)

IAPs communicate with one another through Redis and share data via MongoDB. Simply adding a new IAP node and pointing it to the correct MongoDB and Redis is sufficient to to achieve high availability. As IAPs are added and configured they are enabled to perform work.

IAPs must have the following configurations:

• MongoDB connection strings must contain a reference to all members of the replica set.
• Redis configurations must specify the list of all known Redis Sentinels and their Sentinel username and password. Connections to an HA Redis actually occur through the Sentinels and not directly to Redis.

Highly Available MongoDB

MongoDB clusters operate in a primary/secondary model where data written to the primary will replicate to the secondary. If a primary MongoDB node fails the replica set will detect this failure and force an election for a new primary. During this time the replica set may not accept reads and writes until the new primary is selected, usually after a few seconds. Once it is finished and a new primary is identified the IAP application will resume normal operation. Operators of Itential do not have to do anything during this election.

In order to preserve an odd number of replicas to prevent a split-brain scenario when/if an election occurs, this architecture requires the MongoDB cluster to be split across three data centers or regions, 2 in the primary region, 2 in the secondary region, and a third in a tertiary region. When a region is lost there will remain three voting members of the replica set. The replica set configuration must enforce a preference to influence the voting in this architecture to guarantee that the primary MongoDB will shift to the secondary region in the case of a disaster.

Itential’s MongoDB cluster must have the following requirements in this architecture:

• All replica set members must be defined in the IAP config.
• Authentication between the replica members must be done with either a shared key or X.509 certificate.
• The database must have an admin user able to perform any operation.
• The database must have an “itential” user that is granted the least amount of privileges required by the IAP application. IAP must be configured to use this user account.
• The replica set configuration must leverage the priority settings to influence the voting as follows:

More MongoDB replication information here

Replica Set Elections

Highly Available Redis

Redis clusters operate in a primary/secondary model where data written to the primary will replicate to the secondary. If a primary Redis node fails the replica set will detect this failure via the Redis Sentinels and force an election for a new primary. During this time the replica set may not accept reads and writes until the new primary is selected, usually after a few seconds. Once it is finished and a new primary is identified the IAP application will resume normal operation. Operators of Itential do not have to do anything during this election. Itential’s Redis cluster must have the following requirements:

• All Redis nodes must be defined in the IAP profile configuration.
• Authentication between the replica members is done with users defined in the Redis config file.
• Redis must have an admin user able to perform any operation.
• Redis must have an “itential” user that is granted the least amount of privileges required by the application. IAP must be configured to use this user account.
• Redis must have a replication user that is granted the least amount of privileges required by the replication process.
• Redis Sentinel must be included to monitor the Redis cluster and musts be colocated with Redis.
• Redis Sentinel must have an admin user able to perform any Sentinel task.
• Redis nodes must maintain a low latency connection between nodes to avoid replication failures.

More Redis replication information here

Itential Enterprise Server

MongoDB Server

Redis Server

IAG Server

Hardware & Hypervisor Configurations

Processor

The processor specification requirement is as follow:

• Second generation or better Intel Xeon Platinum 8000 series processors.
• Third generation or better AMD EPYC 7000 series processors.

Memory

The memory specification requirement is as follow:

• DDR5 DRAM 3200 Mhz or higher

Storage

The storage performance requirement in IOPS (16 kiB) is as follows:

• 20000+ IOPS
• Non-spinning media (i.e. SSD, NVMe)

Network

The network speed requirement is:

• 10 Gbps or higher

Note: In some instances, adding additional dedicated interfaces that are focused on routing specific traffic to specific external systems can be explored further. This routing of traffic would be configured at the OS-level (custom interfaces and routes) and require the system administrator to manage this. An example would be separating NSO traffic from Redis/Mongo destined traffic.

Hypervisor/Host OS Settings

These settings are strongly recommended for high load applications of IAP:

• CPU affinity settings or similar functionality to prevent CPU starvation
• Full Memory Reservation
• One physical CPU per VM is preferred
• Huge pages for memory support enabled
  • Except MongoDB
    Note: MongoDB discourages the utilization of Transparent Huge Pages
• Memory compression disabled
• Minimal CPU Allocation settings for scheduler according to CPU clock
  • Example: assuming an IAP VM on a server capable of 2.5GHz nominal speed: CPU clock reservation = 16vCPU * x 2.5GHz

Note: Please follow Hypervisor recommendations when performing CPU reservations. In most cases the total of all CPU reservations for all VMs on a host cannot be more than 90% of the host capacity as 10% is reserved by the host itself.

Ports & Networking

In an environment where components are installed on more than one host, the following network traffic flows need to be allowed. All is TCP protocol unless otherwise noted. Not all ports will need to be open for every supported architecture. Secure ports are only required when explicitly configured.

Required User Accounts in Dependencies

The validated designs are very opinionated installations of Itential and its dependencies. The following user accounts are required by the dependencies for a validated design and are used for the following purposes.

MongoDB

Redis