Introduction
This white paper explores the technologies that provide the foundation for Itential’s Automation Platform (IAP) (version 2023.1 onwards) High Availability (HA) capability and discusses multiple deployment scenarios.
Business-critical applications have always required continuous availability. As more organizations launch services online for consumption by global audiences, availability and scalability across distributed geographic regions become increasingly important considerations in systems design.
There are three principal reasons for the geographic distribution of databases across multiple data centers:
Continuous Availability
Whether the database is deployed on-premises or in a public cloud, the business needs assurance that the service will survive a regional disaster that causes a complete data center outage. Examples include fires, floods, or hurricanes. Gartner estimates downtime costs a business an average of $300,000 per hour, with losses much higher for global, internet-based operations1.
Customer Experience
Global audiences need consistent, low latency experiences, wherever they are located. Amazon famously concluded that each 100ms in added latency resulted in a 1% loss of sales.
Regulatory Compliance
National governments and industry regulatory agencies are placing controls on where customer data is physically located, as well as expectations of service restoration times in case of catastrophic system outages.
Under normal operating conditions, an IAP deployment will perform according to the performance and functional goals of the system. However, from time to time, certain inevitable failures or unintended actions can affect a system in adverse ways. Storage devices, network connectivity, power supplies, and other hardware components will fail.
These risks are often mitigated through use of redundant hardware components. Similarly, an IAP deployment provides configurable redundancy throughout both its software components and its data storage.
IAP Architecture
Before describing IAP’s HA capability, a brief explanation of the platform’s architecture is required. IAP is developed using a combination of MEAN2 and MERN3 stack. Each of these is a set of Open-Source components that together provide an end-to-end framework for building dynamic web applications, starting from the top (code running in the browser) to the bottom (database). The stack is made up of:
MongoDB
Document database – used by a back-end application to store its data as JSON (JavaScript Object Notation) documents.
Express
(sometimes referred to as Express.js): Back-end web application framework running on top of Node.js.
AngularJS and ReactJS
Front-end web app framework; runs JavaScript code in the user’s browser, allowing the application’s UI to be dynamic.
Node.js
JavaScript runtime environment – allows implementation of the application back-end in JavaScript.
In addition to the above, IAP uses Redis4 to manage Inter-Process Communication (IPC), Event Listening, and Client session tokens.
The following sections describe how each element of IAP architecture can be deployed to meet the customers’ High Availability (HA) and Disaster Recovery (DR) requirements, including Server availability, Client sessions management, and database replication and resilience.
Application Server Load Balancing
High Availability (HA) refers to a site being up for as long as possible. This means there is enough infrastructure in the right locations to ensure there is no single point of failure that could take the site down.
Both failover and load balancing are means to achieve high availability:
- Load balancing spreads the load of the application across multiple application servers or multiple web servers to help smooth out the peaks if there is a lot of traffic all at once. Load balancing is one element of implementing high availability.
- Failover protects applications from downtime by ensuring redundant equipment can take over if a certain part fails.
There are numerous techniques that can be used to load balance client access requests across servers. When the load is low, one of the simple load balancing methods will suffice. But in times of high load, the more complex methods are used to ensure an even distribution of requests under network and service stress.
Here is a small list of example techniques that load balancers use.
Ordered List
This method is used when deploying an ‘active/standby’ architecture (see below for more detail).
Each server within a group is assigned a number that corresponds to the rank within the group. Using the ranking of each answer, the load-balancer tries each server in the order that has been assigned, selecting the first available live answer to serve a user request. List members are given precedence and tried in order, and a member is not used unless all previous members fail to provide a suitable result.
Round Robin
A set of servers are configured to handle load in a rotating sequential manner. The algorithm assumes that each server is able to process the same number of requests and isn’t able to account for active connections.
Weighted Round Robin
Servers are rated based on the relative number of requests each one can process. Those with higher capacities are sent more requests.
Least Connected
Requests are sent to the server with the fewest number of active connections, assuming all connections generate an equal amount of server load.
Weighted Least Connections
Servers are rated based on their processing capabilities. Load is distributed according to both the relative capacity of the servers and the number of active connections on each one.
Source IP Hash
The source and destination IP address in a request are combined to generate a hash key, which is then designated to a specific server. This allows a dropped connection to be returned to the same server originally handling it.
More complex/sophisticated algorithms are available, descriptions of which are outside the scope of this white paper.
The white paper assumes use of a load-balancer (either HAProxy5 or nginx6).
Note: The actual configuration of load-balancers is outside the scope of this document. It is enough to know that IAP supports multiple load balancing patterns.
IAP Health Checks
To handle failover, IAP provides a health check API that can be used by load balancers to ensure traffic is not directed to a server that is currently unavailable. By using the [hostname]/status API, load balancer routes requests only to the healthy instances. When the load balancer determines that an instance is unhealthy, it stops routing requests to that instance. The load balancer resumes routing requests to the instance when it has been restored to a healthy state.
IAP Validated Designs
IAP supports multiple patterns that support a wide range of customer deployment requirements, from simple single-server high availability to full blue/green multi-region disaster recovery configuration.
This section outlines the different patterns customers can deploy.
High Availability – The Basics
The basic principle of HA systems specifies a design that aims to minimize the downtime of a system or service and keep them continuously available. IAP achieves this goal by using components that can be set up in high-availability clusters. The diagram below depicts a simplified high availability system architecture. In addition, the sections below on Redis and database replication explain in more details how HA is achieved at each layer.
The recommended minimum number of servers required to implement HA for IAP is 11 (as illustrated in the diagram below) — this being two Itential application servers, three Redis, and three MongoDB servers.
Latency
To avoid potential latency-related issues of data replication between clusters, it is recommended that each IAP stack is deployed within the same data center (DC). However, if each data center is connected via very low latency links (e.g., sub 5m/s), then it is possible to deploy a stack that spans DCs. In that scenario, extensive testing should be performed to ensure latency issues do not arise.
Figure 1 IAP Typical HA Deployment
Figure 2 Active Standby Load Balancing
Basic Active/Standby
Also known as hot/warm or Active/Passive, active-standby consists of at least two nodes. However, as the name “active-standby” implies, not all nodes are going to be active. In the case of two nodes, for example, if the first node is already active, the second node must be passive or on standby.
The standby (a.k.a. failover) server acts as a backup that can take over as soon as the active (a.k.a. primary) server is disconnected or becomes unable to serve.
Highly Available Architecture (HA2)
Also known as ‘hot/hot’, an active-active cluster is also typically made up of at least two nodes, but unlike active/standby, the nodes are actively running the same kind of service simultaneously. The main purpose of an active-active cluster is twofold: to provide component redundancy, which can tolerate at least one catastrophic failure, and to achieve load balancing. Load balancing distributes workloads across all nodes in order to prevent any single node from getting overloaded. Because there are more nodes available to serve, there will also be a marked improvement in throughput and response times.
Figure 3 Active/Active Load Balancing
Assigning of clients to the nodes in the cluster is not an arbitrary process. Rather, it is based on whatever load balancing algorithm is set on the load balancer. For example, in a “Round Robin” algorithm with two servers, the first client to connect is sent to the 1st server, the second client is sent to the 2nd server, the 3rd client back to the 1st server, the 4th client back to the 2nd server, and so on.
Figure 4 Active/Standby Architecture (ASA)
HA Active/Standby Architecture (ASA)
An Active/Standby Architecture (ASA) is an architecture that should be considered when Disaster Recovery is required. It is not required that they be geographically redundant, but they could be. The intent is to provide a standby environment that can be used in the event of a disaster that impacts the active stack. The standby stack should be preconfigured to quickly take over as the active stack.
The ideal ASA architecture will appear as two HA2 deployments, except for MongoDB. One or more of the MongoDB instances must be hosted in the standby location as a replica of the primary. Ideally, the MongoDB cluster will consist of at least 5 members. This will allow for a cluster of three mongos (query router components) in the worst-case disaster scenario.
Blue/Green Deployment (BG)
This deployment requires two separate and complete Itential stacks. There is no constraint from a data center boundary, these stacks can exist in separate or the same data center. Control of the active environment is driven by the GLB load balancer. Components and configurations are not shared between the stacks: meaning, any changes made to one will have zero impact on the other.
This deployment is intended to reduce downtime as much as possible. A major advantage of this pattern is deployments and upgrades can be staged in advance during normal business hours against the inactive stack without impacting the active environment. Changes can be pushed, via automated CI/CD pipelines, to the inactive stack (green) without impacting the active deployment (blue). This allows ‘smoke tests’ to be performed in the production environment (devices, integrated ITSM systems etc.) in a limited and controlled manner, and when successful the flip from inactive to active (and vice versa) can be done via the GLB. In flight jobs on the old stack will continue to run until they are complete.
Figure 5 Blue/Green Architecture (BG)
Figure 6 Developer Architecture
Developer Architecture
A Developer Architecture is an Itential architecture where all of the components are installed as singletons and can not gracefully tolerate failures. This architecture is not recommended for production environments and testing environments that require strict uptimes and redundancies. This architecture is suitable for development environments only. The IAP application performs many reads and writes against the database and is sensitive to great latencies. All components must be installed in the same data center.
The ideal ASA architecture will appear as two HA2 deployments, except for MongoDB. One or more of the MongoDB instances must be hosted in the standby location as a replica of the primary. Ideally, the MongoDB cluster will consist of at least 5 members. This will allow for a cluster of three mongos (query router components) in the worst-case disaster scenario.
Redis Configuration
The Itential Automation Platform (IAP) uses Redis for the following function:
Shared authentication token storage and expiration
Shared Authentication Session Tokens
Once a connection request to IAP has been successfully authenticated, either via an API call or a user connection from a browser, the session is allocated a token, which is then stored in Redis. Any subsequent call made within IAP will remain authenticated while the session token remains valid. The following sections discuss the configuration options available in an active/standby and active/active setup.
Active/Standby
When configured in Active/Standby mode, each instance of Redis is configured standalone, and tokens are not shared. When using a load-balancer that has been configured with ‘sticky’ connections, all subsequent requests will be sent to the same IAP server, and the session tokens are valid.
Using Figure 6 as an example, assuming that the IAP server in Data Center 1 fails, the load balancer will redirect traffic to the alternate IAP server. As the session tokens stored in the Redis instance in DC1 are not replicated to DC2, the session token submitted with the request is invalid and the request is rejected. As a result, any session will need to be re-established via new login request.
Figure 7 Redis Active/Standby support
Figure 8 Redis Active/Active support
Active/Active
When running in active/active mode, the session tokens must be replicated to all Redis databases configured as a cluster. With this configuration, using the example below, when the ‘Itential 1’ server fails, the load balancer will direct traffic to the ‘Itential 2’ server. As the session tokens have been replicated to the Redis database, the session is deemed active, and the client application is not impacted by the failure.
To achieve an HA Topology, Redis Cluster functionality provides primary-secondary data replication across nodes. For monitoring and failover, Redis provides Redis Sentinel to automatically detect failures and bring the cluster back to a stable mode. Sentinel always checks the Primary and Secondary instances in the Redis cluster to determine whether they are working as expected. If Sentinel detects a failure in the Primary node in each cluster, Sentinel will start a failover process. As a result, Sentinel will pick a Secondary instance and promote it to become the Primary instance. Ultimately, the other remaining Secondary instances will be automatically reconfigured to use the new Primary instance.
Given Itential’s enhanced HA capabilities, active/active is, when possible, the recommended configuration option for Redis.
Database Replication
As stated above, the database used by IAP is MongoDB. Out of the box, MongoDB provides extensive HA and data replication functionality, as well as application drivers7.
Using the node.js specific driver allows IAP to leverage all of the HA and replication capabilities that MongoDB provides. The primary functionality is the ability to monitor the state of the MongoDB deployment and ensure that the application is always connected to the Primary server. The sections below describe Replica Sets and member elections in more detail. Additionally, more details on MongoDB drivers are available in the following blog post.8
Replica Sets
MongoDB maintains multiple copies of data, called replica sets, using native replication. Users should deploy replica sets to help prevent database downtime. Replica sets are self-healing as failover and recovery is fully automated, so it is not necessary to manually intervene to restore a system in the event of a failure.
Replica sets also enhance operational flexibility by providing a way to perform system maintenance (i.e., upgrading hardware and software) while preserving service continuity. This is an important capability as these operations can account for as much as one third of all downtime in traditional systems.
Figure 9 Self-Healing MongoDB Replica Sets for Continuous Availability
A replica set consists of multiple database replicas. At any given time, one member acts as the primary replica set member, and the other members act as secondary replica set members. If the primary member suffers an outage (e.g., a power failure, hardware fault, or network partition), one of the secondary members is automatically elected to primary, typically within several seconds, and the client connections automatically failover to that new primary member.
The number of replicas in a MongoDB replica set is configurable, with a larger number of replica members providing increased data durability and protection against database downtime (e.g., in case of multiple machine failures, rack failures, data center failures, or network partitions). Up to 7 voting members can be configured per replica set, providing operational flexibility and wide data distribution across multiple geographic sites.
You can learn more about the members of a replica set from the documentation.
Figure 10 MongoDB Replica Set Elections
Replica Set Elections
In the event of a primary replica set member failing, the election process is controlled by sophisticated algorithms based on an extended implementation of the Raft consensus protocol. Not only does this allow fast failover to maximize service availability, but the algorithms also ensure only the most suitable secondary members are evaluated for election to primary and reduce the risk of unnecessary failovers (“false positives”). Before a secondary replica set member is promoted, the election algorithms evaluate a range of parameters, including:
- Analysis of election identifiers, timestamps, and journal persistence to identify those replica set members that have applied the most recent updates from the primary member.
- Heartbeat and connectivity status with the majority of other replica set members.
- User-defined priorities assigned to replica set members. For example, administrators can configure all replicas located in a secondary data center to be candidates for election only if the primary data center fails.
MongoDB Data Center Awareness
MongoDB provides a rich set of features to help users deploy highly available and scalable systems. In designing for high availability, administrators must evaluate read and write operations in the context of different failure scenarios. The performance and availability SLAs (Service Level Agreements) of a system play a significant role in determining:
- The number of replicas (copies) of the data.
- The physical location of replica sets, both within and across multiple data centers.
Administrators can configure the behavior of MongoDB’s replica sets to enable data center awareness. Configuration can be based on a number of different dimensions, including awareness of geographical regions in deployments to multiple data centers, or racks, networks, and power circuits in a single data center.
With MongoDB, administrators can:
- Ensure write operations propagate to specific members of a replica set, deployed locally and in remote data centers. This reduces the risk of data loss in the event of a complete data center outage. Alternatively, replication can be configured to ensure data is only replicated to nodes within a specific region to ensure data never leaves a country’s borders.
- Place specific data partitions on specific shards, each of which could be deployed in different data centers. Again, this can be used to reduce geographic latency and maintain data sovereignty.
Read and write behaviors can be configured per operation and per collection. Collectively, they enable users to precisely control and scale database operations across regions, based on specific application requirements.
Configuring Write Operations Across Data Centers
MongoDB allows users to specify write availability in the system, using an option called write concern. Each operation can specify the appropriate write concern, ranging from unacknowledged to an acknowledgement that writes have been committed to:
- A Single Replica (i.e., the primary replica set member).
- Multiple replicas.
- A majority of replicas.
- All replicas.
It is also possible to configure the write concern so that writes are only acknowledged once specific policies have been fulfilled, such as writing to at least two replica set members in one data center and at least one replica in a second data center.
Conclusion
As detailed in this white paper, the use of robust load-balancing techniques, clustered Redis deployments, and geographically distributed MongoDB replica sets allows an organization to design an HA infrastructure that will allow rapid response to events, minimize the number of disruptions, and nearly eliminate the resulting periods of downtime.