Advanced HA heartbeat settings
Administrators can use advanced heartbeat settings to fine-tune cluster performance.
Configure the following settings for your environment as needed:
|
Topic |
Summary |
|---|---|
|
The customization of interval settings and loss thresholds to control how fast the cluster detects a peer failure allowing administrators to balance between sub-second failover speeds and tolerance for network latency. |
|
|
A configuration setting that adjusts how long a unit searches for peers during startup or recovery before initializing ensuring stability in environments with communication delays between sites. |
|
|
A security feature that applies AES-128 encryption and SHA1 authentication to heartbeat packets protecting cluster credentials and preventing the injection of false status messages on shared networks. |
Modifying heartbeat timing
The heartbeat interval and heartbeat lost threshold are two variables that dictate the length of time one cluster unit will wait before determining a peer is dead.
config system ha
set hb-interval <integer>
set hb-interval-in-milliseconds {100 | 10}
set hb-lost-threshold <integer>
end
|
Option |
Description |
|---|---|
|
hb-interval <integer> |
Set the time between sending heartbeat packets; increase to reduce false positives (1 - 20, default = 2). |
|
hb-interval-in-milliseconds {100 | 10} |
Set the number of milliseconds for each heartbeat interval (100 or 10, default = 100). |
|
hb-lost-threshold <integer> |
Set the number of lost heartbeats to signal a failure; increase to reduce false positives (1 - 60, default = 20). |
Heartbeats are sent out every 2 × 100 ms, and it takes 20 consecutive lost heartbeats for a cluster member to be detected as dead. Therefore, it takes by default 2 × 100 ms × 20 = 4000 ms, or 4 seconds, for a failure to be detected.
Sub-second heartbeat failure detection can be achieved by lowering the interval and threshold or lowering the heartbeat interval unit of measurement from 100 ms to 10 ms.
If the primary unit does not receive a heartbeat packet from a subordinate unit before the heartbeat threshold expires, the primary unit assumes that the subordinate unit has failed.
If a subordinate unit does not receive a heartbeat packet from the primary unit before the heartbeat threshold expires, the subordinate unit assumes that the primary unit has failed. The subordinate unit then begins negotiating to become the new primary unit.
The HA heartbeat packets consume more bandwidth if the heartbeat interval is short. But if the heartbeat interval is very long, the cluster is not as sensitive to topology and other network changes. Therefore, gauge your settings based on the amount of traffic and CPU usage sustainable by the cluster units versus the tolerance for an outage when the primary unit fails. Avoid using the heartbeat interfaces as traffic ports to prevent congesting the interfaces.
Changing the time to wait in the hello state
The hello state hold down time is the number of seconds that a cluster unit waits before changing from hello state to work state. After a failure or when starting up, cluster units operate in the hello state to send and receive heartbeat packets so that all the cluster units can find each other and form a cluster. A cluster unit should change from the hello state to work state after it finds all the other FortiGates to form a cluster with.
If all cluster units cannot find each other during the hello state, then some cluster units may join the cluster after it has formed. This can cause disruptions to the cluster and affect how it operates. A delay could occur if the cluster units are located at different sites or if communication is delayed between the heartbeat interfaces. If delays occur, increase the cluster units wait time in the hello state.
config system ha
set hello-holddown <integer>
end
|
Option |
Description |
|---|---|
|
hello-holddown <integer> |
Set the time to wait before changing from hello to work state, in seconds (5 - 300, default = 20). |
Configuring HA heartbeat encryption and authentication
HA heartbeat encryption and authentication to encrypt and authenticate HA heartbeat packets can be enabled. HA heartbeat packets should be encrypted and authenticated if the cluster interfaces that send HA heartbeat packets are also connected to the networks. HA heartbeat encryption and authentication are disabled by default. Note that enabling these settings could reduce cluster performance.
config system ha
set authentication {enable | disable}
set encryption {enable | disable}
end
If HA heartbeat packets are not encrypted, the cluster password and changes to the cluster configuration could be exposed. An attacker may be able to sniff HA packets to get cluster information. Enabling HA heartbeat message authentication prevents an attacker from creating false HA heartbeat messages. False HA heartbeat messages could affect the stability of the cluster.
HA authentication and encryption uses AES-128 for encryption and SHA1 for authentication. Heartbeat messages are encrypted and encapsulated in ESP packets for transfer in an IPsec tunnel between the cluster members.