Fortinet black logo

HA troubleshooting

Copy Link
Copy Doc ID 9d9a7f8b-8422-11ea-9384-00505692583a:861362
Download PDF

HA troubleshooting

HA management interface

For HA-AP mode, you are not able to access the slave device directly if you didn’t enable the management interface. You can only access slave CLI from master via executing command: “execute ha manage 0” in this scenario. So we recommend you enable the HA management interface for both nodes.

Reminder: Please use “mgmt.-interface” under “config system ha” instead of the old dedicate interface under “config system interface”, due to the old dedicate interface has some limitations.

In most cases, you could configure the manage-interface with the IP address same subnets with original port1 or mgmt, so it can be conflict. You’d better have the console control, and then clear the old management IP address on the old interface (typical port1 or mgmt), then set it under “config system ha”.

FAD2 # config system ha

FAD2 (ha) # set mgmt-status enable

FAD2 (ha) # set mgmt-interface port1

FAD2 (ha) # set mgmt-ip 10.106.188.42/23

FAD2 (ha) # set mgmt-ip-allowaccess http https ping snmp ssh telnet

FAD2 (ha) # end

Don’t forget to configure the default route accordingly.

For HA-AA and HA-VRRP mode, you don’t have to configure the HA manage-interface, because HA-AA mode uses the IP address of its own local-id, and HA-VRRP mode can have its own interface IP.

For virtualization platform like VMware ESXi, KVM, Hyper-V, please enable the “Promiscuous mode” or “Mac address spoofing” for the management-interface mother interface.

HA config out of sync

Once the HA peers established, all the config can be synced to HA peers by default. There are two kinds of config sync happening, incremental sync and full sync. The incremental sync happens if one of the HA nodes have configuration changes, then the changes will be synced to the HA peers. The full sync happens when the HA daemon restarting triggered, such as the new HA peer joined group.

Normally, you can always see the “In Sync” on the top of the GUI. But if something unknown happened, there could be out of sync happening. In this case, please click the config difference detail at the same position of “In Sync” to see the difference, then you can correct it manually on both devices, or execute the command “execute ha force sync-config” on the correct config side.

In some scenarios, the above 2 methods can’t work. Then you have to backup the correct side full config file, and then restore it to the false HA peer.

HA on Microsoft Hyper-V platform

HA behavior is similar in other platforms except Microsoft Hyper-V. There are some Hyper-V limitations impacting the HA behavior.

  1. Not like in other platform, HA-AP mode on Hyper-V platform utilizes the real Mac address specified by Hyper-V, while in other platform, HA-AP mode uses the virtual Mac address.
  2. You have to enable the “Enable Mac address spoofing” for all the heartbeat/data ports and management-interface.
  3. Please assign the individual virtual switch for the heartbeat/data ports due to Hyper-V virtual switch implementation. Otherwise, the HA state could be unstable.
  4. It is not supported to set Mac address on Hyper-V platform.

HA abnormal state

If somehow you encounter the HA abnormal state, such as dual Master, long time waiting to sync. Please check the heartbeat/data ports connectivity. Technically, the heartbeat/data ports should be connected directly, or at least in the same VLAN via switches. If they are connected correctly, then you can enable the debug to see the abnormal reason for the state. Please refer to “4. HA Debug” in this guide to see how to use HA debug command. Here we put an example to debug.

Example:

You can enable the “heartbeat” debug option to see if the heartbeat message was received and sent successfully. If all the heartbeat messages are received and sent properly. Then enable the “errors” option to see if there are errors happening, if so, record it, and try to resolve it. If no more found, please try other options according to the table listed in “4. HA Debug” section of this guide.

Upgrade Firmware

Users can upgrade all ADC units in that group with one click, that just need to enable the HA sync in upgrade firmware.

But there may be active session traffic loss during image upgrade with HA SYNC enabled. If you want to avoid the traffic loss, please do not enable this HA sync feature, and upgrade the firmware on each ADC nodes separately, make sure the upgraded ADC is fully booted up, then start the next one upgrade.

HA troubleshooting

HA management interface

For HA-AP mode, you are not able to access the slave device directly if you didn’t enable the management interface. You can only access slave CLI from master via executing command: “execute ha manage 0” in this scenario. So we recommend you enable the HA management interface for both nodes.

Reminder: Please use “mgmt.-interface” under “config system ha” instead of the old dedicate interface under “config system interface”, due to the old dedicate interface has some limitations.

In most cases, you could configure the manage-interface with the IP address same subnets with original port1 or mgmt, so it can be conflict. You’d better have the console control, and then clear the old management IP address on the old interface (typical port1 or mgmt), then set it under “config system ha”.

FAD2 # config system ha

FAD2 (ha) # set mgmt-status enable

FAD2 (ha) # set mgmt-interface port1

FAD2 (ha) # set mgmt-ip 10.106.188.42/23

FAD2 (ha) # set mgmt-ip-allowaccess http https ping snmp ssh telnet

FAD2 (ha) # end

Don’t forget to configure the default route accordingly.

For HA-AA and HA-VRRP mode, you don’t have to configure the HA manage-interface, because HA-AA mode uses the IP address of its own local-id, and HA-VRRP mode can have its own interface IP.

For virtualization platform like VMware ESXi, KVM, Hyper-V, please enable the “Promiscuous mode” or “Mac address spoofing” for the management-interface mother interface.

HA config out of sync

Once the HA peers established, all the config can be synced to HA peers by default. There are two kinds of config sync happening, incremental sync and full sync. The incremental sync happens if one of the HA nodes have configuration changes, then the changes will be synced to the HA peers. The full sync happens when the HA daemon restarting triggered, such as the new HA peer joined group.

Normally, you can always see the “In Sync” on the top of the GUI. But if something unknown happened, there could be out of sync happening. In this case, please click the config difference detail at the same position of “In Sync” to see the difference, then you can correct it manually on both devices, or execute the command “execute ha force sync-config” on the correct config side.

In some scenarios, the above 2 methods can’t work. Then you have to backup the correct side full config file, and then restore it to the false HA peer.

HA on Microsoft Hyper-V platform

HA behavior is similar in other platforms except Microsoft Hyper-V. There are some Hyper-V limitations impacting the HA behavior.

  1. Not like in other platform, HA-AP mode on Hyper-V platform utilizes the real Mac address specified by Hyper-V, while in other platform, HA-AP mode uses the virtual Mac address.
  2. You have to enable the “Enable Mac address spoofing” for all the heartbeat/data ports and management-interface.
  3. Please assign the individual virtual switch for the heartbeat/data ports due to Hyper-V virtual switch implementation. Otherwise, the HA state could be unstable.
  4. It is not supported to set Mac address on Hyper-V platform.

HA abnormal state

If somehow you encounter the HA abnormal state, such as dual Master, long time waiting to sync. Please check the heartbeat/data ports connectivity. Technically, the heartbeat/data ports should be connected directly, or at least in the same VLAN via switches. If they are connected correctly, then you can enable the debug to see the abnormal reason for the state. Please refer to “4. HA Debug” in this guide to see how to use HA debug command. Here we put an example to debug.

Example:

You can enable the “heartbeat” debug option to see if the heartbeat message was received and sent successfully. If all the heartbeat messages are received and sent properly. Then enable the “errors” option to see if there are errors happening, if so, record it, and try to resolve it. If no more found, please try other options according to the table listed in “4. HA Debug” section of this guide.

Upgrade Firmware

Users can upgrade all ADC units in that group with one click, that just need to enable the HA sync in upgrade firmware.

But there may be active session traffic loss during image upgrade with HA SYNC enabled. If you want to avoid the traffic loss, please do not enable this HA sync feature, and upgrade the firmware on each ADC nodes separately, make sure the upgraded ADC is fully booted up, then start the next one upgrade.