Does unicast A-P HA support having multiple cluster EIPs and secondary IP addresses on ENI0/port1?
Yes. It moves over any secondary IP addresses associated to ENI0\port1 and EIPs associated to those secondary IP addresses to the new primary FortiGate instance. To configure additional secondary IP addresses on the ENI and in FortiOS for port1, reference this use-case guide on the Fortinet AWS microsite.
Does unicast A-P HA support having multiple routes for ENI1\port2?
Yes. It moves any routes (regardless of the network CIDR) found in AWS route tables that are referencing any of the current master FortiGate instance’s data plane ENIs (ENI0\port1, ENI1\port2, etc) to the new primary unit on a failover event.
What VPC configuration is required when deploying either of the existing VPC CFTs?
The existing VPC CFTs are expecting the same VPC configuration that is provisioned in the new VPC CFTs. The existing customer VPC must have four subnets in the same AZ to cover the required public, private, HAsync, and HAmgmt subnets. Another critical point is that the public and HAmgmt subnets must be configured as public subnets. This means that an Internet gateway must be attached to the VPC and a route table, with a default route using the Internet gateway, must be associated to the public and HAmgmt subnets.
During a failover test, we see successful failover to a new primary FortiGate instance, but then when the original primary unit is online, it becomes the primary unit again.
The primary unit selection process of unicast A-P HA ignores HA uptime differences unless they are larger than five minutes. The HA uptime is visible in the GUI under System > HA. This is expected and the default behavior for FortiOS but can be changed in the CLI with
config system ha. For further details on master selection and how to influence the process, reference primary unit selection section of the HA chapter in the FortiOS Handbook.
During a failover test we see A-P HA select a new primary unit, but the AWS API is not updated to point to the new primary FortiGate instance.
Confirm the FortiGates' configurations are in sync and are properly selecting a new primary unit by seeing the HA role change as expected in the GUI under System > HA or CLI with
get sys ha status. However if during a failover, the secondary IP addresses of ENIs, routes, and cluster EIPs are not updated, then your issue is likely to do with direct Internet access via the HAmgmt interface (ENI3\port4) of the FortiGates or IAM instance role permissions issues.
For details on the IAM instance profile configuration that should be used, reference the policy statement attached to the ‘InstanceRole’ resource in any of the CFTs.
For the HAmgmt interface, confirm this is configured properly in FortiOS under the
config system ha section of the CLI.
Also confirm that the subnet the HAmgmt interface is associated to is a subnet with public Internet access and that this interface has an EIP associated to it. This means that an Internet gateway must be attached to the VPC, and a route table with a default route to the Internet gateway must be associated to the HAmgmt subnet.
You can debug AWS API calls on the FortiGate instance that is becoming the primary instance using the following CLI commands:
# diag deb app awsd -1
# diag deb enable
You can disable this with the following CLI commands:
# diag deb app awsd 0
# diag deb disable
Is it possible to remove direct Internet access from the HAmgmt subnet and provide private AWS EC2 API access via a VPC interface endpoint?
First, a dedicated method of access to the FortiGate instances must be set up to allow dedicated access to the HAmgmt interfaces. This method of access should not use the primary FortiGate instance so that either instance can be accessed regardless of the cluster status. Examples of dedicated access are Direct connect or IPsec VPN connections to an attached AWS VPN gateway. See AWS documentation.
Second, you should configure the FortiGates to use the 169.254.169.253 IP address for the AWS intrinsic DNS server as the primary DNS server to allow proper resolution of AWS API hostnames during failover to a new primary FortiGate. Here is an example of how to configure this with CLI commands:
# config system dns
# set primary 169.254.169.253
Finally, you must deploy the VPC interface endpoint into the HAmgmt subnet and must also have private DNS enabled to allow DNS resolution of the default AWS EC2 API public hostname to the VPC endpoint private IP address. This means that the VPC also must have both DNS resolution and hostname options enabled as well. See AWS documentation.