Multiple FortiSwitches in tiers via aggregate interface with MCLAG enabled on all tiers

This example provides a recommended configuration of FortiLink where multi-tier FortiSwitch devices are managed by an A-P mode HA cluster of FortiGates acting as a switch controller via an aggregate interface. The FortiGates provide A-A links to two distribution FortiSwitches that are connected to each other by MCLAG. All access FortiSwitch devices have A-A links with two upper tier FortiSwitches, as long as the MCLAG-ICL has been enabled between the upper tiers.

Prerequisites:

• The FortiGate model supports an aggregate interface.
• FortiSwitch units have been upgraded to latest released software version.
• Layer-3 path/route in the management VDOM is available to Internet so that the FortiSwitch units can synchronize NTP.
• For the FortiSwitch D series, the models above 4 just support MCLAG. For the FortiSwitch E series, the models above 2 just support MCLAG.

Change the FortiSwitch management mode to FortiLink:

Enter the following CLI commands on the FortiSwitch:

config system global 
   set switch-mgmt-mode fortilink 
end
This operation will cleanup all of the configuration and reboot the system!
Do you want to continue? (y/n)y
Backing up local mode config before entering FortiLink mode....

If the FortiSwitch ports used for the FortiLink connection have auto-discovery-fortilink enabled, executing authorization on FortiGate will trigger the transformation to FortiLink mode automatically.

config switch interface
  edit "port1"
       set auto-discovery-fortilink enable
       ……
  next			
end

Set up an A-P mode HA cluster:

See HA active-passive cluster setup.

Create an aggregate interface and designate it as Fortilink interface on the FortiGate:

Using the CLI:

config system interface
    edit "aggr1"
        set vdom "vdom1"
        set fortilink enable
        set type aggregate
        set member "port11" "port12"
        set fortilink-split-interface disable
    next
end

fortilink-split-interface must be disabled for MCLAG to work.

Using the GUI:

1. Go to WiFi & Switch Controller > FortiLink Interface.
2. In Interface members, select one or more physical ports that are connected to different distribution FortiSwitches to create an aggregate interface.
3. Disable FortiLink split interface.
4. Configure other fields as necessary.
5. Click OK.

Discover and authorize the FortiSwitch:

Using the CLI:

config switch-controller managed-switch
    edit "FSWSerialNum"
        set fsw-wan1-admin enable
        ……
    next
end

Check the CLI output for Connection: Connected to show that FortiLink is up:

execute switch-controller get-conn-status FSWSerialNum

Get managed-switch S248EPTF18001384 connection status:
Admin Status: Authorized 
Connection: Connected
Image Version: S248EP-v6.2.0-build143,190107 (Interim)
Remote Address: 2.2.2.2
Join Time: Fri Jan 11 15:22:32 2019

   interface   status   duplex    speed fortilink stacking       poe status
       port1       up     full  1000Mbps       no       no  Delivering Power
       port2     down      N/A     0           no       no         Searching
        ……

Using the GUI:

1. Go to WiFi & Switch Controller > Managed FortiSwitch.
2. Click Authorize and wait for a few minutes for the connection to be established.

   When FortiLink between the FortiGate and FortiSwitch is established, the Link-up ports change to green and the POE port that is supplying power changes to blue. The dotted line between the FortiGate and FortiSwitch changes to a solid line. The Connection status shows that FortiLink is up.

Enable MCLAG on the ICL link between the distribution FortiSwitch devices:

conf switch trunk
    edit "4DN4K15000008-0"
        set mclag-icl enable
    next
end

When you enable mclag-icl, MCLAG on the FortiLink interface is enabled automatically and active-active backup links between the distribution FortiSwitches are established.

Extend the security perimeter to the edge of FortiSwitch:

1. Configure the VLAN arrangement.
   1. On the FortiGate, go to WiFi & Switch Controller > FortiSwitch VLANs.
   2. Configure the VLAN interfaces that are applied on FortiSwitch.

      On FortiGate, these switch VLAN interfaces are treated as layer-3 interfaces and are available to be applied by firewall policy and other security controls in FortiOS. This means that security boundary is extended to FortiSwitch.

2. Configure FortiSwitch ports.
   1. On the FortiGate, go to WiFi & Switch Controller > FortiSwitch Ports.
   2. Select one or more FortiSwitch ports and assign them to the switch VLAN.
   3. You can also select POE/DHCP Snooping, STP, and other parameters for the FortiSwitch ports to show their real-time status such as link status, data statistics, etc.
3. Configure access authentication.
   1. On the FortiGate, go to WiFi & Switch Controller > FortiSwitch Security Policies.
   2. Configure the 802.1X security policies.
   3. Select Port-based or MAC-based mode and select User groups from the existing VDOM.
   4. Configure other fields as necessary.
   5. Go to WiFi & Switch Controller > FortiSwitch Ports.
   6. Select one or more FortiSwitch ports, click + in the Security Policy column, then make a selection from the pane.

Troubleshooting

Authorized FortiSwitch always offline

If an authorized FortiSwitch is always offline, go to the FortiGate CLI and use the command below to see all the checkpoints. Inspect each checkpoint to find the cause of the problem.

execute switch-controller diagnose-connection  S248EPTF18001384


Fortilink interface ... OK
aggr1  enabled

DHCP server ... OK
aggr1  enabled

NTP server ... OK
aggr1  enabled
NTP server sync ... OK
synchronized: yes, ntpsync: enabled, server-mode: enabled

ipv4 server(ntp1.fortiguard.com) 208.91.113.70 -- reachable(0x80) S:2 T:128 
         no data
ipv4 server(ntp2.fortiguard.com) 208.91.113.71 -- reachable(0x80) S:2 T:128 
         no data
ipv4 server(ntp2.fortiguard.com) 208.91.112.51 -- reachable(0xff) S:2 T:66 selected 
        server-version=4, stratum=2
        reference time is dfe3aec5.744404e6 -- UTC Sat Jan 12 00:09:41 2019
        clock offset is -0.320411 sec, root delay is 0.054535 sec
        root dispersion is 0.533081 sec, peer dispersion is 11495 msec

ipv4 server(ntp1.fortiguard.com) 208.91.112.50 -- reachable(0xff) S:2 T:66 
        server-version=4, stratum=2
        reference time is dfe3aec5.744404e6 -- UTC Sat Jan 12 00:09:41 2019
        clock offset is -0.448087 sec, root delay is 0.054535 sec
        root dispersion is 0.533081 sec, peer dispersion is 12542 msec

HA mode ... disabled

Fortilink
Status ... SWITCH_AUTHORIZED_READY
Last keepalive ... 1 seconds ago

CAPWAP
Remote Address: 2.2.2.2
Status ... CONNECTED
Last keepalive ... 26 seconds ago


PING 2.2.2.2 (2.2.2.2): 56 data bytes
64 bytes from 2.2.2.2: icmp_seq=0 ttl=64 time=1.1 ms
64 bytes from 2.2.2.2: icmp_seq=1 ttl=64 time=13.9 ms
64 bytes from 2.2.2.2: icmp_seq=2 ttl=64 time=12.7 ms
64 bytes from 2.2.2.2: icmp_seq=3 ttl=64 time=2.9 ms
64 bytes from 2.2.2.2: icmp_seq=4 ttl=64 time=1.2 ms

--- 2.2.2.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 1.1/6.3/13.9 ms

HA sync fails

If HA sync fails, use the command below to diagnose and locate the cause.

# diagnose sys ha checksum cluster 

================== FG5H0E39179XXX9 ==================

is_manage_master()=1, is_root_master()=1
debugzone
global: 2b e9 81 38 c2 9d 4f db b7 0e 1f 49 42 c6 1e fb 
vdom5: 3d dc e7 70 69 22 c3 12 a7 ac 68 06 21 21 ef 8f 
vdom3: 89 59 1f 45 7a 75 ae fc 71 bc 42 f4 5e c2 47 c8 
vdom2: b2 a5 f3 e7 85 02 62 e5 2a 23 23 64 04 66 76 cc 
vdom1: 1f b5 11 61 31 c4 0c 72 2e 97 8d d8 45 7e d6 0c 
root: af a6 48 c5 c2 9a 8b 81 a5 53 fb 27 e9 ae 01 6a 
all: 89 1f 63 77 48 8a 30 ee 57 06 ca eb 71 e6 8e ad 

checksum
global: 2b e9 81 38 c2 9d 4f db b7 0e 1f 49 42 c6 1e fb 
vdom5: 3d dc e7 70 69 22 c3 12 a7 ac 68 06 21 21 ef 8f 
vdom3: 89 59 1f 45 7a 75 ae fc 71 bc 42 f4 5e c2 47 c8 
vdom2: b2 a5 f3 e7 85 02 62 e5 2a 23 23 64 04 66 76 cc 
vdom1: 1f b5 11 61 31 c4 0c 72 2e 97 8d d8 45 7e d6 0c 
root: af a6 48 c5 c2 9a 8b 81 a5 53 fb 27 e9 ae 01 6a 
all: 89 1f 63 77 48 8a 30 ee 57 06 ca eb 71 e6 8e ad 

================== FG5H0E391790XXX4 ==================

is_manage_master()=0, is_root_master()=0
debugzone
global: 2b e9 81 38 c2 9d 4f db b7 0e 1f 49 42 c6 1e fb 
vdom5: 3d dc e7 70 69 22 c3 12 a7 ac 68 06 21 21 ef 8f 
vdom3: 89 59 1f 45 7a 75 ae fc 71 bc 42 f4 5e c2 47 c8 
vdom2: b2 a5 f3 e7 85 02 62 e5 2a 23 23 64 04 66 76 cc 
vdom1: 1f b5 11 61 31 c4 0c 72 2e 97 8d d8 45 7e d6 0c 
root: af a6 48 c5 c2 9a 8b 81 a5 53 fb 27 e9 ae 01 6a 
all: 89 1f 63 77 48 8a 30 ee 57 06 ca eb 71 e6 8e ad 

checksum
global: 2b e9 81 38 c2 9d 4f db b7 0e 1f 49 42 c6 1e fb 
vdom5: 3d dc e7 70 69 22 c3 12 a7 ac 68 06 21 21 ef 8f 
vdom3: 89 59 1f 45 7a 75 ae fc 71 bc 42 f4 5e c2 47 c8 
vdom2: b2 a5 f3 e7 85 02 62 e5 2a 23 23 64 04 66 76 cc 
vdom1: 1f b5 11 61 31 c4 0c 72 2e 97 8d d8 45 7e d6 0c 
root: af a6 48 c5 c2 9a 8b 81 a5 53 fb 27 e9 ae 01 6a 
all: 89 1f 63 77 48 8a 30 ee 57 06 ca eb 71 e6 8e ad