Network virtual functions

You must create network virtual functions (VFs) to use with the FortiGate-VM.

To determine which NICs are capable of running VFs:

[root@rhel-tiger-14-6 ~]# lspci -d ::0x200 -vv | egrep "Ethernet controller|SR-IOV"
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
19:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
5e:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
86:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
86:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
86:00.2 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
86:00.3 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
af:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
af:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
af:00.2 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
	 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
af:00.3 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
 Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

To determine the maximum amount of VFs that a PF can run:

[root@rhel-tiger-14-6 ~]# cat /sys/class/net/ens4f0/device/sriov_totalvfs
32

A useful way to see available NICs, the device name, and PCIe bus address is as follows:

[root@rhel-tiger-14-6 ~]# lshw -c network -businfo
Bus info          Device      Class          Description
========================================================
pci@0000:01:00.0  eno3        network        I350 Gigabit Network Connection
pci@0000:01:00.1  eno4        network        I350 Gigabit Network Connection
pci@0000:19:00.0  eno1        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:19:00.1  eno2        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:3b:00.0  ens1f0      network        Ethernet Controller E810-C for QSFP
pci@0000:3b:00.1  ens1f1      network        Ethernet Controller E810-C for QSFP
pci@0000:5e:00.0  ens2f0      network        Ethernet Controller E810-C for QSFP
pci@0000:5e:00.1  ens2f1      network        Ethernet Controller E810-C for QSFP
pci@0000:86:00.0  ens5f0      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:86:00.1  ens5f1      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:86:00.2  ens5f2      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:86:00.3  ens5f3      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:af:00.0  ens4f0      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:af:00.1  ens4f1      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:af:00.2  ens4f2      network        Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:af:00.3  ens4f3      network        Ethernet Controller XL710 for 40GbE QSFP+

There are standard ways to persist this across system restart. The following is an example script that is installed as a system service:

[root@rhel-tiger-14-6 ~]# touch /usr/local/bin/tiger-nic-vf-handler

[root@rhel-tiger-14-6 ~]# chown root:root /usr/local/bin/tiger-nic-vf-handler

[root@rhel-tiger-14-6 ~]# chmod 755 /usr/local/bin/tiger-nic-vf-handler

[root@rhel-tiger-14-6 ~]# cat /usr/local/bin/tiger-nic-vf-handler
#!/bin/bash

numvfs=2

lookup() {
  if [[ -z $1 ]] ; then
    echo ""
  else
    awk -v "id=$1" 'BEGIN { FS = "=" } $1 == id { print $2 ; exit }' $2
  fi
}

start() {
  for _ethdev in ${ethlist//;/ } ; do
    ethdev=${_ethdev#*:}
    echo $numvfs > /sys/class/net/${ethdev}/device/sriov_numvfs
    sleep 2
    for ((vfnum=0;vfnum<${numvfs};vfnum++)); do
      ip link set $ethdev vf $vfnum spoofchk off trust on
      for iface in $(ls /sys/class/net) ; do
        if [[ $(readlink /sys/class/net/${iface}) =~ $(readlink /sys/class/net/${ethdev}/device/virtfn${vfnum} xargs -n 1 basename) ]] ; then
          nmcli device set $iface managed no
        fi
      done 
    done
    ip link set dev $ethdev up
  done
}

stop() {
  for _ethdev in ${ethlist//;/ } ; do
    ethdev=${_ethdev#*:}
    echo 0 > /sys/class/net/${ethdev}/device/sriov_numvfs
  done
}

ethlist=$(lookup unmanaged-devices /etc/NetworkManager/conf.d/10-tiger.conf

case $1 in
  startstop) "$1" ;;
esac

[root@rhel-tiger-14-6 ~]# touch /etc/systemd/system/tiger-nic-vf-handler.service

[root@rhel-tiger-14-6 ~]# chown root:root /etc/systemd/system/tiger-nic-vf-handler.service

[root@rhel-tiger-14-6 ~]# chmod 644 /etc/systemd/system/tiger-nic-vf-handler.service

[root@rhel-tiger-14-6 ~]# cat /etc/systemd/system/tiger-nic-vf-handler.service
[Unit]
Description=Create/Destroy unmanaged virtual functions and NetworkManager
Before=libvirtd.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/tiger-nic-vf-handler start
ExecStop=/usr/local/bin/tiger-nic-vf-handler stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

[root@rhel-tiger-14-6 ~]# systemctl enable --now tiger-nic-vf-handler
Created symlink /etc/systemd/system/multi-user.target.wants/tiger-nic-vf-handler.service → /etc/systemd/system/tiger-nic-vf-handler.service.

The script simply sources the list of unmanaged interfaces from the NetworkManager configuration and sets up the VFs accordingly. The filename in the script must match the filename created earlier. You may want to update the number of VFs created with this script by updating the script variable accordingly. The script is set to run before the libvirtd service is started to allow dependent configuration to be persisted.

The spoofchk off setting allows the VM to define the MAC addresses that it associates to interfaces rather than those that the host. This is important when considering the deployment of LAGs and for FortiGate Clustering Protocol vMAC operation. The trust on setting is important to ensure that the VF tracks and follows the status of the PF, allowing the VM to detect interface down accordingly. This setting is also mandatory for LAG.

You may create the VFs as follows. Note the PCI addresses:

[root@rhel-tiger-14-6 ~]# lspci egrep "Eth.+Virt"
86:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:02.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:06.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:06.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:0a.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:0a.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:0e.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
86:0e.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:02.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:06.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:06.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:0a.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:0a.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:0e.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
af:0e.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02

Note that you can see the mapping of PF to VF as follows:

[root@rhel-tiger-14-6 ~]# ls -l /sys/class/net/ens4f0/device/virtfn*
lrwxrwxrwx 1 root root 0 Nov 17 22:44 /sys/class/net/ens4f0/device/virtfn0 -> ../0000:af:02.0
lrwxrwxrwx 1 root root 0 Nov 17 22:44 /sys/class/net/ens4f0/device/virtfn1 -> ../0000:af:02.1