Deploying the FortiGate-VM on KVM
Deploying a FortiGate-VM on KVM with QAT support consists of the following steps:
- Create the FortiGate-VM on KVM.
- Inject SR-IOV network VFs into the FortiGate-VM.
- Configure interrupt affinities.
To create the FortiGate-VM on KVM:
To create a FortiGate-VM on KVM, refer to Deploying the FortiGate-VM.
To inject SR-IOV network VFs into the FortiGate-VM:
You can inject an SR-IOV network VF into a Linux KVM VM using one of the following ways:
- Connecting an SR-IOV VF to a KVM VM by directly importing the VF as a PCI device using the PCI bus information that the host OS assigned to it when it was created
- Using the Virtual Manager GUI
- Adding an SR-IOV network adapter to the KVM VM as a VF network adapter connected to a macvtap on the host
- Creating an SR-IOV VF network adapter using a KVM virtual network pool of adapters
See Configure SR-IOV Network Virtual Functions in Linux* KVM*.
In the following example, virtual network adapter pools were created for KVM04:
[root@localhost ~]# vnlist Name State Autostart Persistent ---------------------------------------------------------- default active yes yes p5p1-pool active no no p5p2-pool active no no p7p1-pool active no no p7p2-pool active no no [root@localhost ~]# vf2pf Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p1): PCI BDF Interface ======= ========= 0000:86:02.0 p5p1_0 0000:86:02.1 p5p1_1 0000:86:02.2 p5p1_2 0000:86:02.3 p5p1_3 0000:86:02.4 p5p1_4 0000:86:02.5 p5p1_5 0000:86:02.6 p5p1_6 0000:86:02.7 p5p1_7 Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p2): PCI BDF Interface ======= ========= 0000:86:0a.0 p5p2_0 0000:86:0a.1 p5p2_1 0000:86:0a.2 p5p2_2 0000:86:0a.3 p5p2_3 0000:86:0a.4 p5p2_4 0000:86:0a.5 p5p2_5 0000:86:0a.6 p5p2_6 0000:86:0a.7 p5p2_7 Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p1): PCI BDF Interface ======= ========= 0000:88:02.0 p7p1_0 0000:88:02.1 p7p1_1 0000:88:02.2 p7p1_2 0000:88:02.3 p7p1_3 0000:88:02.4 p7p1_4 0000:88:02.5 p7p1_5 0000:88:02.6 p7p1_6 0000:88:02.7 p7p1_7 Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p2): PCI BDF Interface ======= ========= 0000:88:0a.0 p7p2_0 0000:88:0a.1 p7p2_1 0000:88:0a.2 p7p2_2 0000:88:0a.3 p7p2_3 0000:88:0a.4 p7p2_4 0000:88:0a.5 p7p2_5 0000:88:0a.6 p7p2_6 0000:88:0a.7 p7p2_7
The XML file is as follows. <cputune>
locks the virtual CPUs to the same NUMA node, while <hostdev mode='subsystem' type='pci' managed='yes'>
creates the QAT VFs:
[root@localhost ~]# virsh dumpxml vm04numa1 <domain type='kvm'> <name>vm04numa1</name> <uuid>fc5e1cec-8b4e-4bb8-9f89-e86f1abfffeb</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='17'/> <vcpupin vcpu='1' cpuset='19'/> <vcpupin vcpu='2' cpuset='21'/> <vcpupin vcpu='3' cpuset='23'/> <emulatorpin cpuset='17,19,21,23'/> </cputune> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>Skylake-Server-IBRS</model> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm04numa1.0984'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='direct'> <mac address='52:54:00:7c:07:50'/> <source dev='em1' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:7c:07:53'/> <source network='p5p1-pool'/> <model type='i40e'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:7c:07:54'/> <source network='p5p2-pool'/> <model type='i40e'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:7c:07:55'/> <source network='p7p1-pool'/> <model type='i40e'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:7c:07:56'/> <source network='p7p2-pool'/> <model type='i40e'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </interface> <serial type='tcp'> <source mode='bind' host='0.0.0.0' service='10004'/> <protocol type='telnet'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='tcp'> <source mode='bind' host='0.0.0.0' service='10004'/> <protocol type='telnet'/> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5904' autoport='no' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb1' slot='0x01' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb1' slot='0x01' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb1' slot='0x01' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb1' slot='0x01' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>
The following shows the Virtual Manager GUI:
To configure interrupt affinities:
The example topology is as follows:
TestCenter----KVM port2----KVM port3===IPSEC tunnel===KVM port5-----KVM port4----TestCenter
After configuring the IPsec tunnel in the KVM, you must configure interrupt affinities and CPU masking to improve throughput for FortiGate-VM platforms. In this example, you can configure interrupt affinities as follows. You must manually set cpu-affinity
for each QAT VF. Otherwise, they only go through the first CPU.
config system affinity-interrupt edit 1 set interrupt "i40evf-port2-TxRx-0" set affinity-cpumask "0x01" next edit 2 set interrupt "i40evf-port2-TxRx-1" set affinity-cpumask "0x01" next edit 3 set interrupt "i40evf-port2-TxRx-2" set affinity-cpumask "0x01" next edit 4 set interrupt "i40evf-port2-TxRx-3" set affinity-cpumask "0x01" next edit 5 set interrupt "i40evf-port3-TxRx-0" set affinity-cpumask "0x02" next edit 6 set interrupt "i40evf-port3-TxRx-1" set affinity-cpumask "0x02" next edit 7 set interrupt "i40evf-port3-TxRx-2" set affinity-cpumask "0x02" next edit 8 set interrupt "i40evf-port3-TxRx-3" set affinity-cpumask "0x02" next edit 9 set interrupt "i40evf-port4-TxRx-0" set affinity-cpumask "0x04" next edit 10 set interrupt "i40evf-port4-TxRx-1" set affinity-cpumask "0x04" next edit 11 set interrupt "i40evf-port4-TxRx-2" set affinity-cpumask "0x04" next edit 12 set interrupt "i40evf-port4-TxRx-3" set affinity-cpumask "0x04" next edit 13 set interrupt "i40evf-port5-TxRx-0" set affinity-cpumask "0x08" next edit 14 set interrupt "i40evf-port5-TxRx-1" set affinity-cpumask "0x08" next edit 15 set interrupt "i40evf-port5-TxRx-2" set affinity-cpumask "0x08" next edit 16 set interrupt "i40evf-port5-TxRx-3" set affinity-cpumask "0x08" next edit 17 set interrupt "qat_00:14.00" set affinity-cpumask "0x01" next edit 18 set interrupt "qat_00:15.00" set affinity-cpumask "0x02" next edit 19 set interrupt "qat_00:16.00" set affinity-cpumask "0x04" next edit 20 set interrupt "qat_00:17.00" set affinity-cpumask "0x08" next end
This way, all four CPUs are balanced:
FGVM04TM19001384 (global) # get system performance status
CPU states: 0% user 2% system 0% nice 63% idle 0% iowait 0% irq 35% softirq
CPU0 states: 0% user 1% system 0% nice 69% idle 0% iowait 0% irq 30% softirq
CPU1 states: 0% user 3% system 0% nice 55% idle 0% iowait 0% irq 42% softirq
CPU2 states: 0% user 1% system 0% nice 72% idle 0% iowait 0% irq 27% softirq
CPU3 states: 0% user 2% system 0% nice 59% idle 0% iowait 0% irq 39% softirq
Memory: 6131096k total, 1092248k used (17.8%), 4908480k free (80.1%), 130368k freeable (2.1%)
Average network usage: 3825681 / 3813407 kbps in 1 minute, 1392107 / 1389299 kbps in 10 minutes, 632093 / 631744 kbps in 30 minutes
Average sessions: 37 sessions in 1 minute, 31 sessions in 10 minutes, 24 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 0 hours, 7 minutes
For how CPU interrupt affinity optimizes FortiGate-VM performance, see Technical Note: Optimize FortiGate-VM performance by configuring CPU interrupt affinity.
FGVM04TM19001384 (global) # diagnose hardware sysinfo interrupts CPU0 CPU1 CPU2 CPU3 0: 26 0 0 0 IO-APIC-edge timer 1: 9 0 0 0 IO-APIC-edge i8042 4: 15 0 0 0 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 10: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3, uhci_hcd:usb4 11: 16 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb2 12: 3 0 0 0 IO-APIC-edge i8042 14: 0 0 0 0 IO-APIC-edge ata_piix 15: 0 0 0 0 IO-APIC-edge ata_piix 40: 0 0 0 0 PCI-MSI-edge virtio1-config 41: 629 0 0 0 PCI-MSI-edge virtio1-requests 42: 0 0 0 0 PCI-MSI-edge virtio3-config 43: 978 0 0 0 PCI-MSI-edge virtio3-input.0 44: 1 0 0 0 PCI-MSI-edge virtio3-output.0 45: 255083 0 0 0 PCI-MSI-edge qat_00:14.00 46: 17 537891 0 0 PCI-MSI-edge qat_00:15.00 47: 17 0 1244511 0 PCI-MSI-edge qat_00:16.00 48: 17 0 0 1224563 PCI-MSI-edge qat_00:17.00 49: 173 0 0 0 PCI-MSI-edge i40evf-0000:00:0a.0:mbx 50: 119912 0 0 0 PCI-MSI-edge i40evf-port2-TxRx-0 51: 1 200309 0 0 PCI-MSI-edge i40evf-port2-TxRx-1 52: 1 0 538905 0 PCI-MSI-edge i40evf-port2-TxRx-2 53: 1 0 0 532128 PCI-MSI-edge i40evf-port2-TxRx-3 54: 172 0 0 0 PCI-MSI-edge i40evf-0000:00:0b.0:mbx 55: 254849 0 0 0 PCI-MSI-edge i40evf-port3-TxRx-0 56: 1 443186 0 0 PCI-MSI-edge i40evf-port3-TxRx-1 57: 1 0 600793 0 PCI-MSI-edge i40evf-port3-TxRx-2 58: 1 0 0 850484 PCI-MSI-edge i40evf-port3-TxRx-3 59: 172 0 0 0 PCI-MSI-edge i40evf-0000:00:0c.0:mbx 60: 72971 0 0 0 PCI-MSI-edge i40evf-port4-TxRx-0 61: 1 376044 0 0 PCI-MSI-edge i40evf-port4-TxRx-1 62: 1 0 531843 0 PCI-MSI-edge i40evf-port4-TxRx-2 63: 1 0 0 539088 PCI-MSI-edge i40evf-port4-TxRx-3 64: 172 0 0 0 PCI-MSI-edge i40evf-0000:00:0d.0:mbx 65: 197132 0 0 0 PCI-MSI-edge i40evf-port5-TxRx-0 66: 1 421851 0 0 PCI-MSI-edge i40evf-port5-TxRx-1 67: 1 0 850741 0 PCI-MSI-edge i40evf-port5-TxRx-2 68: 1 0 0 600896 PCI-MSI-edge i40evf-port5-TxRx-3 NMI: 0 0 0 0 Non-maskable interrupts LOC: 41936 46038 41842 46773 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RES: 282399 1450 787 751 Rescheduling interrupts CAL: 46 106 62 107 Function call interrupts TLB: 10 11 5 8 TLB shootdowns FGVM04TM19001384 (vdom-1) # diagnose vpn ipsec status All ipsec crypto devices in use: QAT: Encryption (encrypted/decrypted) null : 0 0 des : 0 0 3des : 0 0 aes : 48025403 29252461 aes-gcm : 0 0 aria : 0 0 seed : 0 0 chacha20poly1305 : 0 0 Integrity (generated/validated) null : 0 0 md5 : 0 0 sha1 : 47967645 29250506 sha256 : 0 0 sha384 : 0 0 sha512 : 0 0 SOFTWARE: Encryption (encrypted/decrypted) null : 0 0 des : 0 0 3des : 0 0 aes : 0 0 aes-gcm : 0 0 aria : 0 0 seed : 0 0 chacha20poly1305 : 0 0 Integrity (generated/validated) null : 0 0 md5 : 0 0 sha1 : 0 0 sha256 : 0 0 sha384 : 0 0 sha512 : 0 0 Test Results: 1360 bytes IPSEC packet loss results with QAT in KVM04: 10,654m(v6.2 build 0984)