Fortinet black logo

Handbook

Troubleshooting

6.0.0
Copy Link
Copy Doc ID 4afb0436-a998-11e9-81a4-00505692583a:340035
Download PDF

Troubleshooting

This section contains tips to help you with some common challenges of IPsec VPNs.

A VPN connection has multiple stages that can be confirmed to ensure the connection is working properly. It is easiest to see if the final stage is successful first since if it is successful the other stages will be working properly. Otherwise, you will need to work back through the stages to see where the problem is located.

When a VPN connection is properly established, traffic will flow from one end to the other as if both ends were physically in the same place. If you can determine the connection is working properly then any problems are likely problems with your applications.

On some FortiGate units, such as the FortiGate 94D, you cannot ping over the IPsec tunnel without first setting a source-IP. In this scenario, you must assign an IP address to the virtual IPsec VPN interface. Anything sourced from the FortiGate going over the VPN will use this IP address.

If the egress/outgoing interface (determined by kernel route) has an IP address, then use the IP address of the egress/outgoing interface. Otherwise, use the IP address of the first interface from the interface list (that has an IP address).

The first diagnostic command worth running, in any IPsec VPN troubleshooting situation, is the following:

diagnose vpn tunnel list

This command is very useful for gathering statistical data such as the number of packets encrypted versus decrypted, the number of bytes sent versus received, the SPI identifier, etc. This kind of information in the resulting output can make all the difference in determining the issue with the VPN.

Another appropriate diagnostic command worth trying is:

diagnose debug flow

This command will inform you of any lack of firewall policy, lack of forwarding route, and of policy ordering issues.

Common IPsec VPN problems

The most common IPsec VPN issues are listed below. Please read thoroughly and note that, although the list is extensive, it is not exhaustive.

This section includes support for the following:

Failed VPN connection attempts

If your VPN fails to connect, check the following:

If you are still unable to connect to the VPN tunnel, run the following diagnostic command in the CLI:

diagnose debug application ike -1

diagnose debug enable

The resulting output may indicate where the problem is occurring. When you are finished, disable the diagnostics by using the following command:

diagnose debug reset

diagnose debug disable

View the table below for some assistance in analyzing the debug output.

Debug output table

Problem

Debug output

Common causes

Common solutions

Tunnel is not coming up

Error: negotiation failure

IPsec configuration mismatch

Check phase 1 and 2 settings

Error: no SA proposal chosen

IPsec configuration mismatch

Check phase 1 and 2 settings

FortiGate using the wrong VPN

Missing or wrong local ID

If there are more than one pre-shared key dial-up VPN with the same local gateway, use aggressive mode and different local IDs

Error: connection expiring due to XAUTH failure

Wrong username, password, or user group

Check user credentials and user group configuration

Error: peer has not completed XAUTH exchange

XAuth is disabled in the client

Fix the client's XAuth configuration

Tunnel is bouncing

DPD packets lost

ISP issue

Check the ISP connection

Tunnel is up but traffic does not go through

Error: No matching IPsec selector, drop

Quick mode selector mismatch

Fix the quick mode selector

NAT is enabled

Disable NAT in the firewall policy

Traffic is not routed to the tunnel

Route or firewall policy misconfiguration

Route-based: traffic must be routed to IPsec virtual interface
Policy-based: traffic must match a firewall policy with action set to IPSEC

The options to configure policy-based IPsec VPN are unavailable

Go to System > Feature Visibility. Select Show More and turn on Policy-based IPsec VPN.

The VPN tunnel goes down frequently

If your VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value or enable Autokey Keep Alive.

The pre-shared key does not match (PSK mismatch error)

It is possible to identify a PSK mismatch using the following combination of CLI commands:

diag vpn ike log filter name <phase1-name>

diag debug app ike -1

diag debug enable

This will provide you with clues as to any PSK or other proposal issues. If it is a PSK mismatch, you should see something similar to the following output:

ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch

ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch)

The most common problem with IPsec VPN tunnels is a mismatch between the proposals offered between each party. Without a match and proposal agreement, Phase 1 can never establish. Use the following command to show the proposals presented by both parties.

diag debug app ike -1

diag debug enable

Here is an example of the resulting output:

responder received SA_INIT msg

incoming proposal:

This section represents the remote VPN device:

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 256)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

proposal id = 2:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=3DES_CBC

type=INTEGR, val=AUTH_HMAC_SHA_2_256_128

type=PRF, val=PRF_HMAC_SHA2_256

type=DH_GROUP, val=1536.

This section represents the local FortiGate:

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 128)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

Pre-existing IPsec VPN tunnels need to be cleared

Should you need to clear an IKE gateway, use the following commands:

diagnose vpn ike restart

diagnose vpn ike gateway clear

Other potential VPN issues

  • Ensure that your FortiGate unit is in NAT mode, rather than Transparent.
  • Check your NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling NAT in the security policy. You might need to pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.
  • Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels are being used.
  • Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the VPN tunnel appears on the IPsec Monitor, reboot your FortiGate unit to try and clear the entry.
  • If you have multiple dial-up IPsec VPNs, ensure that the peer ID is configured properly on the FortiGate and that clients have specified the correct local ID. Furthermore, in circumstances where multiple remote dialup VPN tunnels exist, each tunnel must have a peer ID set.
  • If you are using FortiClient, ensure that your version is compatible with the FortiGate firmware by reading the FortiOS Release Notes.
  • If you are using Perfect Forward Secrecy (PFS), ensure that it is used on both peers. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently uses firewall addresses or address groups, try changing it to either specify the IP addresses or use an expanded address range. This is especially useful if the remote endpoint is not a FortiGate device.
  • If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate unit is set to Enable as Server.
  • Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to exceed the overhead of the ESP-header, including the additional ip_header,etc. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • If your FortiGate unit is behind a NAT device, such as a router, configure port forwarding for UDP ports 500 and 4500.

Troubleshooting connection issues

The following section includes troubleshooting suggestions related to:

LAN interface connection

To confirm whether a VPN connection over LAN interfaces has been configured correctly, issue a ping or traceroute command on the network behind the FortiGate unit to test the connection to a computer on the remote network. If the connection is properly configured, a VPN tunnel will be established automatically when the first data packet destined for the remote network is intercepted by the FortiGate unit.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel. You can confirm this by going to Monitor > IPsec Monitor where you will be able to see your connection. A green arrow means the tunnel is up and currently processing traffic. A red arrow means the tunnel is not processing traffic, and this VPN connection has a problem.

If the connection has problems, see Troubleshooting VPN connections.

Dialup connection

A dialup VPN connection has additional steps. To confirm that a VPN between a local network and a dialup client has been configured correctly, at the dialup client, issue a ping command to test the connection to the local network. The VPN tunnel initializes when the dialup client attempts to connect.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel, or dialup client. As with the LAN connection, confirm the VPN tunnel is established by checking Monitor > IPsec Monitor.

Troubleshooting VPN connections

If you have determined that your VPN connection is not working properly through Troubleshooting, the next step is to verify that you have a phase2 connection.

If traffic is not passing through the FortiGate unit as you expect, ensure the traffic does not contain IPcomp packets (IP protocol 108, RFC 3173). FortiGate units do not allow IPcomp packets, they compress packet payload, preventing it from being scanned.

Testing Phase 1 and 2 connections is a bit more difficult than testing the working VPN. This is because they require diagnose CLI commands. These commands are typically used by Fortinet customer support to discover more information about your FortiGate unit and its current configuration.

Before you begin troubleshooting, you must:

  • Configure FortiGate units on both ends for interface VPN
  • Record the information in your VPN Phase 1 and Phase 2 configurations - for our example here the remote IP address is 10.11.101.10 and the names of the phases are Phase 1 and Phase 2
  • Install a telnet or SSH client such as putty that allows logging of output
  • Ensure that the admin interface supports your chosen connection protocol so you can connect to your FortiGate unit admin interface.

For this example, default values were used unless stated otherwise.

Obtaining diagnose information for the VPN connection - CLI
  1. Log into the CLI as admin with the output being logged to a file.
  2. Stop any diagnose debug sessions that are currently running with the CLI command
  3. diagnose debug disable

  4. Clear any existing log-filters by running
  5. diagnose vpn ike log-filter clear

  6. Set the log-filter to the IP address of the remote computer (10.11.101.10). This filters out all VPN connections except ones to the IP address we are concerned with. The command is
  7. diagnose vpn ike log-filter dst-addr4 10.11.101.10.

  8. Set up the commands to output the VPN handshaking. The commands are:
  9. diagnose debug app ike 255

    diagnose debug enable

  10. Have the remote FortiGate initiate the VPN connection in the GUI by going to
    VPN > IPsec Tunnels
    and selecting Bring up.

    This makes the remote FortiGate the initiator and the local FortiGate becomes the responder. Establishing the connection in this manner means the local FortiGate will have its configuration information as well as the information the remote computer sends. Having both sets of information locally makes it easier to troubleshoot your VPN connection.
  11. Watch the screen for output, and after roughly 15 seconds enter the following CLI command to stop the output.
  12. diagnose debug disable

  13. If needed, save the log file of this output to a file on your local computer. Saving the output to a file can make it easier to search for a particular phrase, and is useful for comparisons.
Troubleshooting a Phase 1 VPN connection

Using the output from Obtaining diagnose information for the VPN connection - CLI, search for the word proposal in the output. It may occur once indicating a successful connection, or it will occur two or more times for an unsuccessful connection — there will be one proposal listed for each end of the tunnel and each possible combination in their settings. For example if 10.11.101.10 selected both Diffie-Hellman Groups 1 and 5, that would be at least 2 proposals set.

A successful negotiation proposal will look similar to

IPsec SA connect 26 10.12.101.10->10.11.101.10:500

config found

created connection: 0x2f55860 26 10.12.101.10->10.11.101.10:500

IPsec SA connect 26 10.12.101.10->10.11.101.10:500 negotiating

no suitable ISAKMP SA, queuing quick-mode request and initiating ISAKMP SA negotiation

initiator: main mode is sending 1st message...

cookie 3db6afe559e3df0f/0000000000000000

out [encryption]

sent IKE msg (ident-i1send): 10.12.101.10:500->10.11.101.10:500, len=264, id=3db6afe559e3df0f/0000000000000000

diaike 0: comes 10.12.101.1:500->10.11.101.1:500,ifindex=26....

Note the phrase “initiator: main mode is sending 1st message...” which shows you the handshake between the ends of the tunnel is in progress. Initiator shows the remote unit is sending the first message.

Troubleshooting invalid ESP packets using Wireshark

The following section provides information to help debug an encryption key mismatch. The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established. When an IPsec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch. A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any device in the physical connection.

The following information is required to troubleshoot the problem.

  • Take a packet sniffer trace on both FortiGates.
  • Run the diag vpn tunnel list command a few times on both FortiGates when generating traffic that will pass through the tunnel.

In the following example, the error message was seen on the recipient FortiGate:

date=2010-12-28 time=18:19:35 devname=Kosad_VPN device_id=FG300B3910600118 log_id=0101037132 type=event subtype=ipsec pri=critical vd="root" msg="IPsec ESP" action="error" rem_ip=180.87.33.2 loc_ip=121.133.8.18 rem_port=32528 loc_port=4500 out_intf="port2" cookies="88d40f65d555ccaf/05464e20e4afc835" user="N/A" group="N/A" xauth_user="N/A" xauth_group="N/A" vpn_tunnel="fortinet_0" status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=c32b09f7 seq=00000012

This is the output of the command diag vpn tunnel list on the FortiGate:

inet ver=1 serial=2 192.168.1.205:4500->121.133.8.18:4500 lgwy=dyn tun=intf mode=auto bound_if=4 proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0 stat: rxp=41 txp=56 rxb=4920 txb=3360 dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=696 natt: mode=keepalive draft=32 interval=10 remote_port=4500 proxyid=P2_60C_Fortinet proto=0 sa=1 ref=2 auto_negotiate=0 serial=1 src: 0:182.40.101.0/255.255.255.0:0 dst: 0:100.100.100.0/255.255.255.0:0 SA: ref=3 options=0000000d type=00 soft=0 mtu=1428 expire=1106 replaywin=0 seqno=15 life: type=01 bytes=0/0 timeout=1777/1800 dec: spi=29a26eb6 esp=3des key=24 bf25e69df90257f64c55dda4069f01834cd0382fe4866ff2 ah=sha1 key=20 38b2600170585d2dfa646caed5bc86d920aed7ff enc: spi=c32b09f7 esp=3des key=24 0abd3c70032123c3369a6f225a385d30f0b2fb1cd9687ec8 ah=sha1 key=20 214d8e717306dffceec3760464b6e8edb436c6

This is the packet capture from the FortiGate:

How to verify if the original packet has been encrypted correctly

To verify, it is necessary to decrypt the ESP packet using Wireshark. Open the packet capture that is taken from initiator FortiGate using Wireshark. Go to Edit > Preferences, expand Protocol and look for ESP. Select "Attempt to detect/decode encrypted ESP payloads", and fill in the information for the encryption algorithm and the keys. This information can be obtained from the output of the command diag vpn tunnel list.

If the packet was encrypted correctly using the correct key, then the decryption will be successful and it will be possible to see the original package as shown below:

Repeat the decryption process for the packet capture from the recipient firewall. If the decryption failed using the same key, the packet may be corrupted and the interface should then be checked for CRC or packet errors.

Attempting hardware offloading beyond SHA1

If you are trying to off-load VPN processing to a network processing unit (NPU), remember that only SHA1 authentication is supported. For high levels of authentication such as SHA256, SHA384, and SHA512 hardware offloading is not an option—all VPN processing must be done in software—unless using an NP6 (although the NP4lite variation also supports SHA256, SHA384, and SHA512).

Enable/disable IPsec ASIC-offloading

Much like NPU-offload in IKE phase1 configuration, you can enable or disable the usage of ASIC hardware for IPsec Diffie-Hellman key exchange and IPsec ESP traffic. By default hardware offloading is used. For debugging purposes, sometimes it is best for all the traffic to be processed by software.

config sys global

set ipsec-asic-offload [enable | disable]

end

Check Phase 1 proposal settings

Ensure that both sides have at least one Phase 1 proposal in common. Otherwise they will not connect. If there are many proposals in the list, this will slow down the negotiating of Phase 1. If its too slow, the connection may timeout before completing. If this happens, try removing some of the unused proposals.

NPU offloading is not supported when the local gateway is a loopback interface.

Check your routing

If routing is not properly configured with an entry for the remote end of the VPN tunnel, traffic will not flow properly. You may need static routes on both ends of the tunnel. If routing is the problem, the proposal will likely setup properly but no traffic will flow.

Try enabling XAuth

If one end of an attempted VPN tunnel is using XAuth and the other end is not, the connection attempt will fail. The log messages for the attempted connection will not mention XAuth is the reason, but when connections are failing it is a good idea to ensure both ends have the same XAuth settings. If you do not know the other end’s settings enable or disable XAuth on your end to see if that is the problem.

Troubleshooting

This section contains tips to help you with some common challenges of IPsec VPNs.

A VPN connection has multiple stages that can be confirmed to ensure the connection is working properly. It is easiest to see if the final stage is successful first since if it is successful the other stages will be working properly. Otherwise, you will need to work back through the stages to see where the problem is located.

When a VPN connection is properly established, traffic will flow from one end to the other as if both ends were physically in the same place. If you can determine the connection is working properly then any problems are likely problems with your applications.

On some FortiGate units, such as the FortiGate 94D, you cannot ping over the IPsec tunnel without first setting a source-IP. In this scenario, you must assign an IP address to the virtual IPsec VPN interface. Anything sourced from the FortiGate going over the VPN will use this IP address.

If the egress/outgoing interface (determined by kernel route) has an IP address, then use the IP address of the egress/outgoing interface. Otherwise, use the IP address of the first interface from the interface list (that has an IP address).

The first diagnostic command worth running, in any IPsec VPN troubleshooting situation, is the following:

diagnose vpn tunnel list

This command is very useful for gathering statistical data such as the number of packets encrypted versus decrypted, the number of bytes sent versus received, the SPI identifier, etc. This kind of information in the resulting output can make all the difference in determining the issue with the VPN.

Another appropriate diagnostic command worth trying is:

diagnose debug flow

This command will inform you of any lack of firewall policy, lack of forwarding route, and of policy ordering issues.

Common IPsec VPN problems

The most common IPsec VPN issues are listed below. Please read thoroughly and note that, although the list is extensive, it is not exhaustive.

This section includes support for the following:

Failed VPN connection attempts

If your VPN fails to connect, check the following:

If you are still unable to connect to the VPN tunnel, run the following diagnostic command in the CLI:

diagnose debug application ike -1

diagnose debug enable

The resulting output may indicate where the problem is occurring. When you are finished, disable the diagnostics by using the following command:

diagnose debug reset

diagnose debug disable

View the table below for some assistance in analyzing the debug output.

Debug output table

Problem

Debug output

Common causes

Common solutions

Tunnel is not coming up

Error: negotiation failure

IPsec configuration mismatch

Check phase 1 and 2 settings

Error: no SA proposal chosen

IPsec configuration mismatch

Check phase 1 and 2 settings

FortiGate using the wrong VPN

Missing or wrong local ID

If there are more than one pre-shared key dial-up VPN with the same local gateway, use aggressive mode and different local IDs

Error: connection expiring due to XAUTH failure

Wrong username, password, or user group

Check user credentials and user group configuration

Error: peer has not completed XAUTH exchange

XAuth is disabled in the client

Fix the client's XAuth configuration

Tunnel is bouncing

DPD packets lost

ISP issue

Check the ISP connection

Tunnel is up but traffic does not go through

Error: No matching IPsec selector, drop

Quick mode selector mismatch

Fix the quick mode selector

NAT is enabled

Disable NAT in the firewall policy

Traffic is not routed to the tunnel

Route or firewall policy misconfiguration

Route-based: traffic must be routed to IPsec virtual interface
Policy-based: traffic must match a firewall policy with action set to IPSEC

The options to configure policy-based IPsec VPN are unavailable

Go to System > Feature Visibility. Select Show More and turn on Policy-based IPsec VPN.

The VPN tunnel goes down frequently

If your VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value or enable Autokey Keep Alive.

The pre-shared key does not match (PSK mismatch error)

It is possible to identify a PSK mismatch using the following combination of CLI commands:

diag vpn ike log filter name <phase1-name>

diag debug app ike -1

diag debug enable

This will provide you with clues as to any PSK or other proposal issues. If it is a PSK mismatch, you should see something similar to the following output:

ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch

ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch)

The most common problem with IPsec VPN tunnels is a mismatch between the proposals offered between each party. Without a match and proposal agreement, Phase 1 can never establish. Use the following command to show the proposals presented by both parties.

diag debug app ike -1

diag debug enable

Here is an example of the resulting output:

responder received SA_INIT msg

incoming proposal:

This section represents the remote VPN device:

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 256)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

proposal id = 2:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=3DES_CBC

type=INTEGR, val=AUTH_HMAC_SHA_2_256_128

type=PRF, val=PRF_HMAC_SHA2_256

type=DH_GROUP, val=1536.

This section represents the local FortiGate:

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 128)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

Pre-existing IPsec VPN tunnels need to be cleared

Should you need to clear an IKE gateway, use the following commands:

diagnose vpn ike restart

diagnose vpn ike gateway clear

Other potential VPN issues

  • Ensure that your FortiGate unit is in NAT mode, rather than Transparent.
  • Check your NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling NAT in the security policy. You might need to pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.
  • Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels are being used.
  • Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the VPN tunnel appears on the IPsec Monitor, reboot your FortiGate unit to try and clear the entry.
  • If you have multiple dial-up IPsec VPNs, ensure that the peer ID is configured properly on the FortiGate and that clients have specified the correct local ID. Furthermore, in circumstances where multiple remote dialup VPN tunnels exist, each tunnel must have a peer ID set.
  • If you are using FortiClient, ensure that your version is compatible with the FortiGate firmware by reading the FortiOS Release Notes.
  • If you are using Perfect Forward Secrecy (PFS), ensure that it is used on both peers. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently uses firewall addresses or address groups, try changing it to either specify the IP addresses or use an expanded address range. This is especially useful if the remote endpoint is not a FortiGate device.
  • If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate unit is set to Enable as Server.
  • Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to exceed the overhead of the ESP-header, including the additional ip_header,etc. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • If your FortiGate unit is behind a NAT device, such as a router, configure port forwarding for UDP ports 500 and 4500.

Troubleshooting connection issues

The following section includes troubleshooting suggestions related to:

LAN interface connection

To confirm whether a VPN connection over LAN interfaces has been configured correctly, issue a ping or traceroute command on the network behind the FortiGate unit to test the connection to a computer on the remote network. If the connection is properly configured, a VPN tunnel will be established automatically when the first data packet destined for the remote network is intercepted by the FortiGate unit.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel. You can confirm this by going to Monitor > IPsec Monitor where you will be able to see your connection. A green arrow means the tunnel is up and currently processing traffic. A red arrow means the tunnel is not processing traffic, and this VPN connection has a problem.

If the connection has problems, see Troubleshooting VPN connections.

Dialup connection

A dialup VPN connection has additional steps. To confirm that a VPN between a local network and a dialup client has been configured correctly, at the dialup client, issue a ping command to test the connection to the local network. The VPN tunnel initializes when the dialup client attempts to connect.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel, or dialup client. As with the LAN connection, confirm the VPN tunnel is established by checking Monitor > IPsec Monitor.

Troubleshooting VPN connections

If you have determined that your VPN connection is not working properly through Troubleshooting, the next step is to verify that you have a phase2 connection.

If traffic is not passing through the FortiGate unit as you expect, ensure the traffic does not contain IPcomp packets (IP protocol 108, RFC 3173). FortiGate units do not allow IPcomp packets, they compress packet payload, preventing it from being scanned.

Testing Phase 1 and 2 connections is a bit more difficult than testing the working VPN. This is because they require diagnose CLI commands. These commands are typically used by Fortinet customer support to discover more information about your FortiGate unit and its current configuration.

Before you begin troubleshooting, you must:

  • Configure FortiGate units on both ends for interface VPN
  • Record the information in your VPN Phase 1 and Phase 2 configurations - for our example here the remote IP address is 10.11.101.10 and the names of the phases are Phase 1 and Phase 2
  • Install a telnet or SSH client such as putty that allows logging of output
  • Ensure that the admin interface supports your chosen connection protocol so you can connect to your FortiGate unit admin interface.

For this example, default values were used unless stated otherwise.

Obtaining diagnose information for the VPN connection - CLI
  1. Log into the CLI as admin with the output being logged to a file.
  2. Stop any diagnose debug sessions that are currently running with the CLI command
  3. diagnose debug disable

  4. Clear any existing log-filters by running
  5. diagnose vpn ike log-filter clear

  6. Set the log-filter to the IP address of the remote computer (10.11.101.10). This filters out all VPN connections except ones to the IP address we are concerned with. The command is
  7. diagnose vpn ike log-filter dst-addr4 10.11.101.10.

  8. Set up the commands to output the VPN handshaking. The commands are:
  9. diagnose debug app ike 255

    diagnose debug enable

  10. Have the remote FortiGate initiate the VPN connection in the GUI by going to
    VPN > IPsec Tunnels
    and selecting Bring up.

    This makes the remote FortiGate the initiator and the local FortiGate becomes the responder. Establishing the connection in this manner means the local FortiGate will have its configuration information as well as the information the remote computer sends. Having both sets of information locally makes it easier to troubleshoot your VPN connection.
  11. Watch the screen for output, and after roughly 15 seconds enter the following CLI command to stop the output.
  12. diagnose debug disable

  13. If needed, save the log file of this output to a file on your local computer. Saving the output to a file can make it easier to search for a particular phrase, and is useful for comparisons.
Troubleshooting a Phase 1 VPN connection

Using the output from Obtaining diagnose information for the VPN connection - CLI, search for the word proposal in the output. It may occur once indicating a successful connection, or it will occur two or more times for an unsuccessful connection — there will be one proposal listed for each end of the tunnel and each possible combination in their settings. For example if 10.11.101.10 selected both Diffie-Hellman Groups 1 and 5, that would be at least 2 proposals set.

A successful negotiation proposal will look similar to

IPsec SA connect 26 10.12.101.10->10.11.101.10:500

config found

created connection: 0x2f55860 26 10.12.101.10->10.11.101.10:500

IPsec SA connect 26 10.12.101.10->10.11.101.10:500 negotiating

no suitable ISAKMP SA, queuing quick-mode request and initiating ISAKMP SA negotiation

initiator: main mode is sending 1st message...

cookie 3db6afe559e3df0f/0000000000000000

out [encryption]

sent IKE msg (ident-i1send): 10.12.101.10:500->10.11.101.10:500, len=264, id=3db6afe559e3df0f/0000000000000000

diaike 0: comes 10.12.101.1:500->10.11.101.1:500,ifindex=26....

Note the phrase “initiator: main mode is sending 1st message...” which shows you the handshake between the ends of the tunnel is in progress. Initiator shows the remote unit is sending the first message.

Troubleshooting invalid ESP packets using Wireshark

The following section provides information to help debug an encryption key mismatch. The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established. When an IPsec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch. A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any device in the physical connection.

The following information is required to troubleshoot the problem.

  • Take a packet sniffer trace on both FortiGates.
  • Run the diag vpn tunnel list command a few times on both FortiGates when generating traffic that will pass through the tunnel.

In the following example, the error message was seen on the recipient FortiGate:

date=2010-12-28 time=18:19:35 devname=Kosad_VPN device_id=FG300B3910600118 log_id=0101037132 type=event subtype=ipsec pri=critical vd="root" msg="IPsec ESP" action="error" rem_ip=180.87.33.2 loc_ip=121.133.8.18 rem_port=32528 loc_port=4500 out_intf="port2" cookies="88d40f65d555ccaf/05464e20e4afc835" user="N/A" group="N/A" xauth_user="N/A" xauth_group="N/A" vpn_tunnel="fortinet_0" status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=c32b09f7 seq=00000012

This is the output of the command diag vpn tunnel list on the FortiGate:

inet ver=1 serial=2 192.168.1.205:4500->121.133.8.18:4500 lgwy=dyn tun=intf mode=auto bound_if=4 proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0 stat: rxp=41 txp=56 rxb=4920 txb=3360 dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=696 natt: mode=keepalive draft=32 interval=10 remote_port=4500 proxyid=P2_60C_Fortinet proto=0 sa=1 ref=2 auto_negotiate=0 serial=1 src: 0:182.40.101.0/255.255.255.0:0 dst: 0:100.100.100.0/255.255.255.0:0 SA: ref=3 options=0000000d type=00 soft=0 mtu=1428 expire=1106 replaywin=0 seqno=15 life: type=01 bytes=0/0 timeout=1777/1800 dec: spi=29a26eb6 esp=3des key=24 bf25e69df90257f64c55dda4069f01834cd0382fe4866ff2 ah=sha1 key=20 38b2600170585d2dfa646caed5bc86d920aed7ff enc: spi=c32b09f7 esp=3des key=24 0abd3c70032123c3369a6f225a385d30f0b2fb1cd9687ec8 ah=sha1 key=20 214d8e717306dffceec3760464b6e8edb436c6

This is the packet capture from the FortiGate:

How to verify if the original packet has been encrypted correctly

To verify, it is necessary to decrypt the ESP packet using Wireshark. Open the packet capture that is taken from initiator FortiGate using Wireshark. Go to Edit > Preferences, expand Protocol and look for ESP. Select "Attempt to detect/decode encrypted ESP payloads", and fill in the information for the encryption algorithm and the keys. This information can be obtained from the output of the command diag vpn tunnel list.

If the packet was encrypted correctly using the correct key, then the decryption will be successful and it will be possible to see the original package as shown below:

Repeat the decryption process for the packet capture from the recipient firewall. If the decryption failed using the same key, the packet may be corrupted and the interface should then be checked for CRC or packet errors.

Attempting hardware offloading beyond SHA1

If you are trying to off-load VPN processing to a network processing unit (NPU), remember that only SHA1 authentication is supported. For high levels of authentication such as SHA256, SHA384, and SHA512 hardware offloading is not an option—all VPN processing must be done in software—unless using an NP6 (although the NP4lite variation also supports SHA256, SHA384, and SHA512).

Enable/disable IPsec ASIC-offloading

Much like NPU-offload in IKE phase1 configuration, you can enable or disable the usage of ASIC hardware for IPsec Diffie-Hellman key exchange and IPsec ESP traffic. By default hardware offloading is used. For debugging purposes, sometimes it is best for all the traffic to be processed by software.

config sys global

set ipsec-asic-offload [enable | disable]

end

Check Phase 1 proposal settings

Ensure that both sides have at least one Phase 1 proposal in common. Otherwise they will not connect. If there are many proposals in the list, this will slow down the negotiating of Phase 1. If its too slow, the connection may timeout before completing. If this happens, try removing some of the unused proposals.

NPU offloading is not supported when the local gateway is a loopback interface.

Check your routing

If routing is not properly configured with an entry for the remote end of the VPN tunnel, traffic will not flow properly. You may need static routes on both ends of the tunnel. If routing is the problem, the proposal will likely setup properly but no traffic will flow.

Try enabling XAuth

If one end of an attempted VPN tunnel is using XAuth and the other end is not, the connection attempt will fail. The log messages for the attempted connection will not mention XAuth is the reason, but when connections are failing it is a good idea to ensure both ends have the same XAuth settings. If you do not know the other end’s settings enable or disable XAuth on your end to see if that is the problem.