Fortinet black logo

Handbook

Troubleshooting tips

6.0.0
Copy Link
Copy Doc ID 4afb0436-a998-11e9-81a4-00505692583a:517124
Download PDF

Troubleshooting tips

The following tips present common causes of problems.

How to check hardware connections

  • Are all of the cables and interfaces connected properly?
  • Is the LED for the interface green?

How to check FortiOS network settings

  • If you're having problems connecting to the management interface, is your protocol enabled on the interface for administrative access?
  • Does the interface have an IP address?

How to check CPU and memory resources

  • Is the CPU running at almost 100 percent usage?
  • Is your FortiGate running low on memory?

How to check modem status

  • Is the modem connected?
  • Are there PPP issues?

How to run ping and traceroute

  • Is the FortiGate experiencing complete packet loss?

How to check the logs

  • Do you need to identify a problem?

How to verify the contents of the routing table (in NAT mode)

  • Are there routes in the routing table for default and static routes?
  • Do all connected subnets have a route in the routing table?
  • Does a route have a higher priority than it should?

How to verify the correct route is being used

  • Is the traffic routed correctly?

How to verify the correct firewall policy is being used

  • Is the correct firewall policy applied to the expected traffic?

How to check the bridging information in transparent mode

  • Are you having problems in transparent mode?

How to examine the firewall session list

  • Are there active firewall sessions?

How to check wireless information

  • Is the wireless network working properly?

How to verify FortiGuard connectivity

  • Is the FortiGate communicating properly with FortiGuard?

How to perform a sniffer trace (CLI and packet capture)

  • Is traffic entering the FortiGate? Does the traffic arrive on the expected interface?
  • Is the ARP resolution correct for the next-hop destination?
  • Is the traffic exiting the FortiGate to the destination as expected?
  • Is the FortiGate sending traffic back to the originator?

How to debug the packet flow

  • Is traffic entering or leaving the FortiGate as expected?

How to check hardware connections

If there's no traffic flowing from the FortiGate, you may have a hardware problem.

To check hardware connections:

  1. Ensure network cables are plugged into the interfaces.
  2. Verify that the LED connection lights for the network cables are the right color (usually green).
  3. If the cable or its connector are damaged, change the cable. You should also change the cable if you're not sure about the type or quality of the cable, such as straight through or crossover, of if you see exposed wires at the connector.
  4. Connect the FortiGate to different hardware.
  5. Ensure the link status is set to Up for the interface (see Network > Interfaces). The link status is based on the physical connection and can't be set in FortiOS.

If any of these solve the problem, it was a hardware connection problem. You should still perform some basic software connectivity tests to ensure complete connectivity. The interface might also be disabled, or its Administrative Status might be set to Down.

To enable an interface - GUI
  1. Go to Network > Interfaces.
  2. Select and edit the interface to enable, such as port1.
  3. Find Administrative Status at the bottom of the screen, and select Up.
  4. Select Apply.
To enable an interface - CLI

config system interface

edit port1

set status up

next

end

How to check FortiOS network settings

You can manage FortiOS network settings in the GUI and the CLI. The following information includes troubleshooting and best practice information. The network settings include:

Interface settings

If you can access the FortiGate with the management cable only, the first step is to display the interface settings. To display the settings for the internal interface, use the following CLI command:

FGT# show system interface <interface_name>

For a complete listing of all the possible interface settings, use the following CLI command:

config system interface

edit <interface_name>

get

end

Check the interface settings to ensure that they aren't preventing traffic. Check the following items (only the GUI terms are shown, CLI terms may vary):

Setting

Description

Link Status

Down until a valid cable is plugged into this interface, then it will be Up.

The Link Status is shown physically by the connection LED for the interface. If the LED is green, the connection is good. If Link Status is Down, the interface doesn't work. Link Status is also displayed on the Network > Interfaces screen, by default.

Addressing mode

Don't use DHCP if you don’t have a DHCP server. You won't be able to log in to an interface in DHCP mode as it won't have an IP address.

IP/Network Mask

An interface needs an IP address to be able to connect to other devices. Ensure there is a valid IP address in this field. The one exception is if DHCP is enabled for this interface to get its IP address from an external DHCP server.

IPv6 address

The same protocol must be used by both ends to complete the connection. Ensure both this interface and the remote connection are both using IPv4 or both using IPv6 addressing.

Administrative access

If no protocols are selected, you will have to use the local management cable to connect to the unit. If you're using IPv6, configure the IPv6 administrative access protocols.

Administrative status

Set to Up or the interface won't work.

DNS settings

You can trace many networking problems back to DNS issues. Check the following items:

  1. Are there values for both primary and secondary entries?
  2. Is the local domain name correct?
  3. Are you using IPv6 addressing? If so, are the IPv6 DNS settings correct?
  4. Are you using Dynamic DNS (DDNS)? If so, is it using the correct server, credentials, and interface?
  5. Can you contact both DNS servers to verify the servers are operational?
  6. If an interface addressing mode is set to DHCP and is set to override the internal DNS, is that interface receiving a valid DNS entry from the DHCP server? Is it a reasonable address and can it be contacted to verify it’s operational?
  7. Are there any DENY security policies that need to allow DNS?
  8. Can any internal device perform a successful traceroute to a location using the FQDN?

DHCP server settings

DHCP servers are common on internal and wireless networks. If the DHCP server isn't configured correctly, it can cause problems. Check the following items:

  1. Is the DHCP server entry set to Relay? If so, verify there is another DHCP server to which requests can be relayed. Otherwise, it should be set to Server.
  2. Is the DHCP server enabled?
  3. Does the DHCP server use a valid IP address range? Are other devices using the addresses? If one or more devices are using IP addresses in this range, you can use the IP reservation feature to ensure the DHCP server doesn't use these addresses.
  4. Is there a gateway entry? If not, add a gateway entry to ensure that the server's clients have a default route.
  5. Is the system DNS setting being used? A best practice is to avoid confusion by using the system DNS whenever possible. However, you can specify up to three custom DNS servers, and you should use all three entries for redundancy.

caution icon

There are some situations, such as a new wireless interface, or during the initial FortiGate configuration, where interfaces override the system DNS entries. When this happens, it often shows up as intermittent Internet connectivity. To fix the problem, go to Network > DNS and enable Use FortiGuard Servers.

How to check CPU and memory resources

System resources are shared and a number of processes run simultaneously on the FortiGate unit.

The Resource widgets for CPU and Memory on the Dashboard provide a quick way to monitor usage.

To use the CLI to check the system resources on your FortiGate unit, run the following command:

FGT# get system performance status

This command provides a quick and easy snapshot of the FortiGate.

The first line of output shows the CPU usage by category. A FortiGate that is doing nothing looks like the following example:

CPU states: 0% user 0% system 0% nice 100% idle

However, if your network is running slow you might see something looks like the following example:

CPU states: 1% user 98% system 0% nice 1% idle

This line shows that all the CPU is used up by system processes. Normally this shouldn't happen since it shows that the FortiGate is overloaded for some reason. If you see this overloading, you should investigate further because it’s possible a process, such as scanunitid, is using all the resources to scan traffic, in which case you need to reduce the amount of traffic that's being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions. It's also possible that a hacker has gained access to your network and is overloading it with malicious activity, such as running a spam server or using zombie PCs to attack other networks on the Internet. You can use the get system performance top CLI command to get more information about the CPU. This command shows all of the top processes that are running on the FortiGate (the names are on the left) and their CPU usage. If a process is using most of the CPU cycles, investigate it to determine whether the activity is normal.

The second line of output from the get system performance status command shows the memory usage. Memory usage shouldn't exceed 90%. If memory is too full, some processes won't be able to function properly. For example, if the system is running low on memory, antivirus scanning enters into failopen mode where it drops connections or bypasses the antivirus system.

The other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked, can also help you determine why system resource usage is high. For example, if network usage is high it results in high traffic processing on the FortiGate, or if the session setup rate is very low (or zero) the proxy may be overloaded and unable to do its job.

How to troubleshoot high memory usage

As with any system, a FortiGate has limited hardware resources, such as memory, and all processes running on the FortiGate share the memory. Each process uses more or less memory, depending on its workload. For example, a process usually uses more memory in high traffic situations. If some processes use all of the available memory, other processes won't be able to run.

When high memory usage occurs, the services may freeze up, connections may be lost, or new connections may be refused.

If you see high memory usage in the Memory widget, the FortiGate may be handling high traffic volumes. Alternatively, the FortiGate may have with connection pool limits that are affecting a single proxy. If the FortiGate receives large volumes of traffic on a specific proxy, the unit may exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, issues may occur.

Use the following CLI command, which shows information about current memory usage:

diagnose hardware sysinfo memory

Sample output:

total: used: free: shared: buffers: cached: shm:

Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528

Swap: 0 0 0

MemTotal: 2025572 kB

MemFree: 1286376 kB

MemShared: 0 kB

Buffers: 20216 kB

Cached: 189996 kB

SwapCached: 0 kB

Active: 56644 kB

Inactive: 153648 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 2025572 kB

LowFree: 1286376 kB

SwapTotal: 0 kB

SwapFree: 0 kB

How to troubleshoot high CPU usage

If you deploy too many FortiOS features at the same time, you can overextend the CPU resources on a FortiGate. When this occurs, the FortiGate undergoes connection-related problems.

Some examples of CPU intensive features are:

  • VPN high-level encryption
  • Intensive scanning of all traffic
  • Logging all traffic and packets
  • Dashboard widgets that frequently perform data updates

See CPU and memory thresholds for information on customizing the CPU use threshold.

Determine the current level of CPU usage

There are two ways to do this. The simplest is to look at the CPU widget on the Dashboard in the FortiGate GUI. The real-time CPU usage is displayed for different timeframes. The second method provides precise usage values both for overall usage and for specific processes. To use it, run the diagnose sys top CLI command.

Run Time: 86 days, 0 hours and 10 minutes

0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 3040T, 2437F

bcm.user 93 S < 3.1 0.4

httpsd 18922 S 1.5 0.5

httpsd 19150 S 0.3 0.5

newcli 20195 R 0.1 0.1

cmdbsvr 115 S 0.0 0.8

pyfcgid 20107 S 0.0 0.6

forticron 146 S 0.0 0.5

httpsd 139 S 0.0 0.5

cw_acd 166 S 0.0 0.5

miglogd 136 S 0.0 0.5

pyfcgid 20110 S 0.0 0.4

pyfcgid 20111 S 0.0 0.4

pyfcgid 20109 S 0.0 0.4

httpsd 20192 S 0.0 0.4

miglogd 174 S 0.0 0.4

miglogd 175 S 0.0 0.4

fgfmd 165 S 0.0 0.3

newcli 20191 S 0.0 0.3

initXXXXXXXXXXX 1 S 0.0 0.3

httpsd 184 s 0.0 0.3

The codes displayed on the second output line are explained in the following table:

Code

Description

U

the percentage of user space applications that are currently using the CPU

N

the percentage of time that the CPU spent on low priority processes since the last shutdown

S

the percentage of system processes (or kernel processes) that are using the CPU

I

the percentage of idle CPU resources

WA

the percentage of time that the CPU spent waiting on IO peripherals since the last shutdown

HI

the percentage of time that the CPU spent handling hardware interrupt routines since the last shutdown

SI

the percentage of time that the CPU spent handling software interrupt routines since the last shutdown

ST

or steal time, is the percentage of time that a virtual CPU waits for the physical CPU when the hypervisor is servicing another virtual processor

T

is the total FortiOS system memory, in MB

F

is free memory, in MB

Each additional line of the command output displays information for specific processes running on the FortiGate unit. For example, the third line of the output is: newcli 20195 R 0.1 0.1

The items in the third line of output are explained in the following table:

Item

Description

newcli

The process name.

Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd. Duplicate process names indicate that separate instances of that process that are running.

20195

The process ID, which can be any number.

R

Current state of the process. The process state can be:
  • R - running
  • S - sleep
  • Z - zombie
  • D - disk sleep

0.1

The percentage of CPU capacity that the process is using.

CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that's taking a lot of CPU time

0.1

The amount of memory that the process is using.

Memory usage can range from 0.1 to 5.5 and higher.

When diagnose sys top is running, enter the following single-key commands:

  • q to quit and return to the normal CLI prompt.
  • p to sort the processes by the amount of CPU that the processes are using.
  • m to sort the processes by the amount of memory that the processes are using.

The processes listed are the top processes that are running not all of the processes. For example, if there are 20 processes listed, they are the top 20 processes, sorted by either CPU or memory usage. You can configure the number of processes displayed, using the following CLI command:

diagnose sys top <integer_seconds> <integer_maximum_lines>

Where:

  • <integer_seconds> is the delay in seconds (default is 5)
  • <integer_maximum_lines> is the maximum number of lines (or processes) to list (default is 20)

Determine what features are using most of the CPU resources

There is a command in the CLI that shows the top few processes that are currently running and use the most CPU resources. The get system performance topcommand shows a table of information. The second column from the right shows CPU usage by percentage. If the top few entries are using most of the CPU, note which processes they are and try to reduce their CPU load.

Some examples of processes you'll see include:

  • ipsengine — the IPS engine that scans traffic for intrusions
  • scanunitd — antivirus scanner
  • httpsd — secure HTTP
  • iked — internet key exchange (IKE) in use with IPsec VPN tunnels
  • newcli — active whenever you're accessing the CLI
  • sshd — there are active secure socket connections
  • cmdbsrv — the command database server application

Go to the features that are at the top of the list and look for evidence of them overusing the CPU. Generally, the monitor for a feature is a good place to start.

Check for unnecessary CPU “wasters”

These are some best practices that will reduce your CPU usage, even if the FortiGate isn't experiencing high CPU usage. Note that if the following information instructs you to turn off a feature that you require, disregard that part of the instructions.

  • Use hardware acceleration wherever possible to offload tasks from the CPU. Offloading tasks, such as encryption, frees up the CPU for other tasks.
  • Avoid the use of GUI widgets that require computing cycles, such as the Top Sessions widget. These widgets constantly pol the system for information, which uses CPU and other resources.
  • Schedule antivirus, IPS, and firmware updates during off peak hours. These updates don't usually consume CPU resources but they can disrupt normal operation.
  • Check the log levels and which events are being logged. This is the severity of the messages that are recorded. Consider going up one level to reduce the amount of logging. Also, if there are events you don't need to monitor, remove them from the list.
  • Log to FortiCloud instead of logging to memory or disk. Logging to memory quickly uses up resources and logging to local disk impacts overall performance and reduces the lifetime of the unit. Fortinet recommends that you log to FortiCloud because it doesn’t use much CPU.
  • If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find free space and organize the files.
  • If packet logging is enabled on the FortiGate, consider disabling it. When packet logging is enable, it records every packet that comes through that policy.
  • Halt all sniffers and traces.
  • Ensure the FortiGate isn't scanning traffic twice. If traffic enters the FortiGate on one interface, goes out another, and then comes back in again, that traffic doesn't need to be rescanned. Doing so is a waste of resources. However, ensure that traffic truly is being scanned once.
  • Reduce the session timers to close unused sessions faster. Enter the following CLI commands, which reduce the default values. Note that, by default, the system adds 10 seconds to tcp-timewait.

    config system global

    set tcp-halfclose-timer 30

    set tcp-halfopen-timer 30

    set tcp-timewait-timer 0

    set udp-idle-timer 60

    end

  • In System > Feature Visibility, only enable features that you need.

SNMP monitoring

When CPU usage is under control, use SNMP to monitor CPU usage. Alternatively, use logging to record CPU and memory usage every 5 minutes.

Once things are back to normal, you should set up a warning system that sends you alerts when CPU resources are used excessively. A common method to do this is using SNMP. SNMP monitors many values in FortiOS and allows you to set high water marks that generate events. You run an application on your computer to watch for and record these events. Go to System > SNMP to enable and configure an SNMP community. If this method is too complicated, you can use the System Resources widget to record CPU usage. However, this method only records problems as they happen and won't send you alerts for problems.

How to check modem status

If the modem doesn't work properly or a FortiGate doesn't detect the modem, you can use the following diagnostic command to help you troubleshoot issues with the modem:

diagnose sys modem {cmd | com | detect | history | external-modem | query| reset}

You should always run the following diagnose command after you insert the USB modem into the FortiGate:

diagnose sys modem detect

You can view the modem configuration, using the get system modem command. You can also view the modem’s vendor and the custom product identification number from the information output from the get system modem command.

When there are connectivity issues, use the following to help you resolve them:

  • diagnose debug enable – activates the debug on the console
  • diagnose debug application modemd – dumps communication between the modem and the unit.
  • diagnose debug application ppp – dumps the PPP negotiating messages.
  • execute modem dial – displays modem debug output.

The modem diagnose output shouldn't contain errors on the way to initializing. You should also verify the number that is used to dial into your ISP.

How to run ping and traceroute

Ping and traceroute are useful tools in network troubleshooting. Alone, either one can determine network connectivity between two points. However, ping can be used to generate simple network traffic that you can view using diagnose commands on the FortiGate. This combination can be very powerful when you're trying to locate network problems.

In addition to their normal uses, ping and traceroute can tell you if your computer or network device has access to a domain name server (DNS). While both tools can use IP addresses alone, they can also use domain names for devices. This is an added troubleshooting feature that can be useful in determining why particular services, such as email or web browsing, may not work properly.

caution icon

If ping doesn't work, it may be disabled on at least one of the interface settings, and security policies for that interface.

Both ping and traceroute require particular ports to be open on firewalls, or else they can't function. Since you typically use these tools to troubleshoot, you can allow them in the security policies and on interfaces only when you need them. Otherwise, keep the ports disabled for added security.

Ping

The ping command sends a very small packet to a destination, and waits for a response. The response has a timer that expires when the destination is unreachable.

Ping is part of layer 3 on the OSI Networking Model. Ping sends Internet Control Message Protocol (ICMP) “echo request” packets to the destination, and listens for “echo response” packets in reply. However, many public networks block ICMP packets because ping can be used in a denial of service (DoS) attack (such as Ping of Death or a smurf attack), or by an attacker to find active locations on the network. By default, FortiGate units have ping enabled while broadcast-forward is disabled on the external interface.

What ping can tell you

Beyond the basic connectivity information, ping can tell you the amount of packet loss (if any), how long it takes the packet to make the round trip, and the variation in that time from packet to packet.

If there is some packet loss detected, you should investigate the following:

  • Possible ECMP, split horizon, or network loops.
  • Cabling, to ensure no loose connections.
  • Verify which security policy was used (use the packet count column on the Policy & Objects > IPv4 Policy or Policy & Objects > IPv6 Policy page).

If there is total packet loss, you should investigate the following:

  1. Ensure cabling is correct, and all equipment between the two locations is accounted for.
  2. Ensure all IP addresses and routing information along the route is configured as expected.
  3. Ensure all firewalls, including FortiGate security policies allow PING to pass through.
How to use ping

Ping syntax is the same for nearly every type of system on a network.

To ping from a FortiGate unit
  1. Connect to the CLI either through telnet or through the CLI widget on the Dashboard .
  2. Enter exec ping 10.11.101.101 to send 5 ping packets to the destination IP address. There are no options for this command.

    Sample output:

    Head_Office_620b # exec ping 10.11.101.101

    PING 10.11.101.101 (10.11.101.101): 56 data bytes

    64 bytes from 10.11.101.101: icmp_seq=0 ttl=255 time=0.3 ms

    64 bytes from 10.11.101.101: icmp_seq=1 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=2 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=3 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=4 ttl=255 time=0.2 ms

    --- 10.11.101.101 ping statistics ---

    5 packets transmitted, 5 packets received, 0% packet loss

    round-trip min/avg/max = 0.2/0.2/0.3 ms

To ping from a MicroSoft Windows PC
  1. Open a command window.
  1. Enter ping 10.11.101.100 to ping the default internal interface of the FortiGate with four packets.

Other options include:

  • -t to send packets until you press “Ctrl+C”
  • -a to resolve addresses to domain names where possible
  • -n X to send X ping packets and stop

Sample output:

C:\>ping 10.11.101.101

Pinging 10.11.101.101 with 32 bytes of data:

Reply from 10.11.101.101: bytes=32 time=10ms TTL=255

Reply from 10.11.101.101: bytes=32 time<1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Ping statistics for 10.11.101.101:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 10ms, Average = 3ms

To ping from a Linux PC
  1. Go to a shell prompt.
  2. Enter “ping 10.11.101.101”.

Traceroute

Where ping will only tell you if it reached its destination and returned successfully, traceroute shows each step of the journey to its destination and how long each step takes. If ping finds an outage between two points, you can use traceroute to locate exactly where the problem is.

What is traceroute

Traceroute works by sending ICMP packets to test each hop along the route. It sends three packets, and then increases the time to live (TTL) setting by one each time. This effectively allows the packets to go one hop farther along the route. This is the reason why most traceroute commands display their maximum hop count before they start tracing the route, which is the maximum number of steps it takes before it declares the destination unreachable. Also, the TTL setting may result in steps along the route timing out due to slow responses. There are many possible reasons for this to occur.

By default, traceroute uses UDP datagrams with destination ports numbered from 33434 to 33534. The traceroute utility may also offer the option to select use of ICMP echo request (type 8) instead, which the Windows tracert utility uses. If you must, allow both protocols inbound through the FortiGate security policies (UDP with ports from 33434 to 33534 and ICMP type 8).

You can also use the packet count column of the Policy & Objects > IPv4 Policy (or, Policy & Objects > IPv6 Policy, if applicable) page to track traceroute packets. This allows you to verify the connection and confirm which security policy the traceroute packets are using.

What traceroute can tell you

Ping and traceroute have similar functions, which is to verify connectivity between two points. The big difference is that traceroute shows you each step of the way, where ping doesn't. Also, ping and traceroute use different protocols and ports, so one may succeed where the other fails.

You can verify your DNS connection using traceroute. If you enter an FQDN instead of an IP address for the traceroute, DNS tries to resolve that domain name. If the name isn't resolved, you have DNS issues.

How to use traceroute

The traceroute command varies slightly between operating systems. Note that in MicroSoft Windows, the command name is shortened to “tracert”. Also, your output lists different domain names and IP addresses along your route.

To use traceroute on a MicroSoft Windows PC
  1. Open a command window.
  1. Enter tracert fortinet.com to trace the route from the PC to the Fortinet web site.

Sample output:

C:\>tracert fortinet.com

Tracing route to fortinet.com [208.70.202.225]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.20.120.2

2 66 ms 24 ms 31 ms 209-87-254-xxx.storm.ca [209.87.254.221]

3 52 ms 22 ms 18 ms core-2-g0-0-1104.storm.ca [209.87.239.129]

4 43 ms 36 ms 27 ms core-3-g0-0-1185.storm.ca [209.87.239.222]

5 46 ms 21 ms 16 ms te3-x.1156.mpd01.cogentco.com [38.104.158.69]

6 25 ms 45 ms 53 ms te8-7.mpd01.cogentco.com [154.54.27.249]

7 89 ms 70 ms 36 ms te3-x.mpd01.cogentco.com [154.54.6.206]

8 55 ms 77 ms 58 ms sl-st30-chi-.sprintlink.net [144.232.9.69]

9 53 ms 58 ms 46 ms sl-0-3-3-x.sprintlink.net [144.232.19.181]

10 82 ms 90 ms 75 ms sl-x-12-0-1.sprintlink.net [144.232.20.61]

11 122 ms 123 ms 132 ms sl-0-x-0-3.sprintlink.net [144.232.18.150]

12 129 ms 119 ms 139 ms 144.232.20.7

13 172 ms 164 ms 243 ms sl-321313-0.sprintlink.net [144.223.243.58]

14 99 ms 94 ms 93 ms 203.78.181.18

15 108 ms 102 ms 89 ms 203.78.176.2

16 98 ms 95 ms 97 ms 208.70.202.225

The first column on the left is the hop count, which can't exceed 30 hops. When that number is reached, the traceroute ends.

The second, third, and fourth columns display how much time each of the three packets takes to reach this stage of the route. These values are in milliseconds and normally vary quite a bit. Typically a value of <1ms indicates a local connection.

The fifth column (farthest to the right) shows the domain name of the device and its IP address, or possibly only the IP address.

To perform a traceroute on a Linux PC
  1. Go to a command line prompt.
  2. Enter “traceroute fortinet.com”.

The Linux traceroute output is very similar to the MicroSoft Windows tracert output.

To trace a route from a FortiGate to a destination IP address - CLI

# execute traceroute www.fortinet.com

traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets

1 172.20.120.2 0.637 ms 0.653 ms 0.279 ms

2 209.87.254.221 <static-209-87-254-221.storm.ca> 2.448 ms 2.519 ms 2.458 ms

3 209.87.239.129 <core-2-g0-2.storm.ca> 2.917 ms 2.828 ms 9.324 ms

4 209.87.239.199 <core-3-bdi1739.storm.ca> 13.248 ms 12.401 ms 13.009 ms

5 216.66.41.113 <v502.core1.tor1.he.net> 17.181 ms 12.422 ms 12.268 ms

6 184.105.80.9 <100ge1-2.core1.nyc4.he.net> 21.355 ms 21.518 ms 21.597 ms

7 198.32.118.41 <ny-paix-gni.twgate.net> 83.297 ms 84.416 ms 83.782 ms

8 203.160.228.217 <217-228-160-203.TWGATE-IP.twgate.net> 82.579 ms 82.187 ms 82.066 ms

9 203.160.228.229 <229-228-160-203.TWGATE-IP.twgate.net> 82.055 ms 82.455 ms 81.808 ms

10 203.78.181.2 82.262 ms 81.572 ms 82.015 ms

11 203.78.186.70 83.283 ms 83.243 ms 83.293 ms

12 66.171.127.177 84.030 ms 84.229 ms 83.550 ms

13 66.171.121.34 <www.fortinet.com> 84.023 ms 83.903 ms 84.032 ms

14 66.171.121.34 <www.fortinet.com> 83.874 ms 84.084 ms 83.810 ms

How to check the logs

You might forget his step in troubleshooting, but it's an important one. Logging records the traffic that passes through the FortiGate to your network and what action the FortiGate takes when it scans the traffic. This information record is called a log message.

When you first configure FortiOS, log as much information as you can.If the logs that the FortiGatte generates are too large, you can turn off or scale back the logging for features that you're not using.

As with most troubleshooting steps, before you can determine if the logs indicate a problem, you need to know what logs result from normal operation. Without a baseline it's difficult to troubleshoot.

When you troubleshoot with log files:

  • Compare current logs to a recorded baseline of normal operation.
  • If you need to, increase the level of logging (such as from Warning to Information) to obtain more information.

When increasing logging levels, ensure that you configure email alerts and select both disk usage and log quota. This ensures that you'll be notified if the increas in logging causes problems. Configure log settings by going to Log & Report > Log Settings.

Determine the activities that generate the most log entries:

  • Check all logs to ensure important information isn't overlooked.
  • Filter or order log entries based on different fields, such as level, service, or IP address, to look for patterns that may indicate a specific problem, such as frequent blocked connections on a specific port for all IP addresses.

Logs will help identify and locate any problems, but they will not solve the problems. The purpose of logs is to speed up your problem solving and save you time and effort.

For more information about logging and log reports, see Logging and reporting.

How to verify the contents of the routing table (in NAT mode)

When a FortiGate has limited or no connectivity, a good place to look for information is the routing table.

The routing table is where all the currently used routes are stored for both static and dynamic protocols. If a route is in the routing table, it saves time and resources that you would spend performing a lookup. If a route is not used for a while and a new route needs to be added, the oldest least used route is bumped if the routing table is full. This ensures that the most recently used routes stay in the table. If your FortiGate is in transparent mode, you can't perform this step.

If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table, including local subnets, default routes, specific static routes, and dynamic routing protocols.

To check the routing table in the FortiGate GUI, use the Routing Monitor by going to Monitor > Routing Monitor.

In the CLI, use the command get router info routing-table all.

Sample output:

FGT# get router info routing-table all

Codes:

K - kernel, C - connected, S - static, R - RIP, B - BGP

O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

* - candidate default

S* 0.0.0.0/0 [10/0] via 172.20.120.2, wan1

C 10.31.101.0/24 is directly connected, internal

C 172.20.120.0/24 is directly connected, wan1

How to verify the correct route is being used

If you have more than one default route and want to make sure that traffic is flowing as expected and through the right route, you can run a trace route from a machine in the local area network (LAN). This shows you the first hop that the traffic goes through.

Sample output:

C:\>tracert www.fortinet.com

Tracing route to www.fortinet.com [66.171.121.34]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.10.1.99

2 1 ms <1 ms <1 ms 172.20.120.2

3 3 ms 3 ms 3 ms static-209-87-254-221.storm.ca [209.87.254.221]

4 3 ms 3 ms 3 ms core-2-g0-2.storm.ca [209.87.239.129]

5 13 ms 13 ms 13 ms core-3-bdi1739.storm.ca [209.87.239.199]

6 12 ms 19 ms 11 ms v502.core1.tor1.he.net [216.66.41.113]

7 22 ms 22 ms 21 ms 100ge1-2.core1.nyc4.he.net [184.105.80.9]

8 84 ms 84 ms 84 ms ny-paix-gni.twgate.net [198.32.118.41]

9 82 ms 84 ms 82 ms 217-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.217]

10 82 ms 81 ms 82 ms 229-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.229]

11 82 ms 82 ms 82 ms 203.78.181.2

12 84 ms 83 ms 83 ms 203.78.186.70

13 84 ms * 85 ms 66.171.127.177

14 84 ms 84 ms 84 ms fortinet.com [66.171.121.34]

15 84 ms 84 ms 83 ms fortinet.com [66.171.121.34]

Trace complete.

In this scenario, the first hop contains the IP address 10.10.1.99, which is the internal interface of the FortiGate. The second hop contains the IP address 172.20.120.2, to which the wan1 interface of the FortiGate is connected, so we can conclude that the route through wan1 interface is being used for this traffic.

You can also see the route taken for each session by debugging the packet flow in the CLI. For more information, see How to debug the packet flow.

How to verify the correct firewall policy is being used

If you have more than one firewall policy, use the count column to check which policy is being used, the count must show traffic increasing. To do so, go to the Policy & Objects > IPv4 Policy or Policy & Objects > IPv6 Policy page.

Also, debugging the packet flow in the CLI shows the policy ID that's allowing the traffic. For more information about debugging the packet flow, see How to debug the packet flow.

How to check the bridging information in transparent mode

When the FortiGate is set to transparent mode, it acts like a bridge and sends all incoming traffic out on the other interfaces. The bridge is between interfaces on the FortiGate.

Each bridge that's listed is a link between interfaces. Where traffic is flowing between interfaces, you can expect to find bridges listed. If you're having connectivity issues, and there are no bridges listed, that is a likely cause. Check for the MAC address of the interface or device in question.

How to check the bridging information

To list the existing bridge instances on the FortiGate, use the following command:

diagnose netlink brctl list

Sample output:

#diagnose netlink brctl list

list bridge information

1. root.b fdb: size=256 used=6 num=7 depth=2 simple=no

Total 1 bridges

How to display forwarding domain information

You can use forwarding domains, or collision domains, in routing to limit where packets are forwarded on the network. Layer 2 broadcasts are limited to the same group. By default, all interfaces are in group 0. For example, if the FortiGate has 12 interfaces, only two may be in the same forwarding domain, which limits packets that are broadcast to those two interfaces. This reduces traffic on the rest of the network.

Collision domains prevent the forwarding of ARP packets to all VLANs on an interface. Without collision domains, duplicate MAC addresses on VLANs may cause ARP packets to be duplicated. Duplicate ARP packets can cause some switches to reset. It's important to know what interfaces are part of which forwarding domains because this determines which interfaces can communicate with each other.

To manually configure forwarding domains in transparent mode, use the following CLI command:

config system interface

edit <interface_name>

set forward-domain <integer>

end

To display the information for forward domains, use the following command:

diagnose netlink brctl domain <name> <id>

where <name> is the name of the forwarding domain to display and <id> is the domain ID.

Sample output

diagnose netlink brctl domain ione 101

show bridge root.b ione forward domain.

id=101 dev=trunk_1 6

To list the existing bridge MAC table, use the following command:

diagnose netlink brctl name host <name>

Sample output

show bridge control interface root.b host.

fdb: size=256, used=6, num=7, depth=2, simple=no

Bridge root.b host table

port no

device

devname

mac addr

ttl

attributes

2

7

wan2

02:09:0f:78:69:00

0

Local Static

5

6

vlan_1

02:09:0f:78:69:01

0

Local Static

3

8

dmz

02:09:0f:78:69:01

0

Local Static

4

9

internal

02:09:0f:78:69:02

0

Local Static

3

8

dmz

00:80:c8:39:87:5a

194

4

9

internal

02:09:0f:78:67:68

8

1

3

wan1

00:09:0f:78:69:fe

0

Local Static

To list the existing bridge port list, use the following command:

diagnose netlink brctl name port <name>

Sample output:

show bridge root.b data port.

trunk_1 peer_dev=0

internal peer_dev=0

dmz peer_dev=0

wan2 peer_dev=0

wan1 peer_dev=0

How to examine the firewall session list

The firewall session list displays all sessions that the FortiGate has open. You can see if there are strange patterns, such as no sessions apart from the internal network, or all sessions are only to one IP address.

When you examine the firewall session list in the CLI, you can use filters to reduce the output. In the GUI, the filters are part of the interface.

To examine the firewall session list - GUI

Go to FortiView > All Sessions.

To examine the firewall session list - CLI

When you examine the firewall session list, there may be too many sessions to display. In this case, it's necessary to limit or filter the sessions displayed by source or destination address, or NAT'd address or port. If you want to filter by more than one of these, you need to enter a separate line for each value.

The following example shows filtering the session list based on a source address of 10.11.101.112:

FGT# diagnose sys session filter src 10.11.101.112

FGT# diagnose sys session list

The following example shows filtering the session list based on a destination address of 172.20.120.222.

FGT# diagnose sys session filter dst 172.20.120.222

FGT# diagnose sys session list

To clear all sessions corresponding to a filter - CLI

FGT# diagnose sys session filter dst 172.20.120.222

FGT# diagnose sys session clear

Check source NAT information

When you troubleshoot connections, remember NAT. NAT is especially important if you're troubleshooting from the remote end of the connection outside the firewall. On the FortiView > All Sessions list, pay attention to NAT Source, and NAT Source Port. These columns display the IP and port values after NAT has been applied.

Checking the NAT values can help you to ensure that they are the values you expect, and to ensure the remote end of the sessions can see the expected IP address and port number.

When you display the session list in the CLI, you can match the NAT'd source address (nsrc) and port (nport). This can be useful if multiple internal IP addresses are NAT'd to a common external-facing source IP address.

FGT# diagnose sys session filter nsrc 172.20.120.122

FGT# diagnose sys session filter nport 8888

FGT# diagnose sys session list

How to check wireless information

Wireless connections, stations, and interfaces have different issues than other physical interfaces.

Troubleshooting station connection issue

To check whether a station entry is created on access control, use the following command:

FG600B3909600253 # diagnose wireless-controller wlac -d sta

* vf=0 wtp=70 rId=2 wlan=open ip=0.0.0.0 mac=00:09:0f:db:c4:03 rssi=0 idle=148 bw=0 use=2

vf=0 wtp=70 rId=2 wlan=open ip=172.30.32.122 mac=00:25:9c:e0:47:88 rssi=-40 idle=0 bw=9 use=2

Enable diagnostics for a particular station

This example uses the station MAC address to find where it's failing:

FG600B3909600253 # diagnose wireless-controller wlac sta_filter 00:25:9c:e0:47:88 1

Set filter sta 00:25:9c:e0:47:88 level 1

FG600B3909600253 # 71419.245 <ih> IEEE 802.11 mgmt::disassoc <== 00:25:9c:e0:47:88 vap open rId 1 wId 0 00:09:0f:db:c4:03

71419.246 <dc> STA del 00:25:9c:e0:47:88 vap open rId 1 wId 0

71419.246 <cc> STA_CFG_REQ(34) sta 00:25:9c:e0:47:88 del ==> ws (0-192.168.35.1:5246) rId 1 wId 0

71419.246 <cc> STA del 00:25:9c:e0:47:88 vap open ws (0-192.168.35.1:5246) rId 1 wId 0 00:09:0f:db:c4:03 sec open reason I2C_STA_DEL

71419.247 <cc> STA_CFG_RESP(34) 00:25:9c:e0:47:88 <== ws (0-192.168.35.1:5246) rc 0 (Success).

How to verify connectivity to FortiGuard

You can verify connectivity to FortiGuard in the Licenses widget on the FortiGate Dashboard. When your FortiGate is connected to FortiGuard, a green check mark appears for available FortiGuard services.

To verify connectivity from the CLI, use the execute ping service.fortiguard.net and execute ping update.fortiguard.net commands.

Sample output:

FG100D# execute ping service.fortiguard.net

PING guard.fortinet.net (208.91.112.196): 56 data bytes

64 bytes from 208.91.112.196: icmp_seq=0 ttl=51 time=61.0 ms

64 bytes from 208.91.112.196: icmp_seq=1 ttl=51 time=60.0 ms

64 bytes from 208.91.112.196: icmp_seq=2 ttl=51 time=59.6 ms

64 bytes from 208.91.112.196: icmp_seq=3 ttl=51 time=58.9 ms

64 bytes from 208.91.112.196: icmp_seq=4 ttl=51 time=59.2 ms

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 58.9/59.7/61.0 ms

FG100D# execute ping update.fortiguard.net

PING fds1.fortinet.com (208.91.112.68): 56 data bytes

64 bytes from 208.91.112.68: icmp_seq=0 ttl=53 time=62.0 ms

64 bytes from 208.91.112.68: icmp_seq=1 ttl=53 time=61.8 ms

64 bytes from 208.91.112.68: icmp_seq=2 ttl=53 time=61.3 ms

64 bytes from 208.91.112.68: icmp_seq=3 ttl=53 time=61.9 ms

64 bytes from 208.91.112.68: icmp_seq=4 ttl=53 time=61.8 ms

How to perform a sniffer trace (CLI and packet capture)

When you troubleshoot networks and routing in particular, it helps to look inside the headers of packets to determine if they're traveling along the route that you expect them to take. Packet sniffing can also be called a network tap, packet capture, or logic analyzing.

caution icon

If your FortiGate has NP2, NP4, or NP6 interfaces that are offloading traffic, this will change the sniffer trace. Before you perform a trace on any NP2, NP4, or NP6 interfaces, you should disable offloading on those interfaces.

How do you sniff packets

Before you start sniffing packets on the CLI, you should prepare to capture the output to a file. A large amount of data may scroll by and you won't be able to see it without first saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate CLI. Once the packet sniffing count is reached, you can end the session and analyze the output in the file.

The general form of the internal FortiOS packet sniffer command is:

diagnose sniffer packet <interface_name> <‘filter’> <verbose> <count> <tsformat>

To stop the sniffer, type CTRL+C.

<interface_name>

The name of the interface to sniff, such as “port1” or “internal”. This can also be “any” to sniff all interfaces.

<‘filter’>

What to look for in the information the sniffer reads. “none” indicates no filtering, and all packets are displayed as the other arguments indicate.

The filter must be inside single quotes (‘).

<verbose>

The level of verbosity as one of:

1 - print header of packets
2 - print header and data from IP of packets
3 - print header and data from Ethernet of packets
4 - print header of packets with interface name

<count>

The number of packets the sniffer reads before stopping. If you don't put a number here, the sniffer will run until you stop it with <CTRL+C>.

<tsformat>

The format of timestamp.
  • a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms
  • l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms
  • otherwise: relative to the start of sniffing, ss.ms

For a simple sniffing example, enter the CLI command diagnose sniffer packet port1 none 1 3. This displays the next three packets on the port1 interface using no filtering, and verbose level 1. At this verbosity level, you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.

In the output below, port 443 indicates these are HTTPS packets and that 172.20.120.17 is both sending and receiving traffic.

Head_Office_620b # diagnose sniffer packet port1 none 1 3

interfaces=[port1]

filters=[none]

0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757

0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808

0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933

For a more advanced example of packet sniffing, the following commands will report packets on any interface that are traveling between a computer with the host name of “PC1” and a computer with the host name of “PC2”. With verbosity 4 and above, the sniffer trace displays the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C.

FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4

or

FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4

The following CLI command for a sniffer includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for example, PC2 may be down and not responding to the FortiGate ARP requests).

FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4

How do you use packet capture

To use packet capture, the FortiGate must have a disk. You can enable the capture-packet in the firewall policy, using the following CLI commands:

config firewall policy

edit <id>

set capture-packet enable

end

To configure packet capture filters, go to Network > Packet Capture.

When you add a packet capture filter, enter the following information and select OK.

Interface

Select the interface to sniff from the drop-down menu.

You must select one interface. You can't change the interface without deleting the filter and creating a new one, unlike the other fields.

Max Packets to Save

Enter the number of packets to capture before the filter stops.

This number can't be zero. You can halt the capturing before this number is reached.

Enable Filters

Select this option to specify filter fields

Host(s)

Enter the IP address of one or more hosts.

Separate multiple hosts with commas. To enter a range, use a dash without spaces, for example 172.16.1.5-172.16.1.15, or enter a subnet.

Port(s)

Enter one or more ports to capture on the selected interface.

Separate multiple ports with commas. To enter a range, use a dash without spaces, for example 88-90

VLAN(s)

Enter one or more VLANs (if any). Separate multiple VLANs with commas.

Protocol

Enter one or more protocols. Separate multiple protocols with commas. To enter a range, use a dash without spaces, for example 1-6, 17, 21-25.

Include IPv6 Packets

Select this option if you're troubleshooting IPv6 networking, or if your network uses IPv6. Otherwise, leave it disabled.

Include Non-IP Packets

The protocols in the list are all IP based except for ICMP (ping). To capture non-IP based packets, select this feature. Examples of non-IP packets include IPsec, IGMP, ARP, and ICMP.

If you select a filter, you have the option to start and stop packet capture in the edit window, or download the captured packets. You can also see the filter status and the number of packets captured.

You can select the filter and start capturing packets. When the filter is running, the number of captured packets increases until it reaches the Max Packet Count or you stop it. When the filter is running, you can't download the output file.

When the packet capture is complete, you can download the *.pcap file. You must use a third party application, such as Wireshark, to read *,pcap files. This tool provides you with extensive analytics and the full contents of the packets that were captured.

To start, stop, or resume packet capture, use the symbols on the screen. These symbols are the same as those used for audio or video playback. Hover over the symbol to reveal explanatory text. Similarly, to download the *.pcap file, use the download symbol on the screen.

For more information on troubleshooting with packet capture and packet sniffing, see Packet sniffing and packet capture.

How to debug the packet flow

Traffic should come in and leave the FortiGate. If you determine that network traffic isn't entering and leaving the FortiGate as expected, debug the packet flow.

You can only perform debugging using CLI commands. Debugging the packet flow requires that you enter a number of debug commands as each one configures part of the debug action, with the final command starting the debug.

caution icon

If your FortiGate has FortiASIC NP4 or NP6 interface pairs that are offloading traffic, this changes the packet flow. Before you perform the debug on any NP4 or NP6 interfaces, you should disable offloading on those interfaces. Enter diagnose npu <interface pair> fastpath disable, where interface pair is np4, np6, np4lite, or np6lite.

The following configuration assumes that PC1 is connected to the internal interface of the FortiGate and has an IP address of 10.11.101.200. PC1 is the host name of the computer.

To debug the packet flow in the CLI, enter the following commands:

FGT# diagnose debug disable

FGT# diagnose debug flow filter add <PC1>

FGT# diagnose debug flow show console enable

FGT# diagnose debug flow show function-name enable

FGT# diagnose debug flow trace start 100

FGT# diagnose debug enable

The start 100 argument in the above list of commands limits the output to 100 packets from the flow. This is useful to look at the flow without flooding your log or displaying too much information.

To stop all other debug activities, enter the command:

FGT# diagnose debug flow trace stop

The following is an example of debug flow output for traffic that has no matching security policy, and is in turn blocked by the FortiGate unit. The denied message indicates that the traffic was blocked.

id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg="vd-root received a packet(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3."

id=20085 trace_id=319 func=resolve_ip_tuple line=2924 msg="allocate a new session-013004ac"

id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-192.168.150.129 via port1"

id=20085 trace_id=319 func=fw_forward_handler line=248 msg=" Denied by forward policy check"

Troubleshooting tips

The following tips present common causes of problems.

How to check hardware connections

  • Are all of the cables and interfaces connected properly?
  • Is the LED for the interface green?

How to check FortiOS network settings

  • If you're having problems connecting to the management interface, is your protocol enabled on the interface for administrative access?
  • Does the interface have an IP address?

How to check CPU and memory resources

  • Is the CPU running at almost 100 percent usage?
  • Is your FortiGate running low on memory?

How to check modem status

  • Is the modem connected?
  • Are there PPP issues?

How to run ping and traceroute

  • Is the FortiGate experiencing complete packet loss?

How to check the logs

  • Do you need to identify a problem?

How to verify the contents of the routing table (in NAT mode)

  • Are there routes in the routing table for default and static routes?
  • Do all connected subnets have a route in the routing table?
  • Does a route have a higher priority than it should?

How to verify the correct route is being used

  • Is the traffic routed correctly?

How to verify the correct firewall policy is being used

  • Is the correct firewall policy applied to the expected traffic?

How to check the bridging information in transparent mode

  • Are you having problems in transparent mode?

How to examine the firewall session list

  • Are there active firewall sessions?

How to check wireless information

  • Is the wireless network working properly?

How to verify FortiGuard connectivity

  • Is the FortiGate communicating properly with FortiGuard?

How to perform a sniffer trace (CLI and packet capture)

  • Is traffic entering the FortiGate? Does the traffic arrive on the expected interface?
  • Is the ARP resolution correct for the next-hop destination?
  • Is the traffic exiting the FortiGate to the destination as expected?
  • Is the FortiGate sending traffic back to the originator?

How to debug the packet flow

  • Is traffic entering or leaving the FortiGate as expected?

How to check hardware connections

If there's no traffic flowing from the FortiGate, you may have a hardware problem.

To check hardware connections:

  1. Ensure network cables are plugged into the interfaces.
  2. Verify that the LED connection lights for the network cables are the right color (usually green).
  3. If the cable or its connector are damaged, change the cable. You should also change the cable if you're not sure about the type or quality of the cable, such as straight through or crossover, of if you see exposed wires at the connector.
  4. Connect the FortiGate to different hardware.
  5. Ensure the link status is set to Up for the interface (see Network > Interfaces). The link status is based on the physical connection and can't be set in FortiOS.

If any of these solve the problem, it was a hardware connection problem. You should still perform some basic software connectivity tests to ensure complete connectivity. The interface might also be disabled, or its Administrative Status might be set to Down.

To enable an interface - GUI
  1. Go to Network > Interfaces.
  2. Select and edit the interface to enable, such as port1.
  3. Find Administrative Status at the bottom of the screen, and select Up.
  4. Select Apply.
To enable an interface - CLI

config system interface

edit port1

set status up

next

end

How to check FortiOS network settings

You can manage FortiOS network settings in the GUI and the CLI. The following information includes troubleshooting and best practice information. The network settings include:

Interface settings

If you can access the FortiGate with the management cable only, the first step is to display the interface settings. To display the settings for the internal interface, use the following CLI command:

FGT# show system interface <interface_name>

For a complete listing of all the possible interface settings, use the following CLI command:

config system interface

edit <interface_name>

get

end

Check the interface settings to ensure that they aren't preventing traffic. Check the following items (only the GUI terms are shown, CLI terms may vary):

Setting

Description

Link Status

Down until a valid cable is plugged into this interface, then it will be Up.

The Link Status is shown physically by the connection LED for the interface. If the LED is green, the connection is good. If Link Status is Down, the interface doesn't work. Link Status is also displayed on the Network > Interfaces screen, by default.

Addressing mode

Don't use DHCP if you don’t have a DHCP server. You won't be able to log in to an interface in DHCP mode as it won't have an IP address.

IP/Network Mask

An interface needs an IP address to be able to connect to other devices. Ensure there is a valid IP address in this field. The one exception is if DHCP is enabled for this interface to get its IP address from an external DHCP server.

IPv6 address

The same protocol must be used by both ends to complete the connection. Ensure both this interface and the remote connection are both using IPv4 or both using IPv6 addressing.

Administrative access

If no protocols are selected, you will have to use the local management cable to connect to the unit. If you're using IPv6, configure the IPv6 administrative access protocols.

Administrative status

Set to Up or the interface won't work.

DNS settings

You can trace many networking problems back to DNS issues. Check the following items:

  1. Are there values for both primary and secondary entries?
  2. Is the local domain name correct?
  3. Are you using IPv6 addressing? If so, are the IPv6 DNS settings correct?
  4. Are you using Dynamic DNS (DDNS)? If so, is it using the correct server, credentials, and interface?
  5. Can you contact both DNS servers to verify the servers are operational?
  6. If an interface addressing mode is set to DHCP and is set to override the internal DNS, is that interface receiving a valid DNS entry from the DHCP server? Is it a reasonable address and can it be contacted to verify it’s operational?
  7. Are there any DENY security policies that need to allow DNS?
  8. Can any internal device perform a successful traceroute to a location using the FQDN?

DHCP server settings

DHCP servers are common on internal and wireless networks. If the DHCP server isn't configured correctly, it can cause problems. Check the following items:

  1. Is the DHCP server entry set to Relay? If so, verify there is another DHCP server to which requests can be relayed. Otherwise, it should be set to Server.
  2. Is the DHCP server enabled?
  3. Does the DHCP server use a valid IP address range? Are other devices using the addresses? If one or more devices are using IP addresses in this range, you can use the IP reservation feature to ensure the DHCP server doesn't use these addresses.
  4. Is there a gateway entry? If not, add a gateway entry to ensure that the server's clients have a default route.
  5. Is the system DNS setting being used? A best practice is to avoid confusion by using the system DNS whenever possible. However, you can specify up to three custom DNS servers, and you should use all three entries for redundancy.

caution icon

There are some situations, such as a new wireless interface, or during the initial FortiGate configuration, where interfaces override the system DNS entries. When this happens, it often shows up as intermittent Internet connectivity. To fix the problem, go to Network > DNS and enable Use FortiGuard Servers.

How to check CPU and memory resources

System resources are shared and a number of processes run simultaneously on the FortiGate unit.

The Resource widgets for CPU and Memory on the Dashboard provide a quick way to monitor usage.

To use the CLI to check the system resources on your FortiGate unit, run the following command:

FGT# get system performance status

This command provides a quick and easy snapshot of the FortiGate.

The first line of output shows the CPU usage by category. A FortiGate that is doing nothing looks like the following example:

CPU states: 0% user 0% system 0% nice 100% idle

However, if your network is running slow you might see something looks like the following example:

CPU states: 1% user 98% system 0% nice 1% idle

This line shows that all the CPU is used up by system processes. Normally this shouldn't happen since it shows that the FortiGate is overloaded for some reason. If you see this overloading, you should investigate further because it’s possible a process, such as scanunitid, is using all the resources to scan traffic, in which case you need to reduce the amount of traffic that's being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions. It's also possible that a hacker has gained access to your network and is overloading it with malicious activity, such as running a spam server or using zombie PCs to attack other networks on the Internet. You can use the get system performance top CLI command to get more information about the CPU. This command shows all of the top processes that are running on the FortiGate (the names are on the left) and their CPU usage. If a process is using most of the CPU cycles, investigate it to determine whether the activity is normal.

The second line of output from the get system performance status command shows the memory usage. Memory usage shouldn't exceed 90%. If memory is too full, some processes won't be able to function properly. For example, if the system is running low on memory, antivirus scanning enters into failopen mode where it drops connections or bypasses the antivirus system.

The other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked, can also help you determine why system resource usage is high. For example, if network usage is high it results in high traffic processing on the FortiGate, or if the session setup rate is very low (or zero) the proxy may be overloaded and unable to do its job.

How to troubleshoot high memory usage

As with any system, a FortiGate has limited hardware resources, such as memory, and all processes running on the FortiGate share the memory. Each process uses more or less memory, depending on its workload. For example, a process usually uses more memory in high traffic situations. If some processes use all of the available memory, other processes won't be able to run.

When high memory usage occurs, the services may freeze up, connections may be lost, or new connections may be refused.

If you see high memory usage in the Memory widget, the FortiGate may be handling high traffic volumes. Alternatively, the FortiGate may have with connection pool limits that are affecting a single proxy. If the FortiGate receives large volumes of traffic on a specific proxy, the unit may exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, issues may occur.

Use the following CLI command, which shows information about current memory usage:

diagnose hardware sysinfo memory

Sample output:

total: used: free: shared: buffers: cached: shm:

Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528

Swap: 0 0 0

MemTotal: 2025572 kB

MemFree: 1286376 kB

MemShared: 0 kB

Buffers: 20216 kB

Cached: 189996 kB

SwapCached: 0 kB

Active: 56644 kB

Inactive: 153648 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 2025572 kB

LowFree: 1286376 kB

SwapTotal: 0 kB

SwapFree: 0 kB

How to troubleshoot high CPU usage

If you deploy too many FortiOS features at the same time, you can overextend the CPU resources on a FortiGate. When this occurs, the FortiGate undergoes connection-related problems.

Some examples of CPU intensive features are:

  • VPN high-level encryption
  • Intensive scanning of all traffic
  • Logging all traffic and packets
  • Dashboard widgets that frequently perform data updates

See CPU and memory thresholds for information on customizing the CPU use threshold.

Determine the current level of CPU usage

There are two ways to do this. The simplest is to look at the CPU widget on the Dashboard in the FortiGate GUI. The real-time CPU usage is displayed for different timeframes. The second method provides precise usage values both for overall usage and for specific processes. To use it, run the diagnose sys top CLI command.

Run Time: 86 days, 0 hours and 10 minutes

0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 3040T, 2437F

bcm.user 93 S < 3.1 0.4

httpsd 18922 S 1.5 0.5

httpsd 19150 S 0.3 0.5

newcli 20195 R 0.1 0.1

cmdbsvr 115 S 0.0 0.8

pyfcgid 20107 S 0.0 0.6

forticron 146 S 0.0 0.5

httpsd 139 S 0.0 0.5

cw_acd 166 S 0.0 0.5

miglogd 136 S 0.0 0.5

pyfcgid 20110 S 0.0 0.4

pyfcgid 20111 S 0.0 0.4

pyfcgid 20109 S 0.0 0.4

httpsd 20192 S 0.0 0.4

miglogd 174 S 0.0 0.4

miglogd 175 S 0.0 0.4

fgfmd 165 S 0.0 0.3

newcli 20191 S 0.0 0.3

initXXXXXXXXXXX 1 S 0.0 0.3

httpsd 184 s 0.0 0.3

The codes displayed on the second output line are explained in the following table:

Code

Description

U

the percentage of user space applications that are currently using the CPU

N

the percentage of time that the CPU spent on low priority processes since the last shutdown

S

the percentage of system processes (or kernel processes) that are using the CPU

I

the percentage of idle CPU resources

WA

the percentage of time that the CPU spent waiting on IO peripherals since the last shutdown

HI

the percentage of time that the CPU spent handling hardware interrupt routines since the last shutdown

SI

the percentage of time that the CPU spent handling software interrupt routines since the last shutdown

ST

or steal time, is the percentage of time that a virtual CPU waits for the physical CPU when the hypervisor is servicing another virtual processor

T

is the total FortiOS system memory, in MB

F

is free memory, in MB

Each additional line of the command output displays information for specific processes running on the FortiGate unit. For example, the third line of the output is: newcli 20195 R 0.1 0.1

The items in the third line of output are explained in the following table:

Item

Description

newcli

The process name.

Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd. Duplicate process names indicate that separate instances of that process that are running.

20195

The process ID, which can be any number.

R

Current state of the process. The process state can be:
  • R - running
  • S - sleep
  • Z - zombie
  • D - disk sleep

0.1

The percentage of CPU capacity that the process is using.

CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that's taking a lot of CPU time

0.1

The amount of memory that the process is using.

Memory usage can range from 0.1 to 5.5 and higher.

When diagnose sys top is running, enter the following single-key commands:

  • q to quit and return to the normal CLI prompt.
  • p to sort the processes by the amount of CPU that the processes are using.
  • m to sort the processes by the amount of memory that the processes are using.

The processes listed are the top processes that are running not all of the processes. For example, if there are 20 processes listed, they are the top 20 processes, sorted by either CPU or memory usage. You can configure the number of processes displayed, using the following CLI command:

diagnose sys top <integer_seconds> <integer_maximum_lines>

Where:

  • <integer_seconds> is the delay in seconds (default is 5)
  • <integer_maximum_lines> is the maximum number of lines (or processes) to list (default is 20)

Determine what features are using most of the CPU resources

There is a command in the CLI that shows the top few processes that are currently running and use the most CPU resources. The get system performance topcommand shows a table of information. The second column from the right shows CPU usage by percentage. If the top few entries are using most of the CPU, note which processes they are and try to reduce their CPU load.

Some examples of processes you'll see include:

  • ipsengine — the IPS engine that scans traffic for intrusions
  • scanunitd — antivirus scanner
  • httpsd — secure HTTP
  • iked — internet key exchange (IKE) in use with IPsec VPN tunnels
  • newcli — active whenever you're accessing the CLI
  • sshd — there are active secure socket connections
  • cmdbsrv — the command database server application

Go to the features that are at the top of the list and look for evidence of them overusing the CPU. Generally, the monitor for a feature is a good place to start.

Check for unnecessary CPU “wasters”

These are some best practices that will reduce your CPU usage, even if the FortiGate isn't experiencing high CPU usage. Note that if the following information instructs you to turn off a feature that you require, disregard that part of the instructions.

  • Use hardware acceleration wherever possible to offload tasks from the CPU. Offloading tasks, such as encryption, frees up the CPU for other tasks.
  • Avoid the use of GUI widgets that require computing cycles, such as the Top Sessions widget. These widgets constantly pol the system for information, which uses CPU and other resources.
  • Schedule antivirus, IPS, and firmware updates during off peak hours. These updates don't usually consume CPU resources but they can disrupt normal operation.
  • Check the log levels and which events are being logged. This is the severity of the messages that are recorded. Consider going up one level to reduce the amount of logging. Also, if there are events you don't need to monitor, remove them from the list.
  • Log to FortiCloud instead of logging to memory or disk. Logging to memory quickly uses up resources and logging to local disk impacts overall performance and reduces the lifetime of the unit. Fortinet recommends that you log to FortiCloud because it doesn’t use much CPU.
  • If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find free space and organize the files.
  • If packet logging is enabled on the FortiGate, consider disabling it. When packet logging is enable, it records every packet that comes through that policy.
  • Halt all sniffers and traces.
  • Ensure the FortiGate isn't scanning traffic twice. If traffic enters the FortiGate on one interface, goes out another, and then comes back in again, that traffic doesn't need to be rescanned. Doing so is a waste of resources. However, ensure that traffic truly is being scanned once.
  • Reduce the session timers to close unused sessions faster. Enter the following CLI commands, which reduce the default values. Note that, by default, the system adds 10 seconds to tcp-timewait.

    config system global

    set tcp-halfclose-timer 30

    set tcp-halfopen-timer 30

    set tcp-timewait-timer 0

    set udp-idle-timer 60

    end

  • In System > Feature Visibility, only enable features that you need.

SNMP monitoring

When CPU usage is under control, use SNMP to monitor CPU usage. Alternatively, use logging to record CPU and memory usage every 5 minutes.

Once things are back to normal, you should set up a warning system that sends you alerts when CPU resources are used excessively. A common method to do this is using SNMP. SNMP monitors many values in FortiOS and allows you to set high water marks that generate events. You run an application on your computer to watch for and record these events. Go to System > SNMP to enable and configure an SNMP community. If this method is too complicated, you can use the System Resources widget to record CPU usage. However, this method only records problems as they happen and won't send you alerts for problems.

How to check modem status

If the modem doesn't work properly or a FortiGate doesn't detect the modem, you can use the following diagnostic command to help you troubleshoot issues with the modem:

diagnose sys modem {cmd | com | detect | history | external-modem | query| reset}

You should always run the following diagnose command after you insert the USB modem into the FortiGate:

diagnose sys modem detect

You can view the modem configuration, using the get system modem command. You can also view the modem’s vendor and the custom product identification number from the information output from the get system modem command.

When there are connectivity issues, use the following to help you resolve them:

  • diagnose debug enable – activates the debug on the console
  • diagnose debug application modemd – dumps communication between the modem and the unit.
  • diagnose debug application ppp – dumps the PPP negotiating messages.
  • execute modem dial – displays modem debug output.

The modem diagnose output shouldn't contain errors on the way to initializing. You should also verify the number that is used to dial into your ISP.

How to run ping and traceroute

Ping and traceroute are useful tools in network troubleshooting. Alone, either one can determine network connectivity between two points. However, ping can be used to generate simple network traffic that you can view using diagnose commands on the FortiGate. This combination can be very powerful when you're trying to locate network problems.

In addition to their normal uses, ping and traceroute can tell you if your computer or network device has access to a domain name server (DNS). While both tools can use IP addresses alone, they can also use domain names for devices. This is an added troubleshooting feature that can be useful in determining why particular services, such as email or web browsing, may not work properly.

caution icon

If ping doesn't work, it may be disabled on at least one of the interface settings, and security policies for that interface.

Both ping and traceroute require particular ports to be open on firewalls, or else they can't function. Since you typically use these tools to troubleshoot, you can allow them in the security policies and on interfaces only when you need them. Otherwise, keep the ports disabled for added security.

Ping

The ping command sends a very small packet to a destination, and waits for a response. The response has a timer that expires when the destination is unreachable.

Ping is part of layer 3 on the OSI Networking Model. Ping sends Internet Control Message Protocol (ICMP) “echo request” packets to the destination, and listens for “echo response” packets in reply. However, many public networks block ICMP packets because ping can be used in a denial of service (DoS) attack (such as Ping of Death or a smurf attack), or by an attacker to find active locations on the network. By default, FortiGate units have ping enabled while broadcast-forward is disabled on the external interface.

What ping can tell you

Beyond the basic connectivity information, ping can tell you the amount of packet loss (if any), how long it takes the packet to make the round trip, and the variation in that time from packet to packet.

If there is some packet loss detected, you should investigate the following:

  • Possible ECMP, split horizon, or network loops.
  • Cabling, to ensure no loose connections.
  • Verify which security policy was used (use the packet count column on the Policy & Objects > IPv4 Policy or Policy & Objects > IPv6 Policy page).

If there is total packet loss, you should investigate the following:

  1. Ensure cabling is correct, and all equipment between the two locations is accounted for.
  2. Ensure all IP addresses and routing information along the route is configured as expected.
  3. Ensure all firewalls, including FortiGate security policies allow PING to pass through.
How to use ping

Ping syntax is the same for nearly every type of system on a network.

To ping from a FortiGate unit
  1. Connect to the CLI either through telnet or through the CLI widget on the Dashboard .
  2. Enter exec ping 10.11.101.101 to send 5 ping packets to the destination IP address. There are no options for this command.

    Sample output:

    Head_Office_620b # exec ping 10.11.101.101

    PING 10.11.101.101 (10.11.101.101): 56 data bytes

    64 bytes from 10.11.101.101: icmp_seq=0 ttl=255 time=0.3 ms

    64 bytes from 10.11.101.101: icmp_seq=1 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=2 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=3 ttl=255 time=0.2 ms

    64 bytes from 10.11.101.101: icmp_seq=4 ttl=255 time=0.2 ms

    --- 10.11.101.101 ping statistics ---

    5 packets transmitted, 5 packets received, 0% packet loss

    round-trip min/avg/max = 0.2/0.2/0.3 ms

To ping from a MicroSoft Windows PC
  1. Open a command window.
  1. Enter ping 10.11.101.100 to ping the default internal interface of the FortiGate with four packets.

Other options include:

  • -t to send packets until you press “Ctrl+C”
  • -a to resolve addresses to domain names where possible
  • -n X to send X ping packets and stop

Sample output:

C:\>ping 10.11.101.101

Pinging 10.11.101.101 with 32 bytes of data:

Reply from 10.11.101.101: bytes=32 time=10ms TTL=255

Reply from 10.11.101.101: bytes=32 time<1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Ping statistics for 10.11.101.101:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 10ms, Average = 3ms

To ping from a Linux PC
  1. Go to a shell prompt.
  2. Enter “ping 10.11.101.101”.

Traceroute

Where ping will only tell you if it reached its destination and returned successfully, traceroute shows each step of the journey to its destination and how long each step takes. If ping finds an outage between two points, you can use traceroute to locate exactly where the problem is.

What is traceroute

Traceroute works by sending ICMP packets to test each hop along the route. It sends three packets, and then increases the time to live (TTL) setting by one each time. This effectively allows the packets to go one hop farther along the route. This is the reason why most traceroute commands display their maximum hop count before they start tracing the route, which is the maximum number of steps it takes before it declares the destination unreachable. Also, the TTL setting may result in steps along the route timing out due to slow responses. There are many possible reasons for this to occur.

By default, traceroute uses UDP datagrams with destination ports numbered from 33434 to 33534. The traceroute utility may also offer the option to select use of ICMP echo request (type 8) instead, which the Windows tracert utility uses. If you must, allow both protocols inbound through the FortiGate security policies (UDP with ports from 33434 to 33534 and ICMP type 8).

You can also use the packet count column of the Policy & Objects > IPv4 Policy (or, Policy & Objects > IPv6 Policy, if applicable) page to track traceroute packets. This allows you to verify the connection and confirm which security policy the traceroute packets are using.

What traceroute can tell you

Ping and traceroute have similar functions, which is to verify connectivity between two points. The big difference is that traceroute shows you each step of the way, where ping doesn't. Also, ping and traceroute use different protocols and ports, so one may succeed where the other fails.

You can verify your DNS connection using traceroute. If you enter an FQDN instead of an IP address for the traceroute, DNS tries to resolve that domain name. If the name isn't resolved, you have DNS issues.

How to use traceroute

The traceroute command varies slightly between operating systems. Note that in MicroSoft Windows, the command name is shortened to “tracert”. Also, your output lists different domain names and IP addresses along your route.

To use traceroute on a MicroSoft Windows PC
  1. Open a command window.
  1. Enter tracert fortinet.com to trace the route from the PC to the Fortinet web site.

Sample output:

C:\>tracert fortinet.com

Tracing route to fortinet.com [208.70.202.225]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.20.120.2

2 66 ms 24 ms 31 ms 209-87-254-xxx.storm.ca [209.87.254.221]

3 52 ms 22 ms 18 ms core-2-g0-0-1104.storm.ca [209.87.239.129]

4 43 ms 36 ms 27 ms core-3-g0-0-1185.storm.ca [209.87.239.222]

5 46 ms 21 ms 16 ms te3-x.1156.mpd01.cogentco.com [38.104.158.69]

6 25 ms 45 ms 53 ms te8-7.mpd01.cogentco.com [154.54.27.249]

7 89 ms 70 ms 36 ms te3-x.mpd01.cogentco.com [154.54.6.206]

8 55 ms 77 ms 58 ms sl-st30-chi-.sprintlink.net [144.232.9.69]

9 53 ms 58 ms 46 ms sl-0-3-3-x.sprintlink.net [144.232.19.181]

10 82 ms 90 ms 75 ms sl-x-12-0-1.sprintlink.net [144.232.20.61]

11 122 ms 123 ms 132 ms sl-0-x-0-3.sprintlink.net [144.232.18.150]

12 129 ms 119 ms 139 ms 144.232.20.7

13 172 ms 164 ms 243 ms sl-321313-0.sprintlink.net [144.223.243.58]

14 99 ms 94 ms 93 ms 203.78.181.18

15 108 ms 102 ms 89 ms 203.78.176.2

16 98 ms 95 ms 97 ms 208.70.202.225

The first column on the left is the hop count, which can't exceed 30 hops. When that number is reached, the traceroute ends.

The second, third, and fourth columns display how much time each of the three packets takes to reach this stage of the route. These values are in milliseconds and normally vary quite a bit. Typically a value of <1ms indicates a local connection.

The fifth column (farthest to the right) shows the domain name of the device and its IP address, or possibly only the IP address.

To perform a traceroute on a Linux PC
  1. Go to a command line prompt.
  2. Enter “traceroute fortinet.com”.

The Linux traceroute output is very similar to the MicroSoft Windows tracert output.

To trace a route from a FortiGate to a destination IP address - CLI

# execute traceroute www.fortinet.com

traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets

1 172.20.120.2 0.637 ms 0.653 ms 0.279 ms

2 209.87.254.221 <static-209-87-254-221.storm.ca> 2.448 ms 2.519 ms 2.458 ms

3 209.87.239.129 <core-2-g0-2.storm.ca> 2.917 ms 2.828 ms 9.324 ms

4 209.87.239.199 <core-3-bdi1739.storm.ca> 13.248 ms 12.401 ms 13.009 ms

5 216.66.41.113 <v502.core1.tor1.he.net> 17.181 ms 12.422 ms 12.268 ms

6 184.105.80.9 <100ge1-2.core1.nyc4.he.net> 21.355 ms 21.518 ms 21.597 ms

7 198.32.118.41 <ny-paix-gni.twgate.net> 83.297 ms 84.416 ms 83.782 ms

8 203.160.228.217 <217-228-160-203.TWGATE-IP.twgate.net> 82.579 ms 82.187 ms 82.066 ms

9 203.160.228.229 <229-228-160-203.TWGATE-IP.twgate.net> 82.055 ms 82.455 ms 81.808 ms

10 203.78.181.2 82.262 ms 81.572 ms 82.015 ms

11 203.78.186.70 83.283 ms 83.243 ms 83.293 ms

12 66.171.127.177 84.030 ms 84.229 ms 83.550 ms

13 66.171.121.34 <www.fortinet.com> 84.023 ms 83.903 ms 84.032 ms

14 66.171.121.34 <www.fortinet.com> 83.874 ms 84.084 ms 83.810 ms

How to check the logs

You might forget his step in troubleshooting, but it's an important one. Logging records the traffic that passes through the FortiGate to your network and what action the FortiGate takes when it scans the traffic. This information record is called a log message.

When you first configure FortiOS, log as much information as you can.If the logs that the FortiGatte generates are too large, you can turn off or scale back the logging for features that you're not using.

As with most troubleshooting steps, before you can determine if the logs indicate a problem, you need to know what logs result from normal operation. Without a baseline it's difficult to troubleshoot.

When you troubleshoot with log files:

  • Compare current logs to a recorded baseline of normal operation.
  • If you need to, increase the level of logging (such as from Warning to Information) to obtain more information.

When increasing logging levels, ensure that you configure email alerts and select both disk usage and log quota. This ensures that you'll be notified if the increas in logging causes problems. Configure log settings by going to Log & Report > Log Settings.

Determine the activities that generate the most log entries:

  • Check all logs to ensure important information isn't overlooked.
  • Filter or order log entries based on different fields, such as level, service, or IP address, to look for patterns that may indicate a specific problem, such as frequent blocked connections on a specific port for all IP addresses.

Logs will help identify and locate any problems, but they will not solve the problems. The purpose of logs is to speed up your problem solving and save you time and effort.

For more information about logging and log reports, see Logging and reporting.

How to verify the contents of the routing table (in NAT mode)

When a FortiGate has limited or no connectivity, a good place to look for information is the routing table.

The routing table is where all the currently used routes are stored for both static and dynamic protocols. If a route is in the routing table, it saves time and resources that you would spend performing a lookup. If a route is not used for a while and a new route needs to be added, the oldest least used route is bumped if the routing table is full. This ensures that the most recently used routes stay in the table. If your FortiGate is in transparent mode, you can't perform this step.

If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table, including local subnets, default routes, specific static routes, and dynamic routing protocols.

To check the routing table in the FortiGate GUI, use the Routing Monitor by going to Monitor > Routing Monitor.

In the CLI, use the command get router info routing-table all.

Sample output:

FGT# get router info routing-table all

Codes:

K - kernel, C - connected, S - static, R - RIP, B - BGP

O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

* - candidate default

S* 0.0.0.0/0 [10/0] via 172.20.120.2, wan1

C 10.31.101.0/24 is directly connected, internal

C 172.20.120.0/24 is directly connected, wan1

How to verify the correct route is being used

If you have more than one default route and want to make sure that traffic is flowing as expected and through the right route, you can run a trace route from a machine in the local area network (LAN). This shows you the first hop that the traffic goes through.

Sample output:

C:\>tracert www.fortinet.com

Tracing route to www.fortinet.com [66.171.121.34]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.10.1.99

2 1 ms <1 ms <1 ms 172.20.120.2

3 3 ms 3 ms 3 ms static-209-87-254-221.storm.ca [209.87.254.221]

4 3 ms 3 ms 3 ms core-2-g0-2.storm.ca [209.87.239.129]

5 13 ms 13 ms 13 ms core-3-bdi1739.storm.ca [209.87.239.199]

6 12 ms 19 ms 11 ms v502.core1.tor1.he.net [216.66.41.113]

7 22 ms 22 ms 21 ms 100ge1-2.core1.nyc4.he.net [184.105.80.9]

8 84 ms 84 ms 84 ms ny-paix-gni.twgate.net [198.32.118.41]

9 82 ms 84 ms 82 ms 217-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.217]

10 82 ms 81 ms 82 ms 229-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.229]

11 82 ms 82 ms 82 ms 203.78.181.2

12 84 ms 83 ms 83 ms 203.78.186.70

13 84 ms * 85 ms 66.171.127.177

14 84 ms 84 ms 84 ms fortinet.com [66.171.121.34]

15 84 ms 84 ms 83 ms fortinet.com [66.171.121.34]

Trace complete.

In this scenario, the first hop contains the IP address 10.10.1.99, which is the internal interface of the FortiGate. The second hop contains the IP address 172.20.120.2, to which the wan1 interface of the FortiGate is connected, so we can conclude that the route through wan1 interface is being used for this traffic.

You can also see the route taken for each session by debugging the packet flow in the CLI. For more information, see How to debug the packet flow.

How to verify the correct firewall policy is being used

If you have more than one firewall policy, use the count column to check which policy is being used, the count must show traffic increasing. To do so, go to the Policy & Objects > IPv4 Policy or Policy & Objects > IPv6 Policy page.

Also, debugging the packet flow in the CLI shows the policy ID that's allowing the traffic. For more information about debugging the packet flow, see How to debug the packet flow.

How to check the bridging information in transparent mode

When the FortiGate is set to transparent mode, it acts like a bridge and sends all incoming traffic out on the other interfaces. The bridge is between interfaces on the FortiGate.

Each bridge that's listed is a link between interfaces. Where traffic is flowing between interfaces, you can expect to find bridges listed. If you're having connectivity issues, and there are no bridges listed, that is a likely cause. Check for the MAC address of the interface or device in question.

How to check the bridging information

To list the existing bridge instances on the FortiGate, use the following command:

diagnose netlink brctl list

Sample output:

#diagnose netlink brctl list

list bridge information

1. root.b fdb: size=256 used=6 num=7 depth=2 simple=no

Total 1 bridges

How to display forwarding domain information

You can use forwarding domains, or collision domains, in routing to limit where packets are forwarded on the network. Layer 2 broadcasts are limited to the same group. By default, all interfaces are in group 0. For example, if the FortiGate has 12 interfaces, only two may be in the same forwarding domain, which limits packets that are broadcast to those two interfaces. This reduces traffic on the rest of the network.

Collision domains prevent the forwarding of ARP packets to all VLANs on an interface. Without collision domains, duplicate MAC addresses on VLANs may cause ARP packets to be duplicated. Duplicate ARP packets can cause some switches to reset. It's important to know what interfaces are part of which forwarding domains because this determines which interfaces can communicate with each other.

To manually configure forwarding domains in transparent mode, use the following CLI command:

config system interface

edit <interface_name>

set forward-domain <integer>

end

To display the information for forward domains, use the following command:

diagnose netlink brctl domain <name> <id>

where <name> is the name of the forwarding domain to display and <id> is the domain ID.

Sample output

diagnose netlink brctl domain ione 101

show bridge root.b ione forward domain.

id=101 dev=trunk_1 6

To list the existing bridge MAC table, use the following command:

diagnose netlink brctl name host <name>

Sample output

show bridge control interface root.b host.

fdb: size=256, used=6, num=7, depth=2, simple=no

Bridge root.b host table

port no

device

devname

mac addr

ttl

attributes

2

7

wan2

02:09:0f:78:69:00

0

Local Static

5

6

vlan_1

02:09:0f:78:69:01

0

Local Static

3

8

dmz

02:09:0f:78:69:01

0

Local Static

4

9

internal

02:09:0f:78:69:02

0

Local Static

3

8

dmz

00:80:c8:39:87:5a

194

4

9

internal

02:09:0f:78:67:68

8

1

3

wan1

00:09:0f:78:69:fe

0

Local Static

To list the existing bridge port list, use the following command:

diagnose netlink brctl name port <name>

Sample output:

show bridge root.b data port.

trunk_1 peer_dev=0

internal peer_dev=0

dmz peer_dev=0

wan2 peer_dev=0

wan1 peer_dev=0

How to examine the firewall session list

The firewall session list displays all sessions that the FortiGate has open. You can see if there are strange patterns, such as no sessions apart from the internal network, or all sessions are only to one IP address.

When you examine the firewall session list in the CLI, you can use filters to reduce the output. In the GUI, the filters are part of the interface.

To examine the firewall session list - GUI

Go to FortiView > All Sessions.

To examine the firewall session list - CLI

When you examine the firewall session list, there may be too many sessions to display. In this case, it's necessary to limit or filter the sessions displayed by source or destination address, or NAT'd address or port. If you want to filter by more than one of these, you need to enter a separate line for each value.

The following example shows filtering the session list based on a source address of 10.11.101.112:

FGT# diagnose sys session filter src 10.11.101.112

FGT# diagnose sys session list

The following example shows filtering the session list based on a destination address of 172.20.120.222.

FGT# diagnose sys session filter dst 172.20.120.222

FGT# diagnose sys session list

To clear all sessions corresponding to a filter - CLI

FGT# diagnose sys session filter dst 172.20.120.222

FGT# diagnose sys session clear

Check source NAT information

When you troubleshoot connections, remember NAT. NAT is especially important if you're troubleshooting from the remote end of the connection outside the firewall. On the FortiView > All Sessions list, pay attention to NAT Source, and NAT Source Port. These columns display the IP and port values after NAT has been applied.

Checking the NAT values can help you to ensure that they are the values you expect, and to ensure the remote end of the sessions can see the expected IP address and port number.

When you display the session list in the CLI, you can match the NAT'd source address (nsrc) and port (nport). This can be useful if multiple internal IP addresses are NAT'd to a common external-facing source IP address.

FGT# diagnose sys session filter nsrc 172.20.120.122

FGT# diagnose sys session filter nport 8888

FGT# diagnose sys session list

How to check wireless information

Wireless connections, stations, and interfaces have different issues than other physical interfaces.

Troubleshooting station connection issue

To check whether a station entry is created on access control, use the following command:

FG600B3909600253 # diagnose wireless-controller wlac -d sta

* vf=0 wtp=70 rId=2 wlan=open ip=0.0.0.0 mac=00:09:0f:db:c4:03 rssi=0 idle=148 bw=0 use=2

vf=0 wtp=70 rId=2 wlan=open ip=172.30.32.122 mac=00:25:9c:e0:47:88 rssi=-40 idle=0 bw=9 use=2

Enable diagnostics for a particular station

This example uses the station MAC address to find where it's failing:

FG600B3909600253 # diagnose wireless-controller wlac sta_filter 00:25:9c:e0:47:88 1

Set filter sta 00:25:9c:e0:47:88 level 1

FG600B3909600253 # 71419.245 <ih> IEEE 802.11 mgmt::disassoc <== 00:25:9c:e0:47:88 vap open rId 1 wId 0 00:09:0f:db:c4:03

71419.246 <dc> STA del 00:25:9c:e0:47:88 vap open rId 1 wId 0

71419.246 <cc> STA_CFG_REQ(34) sta 00:25:9c:e0:47:88 del ==> ws (0-192.168.35.1:5246) rId 1 wId 0

71419.246 <cc> STA del 00:25:9c:e0:47:88 vap open ws (0-192.168.35.1:5246) rId 1 wId 0 00:09:0f:db:c4:03 sec open reason I2C_STA_DEL

71419.247 <cc> STA_CFG_RESP(34) 00:25:9c:e0:47:88 <== ws (0-192.168.35.1:5246) rc 0 (Success).

How to verify connectivity to FortiGuard

You can verify connectivity to FortiGuard in the Licenses widget on the FortiGate Dashboard. When your FortiGate is connected to FortiGuard, a green check mark appears for available FortiGuard services.

To verify connectivity from the CLI, use the execute ping service.fortiguard.net and execute ping update.fortiguard.net commands.

Sample output:

FG100D# execute ping service.fortiguard.net

PING guard.fortinet.net (208.91.112.196): 56 data bytes

64 bytes from 208.91.112.196: icmp_seq=0 ttl=51 time=61.0 ms

64 bytes from 208.91.112.196: icmp_seq=1 ttl=51 time=60.0 ms

64 bytes from 208.91.112.196: icmp_seq=2 ttl=51 time=59.6 ms

64 bytes from 208.91.112.196: icmp_seq=3 ttl=51 time=58.9 ms

64 bytes from 208.91.112.196: icmp_seq=4 ttl=51 time=59.2 ms

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 58.9/59.7/61.0 ms

FG100D# execute ping update.fortiguard.net

PING fds1.fortinet.com (208.91.112.68): 56 data bytes

64 bytes from 208.91.112.68: icmp_seq=0 ttl=53 time=62.0 ms

64 bytes from 208.91.112.68: icmp_seq=1 ttl=53 time=61.8 ms

64 bytes from 208.91.112.68: icmp_seq=2 ttl=53 time=61.3 ms

64 bytes from 208.91.112.68: icmp_seq=3 ttl=53 time=61.9 ms

64 bytes from 208.91.112.68: icmp_seq=4 ttl=53 time=61.8 ms

How to perform a sniffer trace (CLI and packet capture)

When you troubleshoot networks and routing in particular, it helps to look inside the headers of packets to determine if they're traveling along the route that you expect them to take. Packet sniffing can also be called a network tap, packet capture, or logic analyzing.

caution icon

If your FortiGate has NP2, NP4, or NP6 interfaces that are offloading traffic, this will change the sniffer trace. Before you perform a trace on any NP2, NP4, or NP6 interfaces, you should disable offloading on those interfaces.

How do you sniff packets

Before you start sniffing packets on the CLI, you should prepare to capture the output to a file. A large amount of data may scroll by and you won't be able to see it without first saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate CLI. Once the packet sniffing count is reached, you can end the session and analyze the output in the file.

The general form of the internal FortiOS packet sniffer command is:

diagnose sniffer packet <interface_name> <‘filter’> <verbose> <count> <tsformat>

To stop the sniffer, type CTRL+C.

<interface_name>

The name of the interface to sniff, such as “port1” or “internal”. This can also be “any” to sniff all interfaces.

<‘filter’>

What to look for in the information the sniffer reads. “none” indicates no filtering, and all packets are displayed as the other arguments indicate.

The filter must be inside single quotes (‘).

<verbose>

The level of verbosity as one of:

1 - print header of packets
2 - print header and data from IP of packets
3 - print header and data from Ethernet of packets
4 - print header of packets with interface name

<count>

The number of packets the sniffer reads before stopping. If you don't put a number here, the sniffer will run until you stop it with <CTRL+C>.

<tsformat>

The format of timestamp.
  • a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms
  • l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms
  • otherwise: relative to the start of sniffing, ss.ms

For a simple sniffing example, enter the CLI command diagnose sniffer packet port1 none 1 3. This displays the next three packets on the port1 interface using no filtering, and verbose level 1. At this verbosity level, you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.

In the output below, port 443 indicates these are HTTPS packets and that 172.20.120.17 is both sending and receiving traffic.

Head_Office_620b # diagnose sniffer packet port1 none 1 3

interfaces=[port1]

filters=[none]

0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757

0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808

0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933

For a more advanced example of packet sniffing, the following commands will report packets on any interface that are traveling between a computer with the host name of “PC1” and a computer with the host name of “PC2”. With verbosity 4 and above, the sniffer trace displays the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C.

FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4

or

FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4

The following CLI command for a sniffer includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for example, PC2 may be down and not responding to the FortiGate ARP requests).

FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4

How do you use packet capture

To use packet capture, the FortiGate must have a disk. You can enable the capture-packet in the firewall policy, using the following CLI commands:

config firewall policy

edit <id>

set capture-packet enable

end

To configure packet capture filters, go to Network > Packet Capture.

When you add a packet capture filter, enter the following information and select OK.

Interface

Select the interface to sniff from the drop-down menu.

You must select one interface. You can't change the interface without deleting the filter and creating a new one, unlike the other fields.

Max Packets to Save

Enter the number of packets to capture before the filter stops.

This number can't be zero. You can halt the capturing before this number is reached.

Enable Filters

Select this option to specify filter fields

Host(s)

Enter the IP address of one or more hosts.

Separate multiple hosts with commas. To enter a range, use a dash without spaces, for example 172.16.1.5-172.16.1.15, or enter a subnet.

Port(s)

Enter one or more ports to capture on the selected interface.

Separate multiple ports with commas. To enter a range, use a dash without spaces, for example 88-90

VLAN(s)

Enter one or more VLANs (if any). Separate multiple VLANs with commas.

Protocol

Enter one or more protocols. Separate multiple protocols with commas. To enter a range, use a dash without spaces, for example 1-6, 17, 21-25.

Include IPv6 Packets

Select this option if you're troubleshooting IPv6 networking, or if your network uses IPv6. Otherwise, leave it disabled.

Include Non-IP Packets

The protocols in the list are all IP based except for ICMP (ping). To capture non-IP based packets, select this feature. Examples of non-IP packets include IPsec, IGMP, ARP, and ICMP.

If you select a filter, you have the option to start and stop packet capture in the edit window, or download the captured packets. You can also see the filter status and the number of packets captured.

You can select the filter and start capturing packets. When the filter is running, the number of captured packets increases until it reaches the Max Packet Count or you stop it. When the filter is running, you can't download the output file.

When the packet capture is complete, you can download the *.pcap file. You must use a third party application, such as Wireshark, to read *,pcap files. This tool provides you with extensive analytics and the full contents of the packets that were captured.

To start, stop, or resume packet capture, use the symbols on the screen. These symbols are the same as those used for audio or video playback. Hover over the symbol to reveal explanatory text. Similarly, to download the *.pcap file, use the download symbol on the screen.

For more information on troubleshooting with packet capture and packet sniffing, see Packet sniffing and packet capture.

How to debug the packet flow

Traffic should come in and leave the FortiGate. If you determine that network traffic isn't entering and leaving the FortiGate as expected, debug the packet flow.

You can only perform debugging using CLI commands. Debugging the packet flow requires that you enter a number of debug commands as each one configures part of the debug action, with the final command starting the debug.

caution icon

If your FortiGate has FortiASIC NP4 or NP6 interface pairs that are offloading traffic, this changes the packet flow. Before you perform the debug on any NP4 or NP6 interfaces, you should disable offloading on those interfaces. Enter diagnose npu <interface pair> fastpath disable, where interface pair is np4, np6, np4lite, or np6lite.

The following configuration assumes that PC1 is connected to the internal interface of the FortiGate and has an IP address of 10.11.101.200. PC1 is the host name of the computer.

To debug the packet flow in the CLI, enter the following commands:

FGT# diagnose debug disable

FGT# diagnose debug flow filter add <PC1>

FGT# diagnose debug flow show console enable

FGT# diagnose debug flow show function-name enable

FGT# diagnose debug flow trace start 100

FGT# diagnose debug enable

The start 100 argument in the above list of commands limits the output to 100 packets from the flow. This is useful to look at the flow without flooding your log or displaying too much information.

To stop all other debug activities, enter the command:

FGT# diagnose debug flow trace stop

The following is an example of debug flow output for traffic that has no matching security policy, and is in turn blocked by the FortiGate unit. The denied message indicates that the traffic was blocked.

id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg="vd-root received a packet(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3."

id=20085 trace_id=319 func=resolve_ip_tuple line=2924 msg="allocate a new session-013004ac"

id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-192.168.150.129 via port1"

id=20085 trace_id=319 func=fw_forward_handler line=248 msg=" Denied by forward policy check"