Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:


Table of Contents

Administration Guide

Checking CPU information&Issues

  1. Check CPU information

    FortiWeb# diagnose hardware cpu list    #show the detail info for all CPU/vCPU

    FortiWeb-AWS-M01 # diagnose hardware cpu list

    processor    : 0

    vendor_id    : GenuineIntel

    cpu family    : 6

    model        : 79

    model name    : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz

    stepping    : 1

    microcode    : 0xb000038

    cpu MHz    : 2300.049

    cache size    : 46080 KB

    physical id    : 0

    siblings    : 2

    core id        : 0

    cpu cores    : 2

    apicid        : 0

    initial apicid    : 0

    fpu        : yes

    fpu_exception    : yes

    cpuid level    : 13

    wp        : yes

  2. CPU & processor numbers

    /# grep "cpu cores" /proc/cpuinfo | uniq    #Check physical CPU cores

    cpu cores    : 16

    /# cat /proc/cpuinfo |grep "processor" | sort -u | wc -l    #Check logical CPU cores when hyperthread is enabled

    32

  3. Check which daemon or process consuming the most CPU usage

    To determine if high load is frequently a problem, you can display the average load level by using these CLI commands:

    FortiWeb # get system performance

    CPU states:    5% used, 95% idle

    Memory states: 29% used

    Up:            9 days,  12 hours,  52 minutes.

    top

    Use the CLI to view the per-CPU/core process load level and a list of the most system-intensive processes. This may show processes that are consuming resources unusually.

    While the command is running, you can press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage.

    FortiWeb# diagnose system top 10

    Mem: 4867300K used, 126120392K free, 16536K shrd, 10792K buff, 117620K cached

    CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq

    Load average: 1.71 1.55 1.49 2/953 52110

      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND

    6262     1 root     S    9582m  7.4  31  0.3 /bin/proxyd

    6264     1 root     S   6539m  5.1  29  0.0 /bin/bot_daemon

    6273     1 root     S    2498m  1.9  21  0.0 /bin/garbage -o standalone

    6316  6238 root     S    2098m  1.6  24  0.0 /bin/mysqld --defaults-file=/data/e

    6251     1 root     S     803m  0.6  10  0.0 /bin/monitord

    6269     1 root     S     411m  0.3  21  0.0 /bin/sandboxd

    6271     1 root     S     400m  0.3  43  0.0 /bin/shibd -F -f -p /var/run/shibd.

    6287     1 root     S     256m  0.2  59  0.0 /bin/statusd

    The above command generates a report of processes every 10 seconds. The report provides the process names, their process ID (pid), status, CPU usage, and memory usage.

    The report continues to refresh and display in the CLI until you press q (quit).

    perf top

    The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space.

    FortiWeb# diagnose system perf    # or “perf top” in backend shell

    FortiWeb# diagnose system perf

       PerfTop:   69182 irqs/sec  kernel:96.4%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 64 CPUs)

    --------------------------------------------------------------------------------

        13.50%  [kernel]          [k] find_busiest_group

         3.20% [kernel]          [k] idle_cpu

         3.15%  [kernel]          [k] _raw_spin_lock

         2.44%  [kernel]          [k] __schedule

         2.42%  [kernel]          [k] rcu_sched_clock_irq

         2.07%  [kernel]          [k] _raw_spin_trylock

         1.95%  [kernel]          [k] native_irq_return_iret

  4. Kill processes

    Once you locate an offending PID from “diagnose system top”, you may want to terminate it. For example, in a test environment or when you fail to locate the cause when access to a server-policy always fails, you may try to kill proxyd or dnsproxyd.

    Under normal conditions, killing a process is not recommended.

    diagnose system kill 9 <pid>

    or

    Fn kill 9 <pid>

  5. Check if high CPU usage is caused by heavy traffic load

    Heavy traffic loads can cause sustained high CPU or RAM usage. If this is unusual, no action may be required, unless you are being subject to a DoS attack. Sustained heavy traffic load may indicate that you need a more powerful model of FortiWeb.

    You can check traffic load via GUI or debug logs in several ways:

    1) Monitor Total Connection per Second, Total Connections and Total HTTP Transaction, Throughput on the GUI dashboard.

       Total Connection per Second, Total Connections (also Concurrent Connection) are displayed directly in the widgets “System Resource” and “Policy Sessions”, whereas the current HTTP transaction per second is not displayed directly on GUI. You need to enable/add a widget named “HTTP Transactions” and calculate the TPS by dividing the total transaction in 5 minutes.

        Taking the screenshot below for example, the concurrent connection is 100000 and there are no new connections established per second, whereas there are nearly 6000000 transactions in the past 5 minutes - equal to 20000 transactions per second (TPS), so this might be the main cause why CPU usage reaches 10%.

    2) Check TCP connections in TIME_WAIT status

        TIME_WAIT connections cannot be displayed in dashboard widgets but also consume system connection/memory resources. You can also check connection in backend shell:

    /# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r

    199101 ESTABLISHED          #Concurrent connections

        251 LISTEN

          7 TIME_WAIT

          1 established)

          1 Foreign

    3) Examine traffic history in the traffic log. Go to Logs&Report > Log Access > Traffic.

       If massive traffic logs are generated in a short period, it indicates heavy traffic load.

  6. Check if high CPU usage is caused by Attacks

    A prolonged denial of service (DoS) or brute-force login attack (to name just a few) can bring your web servers to a standstill, if your FortiWeb appliance is not configured for it.

    In the FortiWeb appliance's web UI, you can watch for attacks in two ways:

    1) Monitor current HTTP traffic on the dashboard. Go to System > Status > Status and examine the attack event history graph in the Policy Summary widget.

    2) Examine attack history in the traffic log. Go to Logs&Report > Log Access > Attack.

    Before attacks occur, use the FortiWeb appliance's rich feature set to configure attack defenses.

  7. Check system and debug logs to see CPU resource status:

    1) Log&Report > Event > Filter > Action > check-resource

        Log example:

    CPU usage too high,CPU usage is 95, process proxyd

    2) Analyze NMON files with all relevant statistics

        NMON files include CPU, Mem, I/O statistics, you can do a comprehensive analysis from these relevant information.

Checking CPU information&Issues

  1. Check CPU information

    FortiWeb# diagnose hardware cpu list    #show the detail info for all CPU/vCPU

    FortiWeb-AWS-M01 # diagnose hardware cpu list

    processor    : 0

    vendor_id    : GenuineIntel

    cpu family    : 6

    model        : 79

    model name    : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz

    stepping    : 1

    microcode    : 0xb000038

    cpu MHz    : 2300.049

    cache size    : 46080 KB

    physical id    : 0

    siblings    : 2

    core id        : 0

    cpu cores    : 2

    apicid        : 0

    initial apicid    : 0

    fpu        : yes

    fpu_exception    : yes

    cpuid level    : 13

    wp        : yes

  2. CPU & processor numbers

    /# grep "cpu cores" /proc/cpuinfo | uniq    #Check physical CPU cores

    cpu cores    : 16

    /# cat /proc/cpuinfo |grep "processor" | sort -u | wc -l    #Check logical CPU cores when hyperthread is enabled

    32

  3. Check which daemon or process consuming the most CPU usage

    To determine if high load is frequently a problem, you can display the average load level by using these CLI commands:

    FortiWeb # get system performance

    CPU states:    5% used, 95% idle

    Memory states: 29% used

    Up:            9 days,  12 hours,  52 minutes.

    top

    Use the CLI to view the per-CPU/core process load level and a list of the most system-intensive processes. This may show processes that are consuming resources unusually.

    While the command is running, you can press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage.

    FortiWeb# diagnose system top 10

    Mem: 4867300K used, 126120392K free, 16536K shrd, 10792K buff, 117620K cached

    CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq

    Load average: 1.71 1.55 1.49 2/953 52110

      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND

    6262     1 root     S    9582m  7.4  31  0.3 /bin/proxyd

    6264     1 root     S   6539m  5.1  29  0.0 /bin/bot_daemon

    6273     1 root     S    2498m  1.9  21  0.0 /bin/garbage -o standalone

    6316  6238 root     S    2098m  1.6  24  0.0 /bin/mysqld --defaults-file=/data/e

    6251     1 root     S     803m  0.6  10  0.0 /bin/monitord

    6269     1 root     S     411m  0.3  21  0.0 /bin/sandboxd

    6271     1 root     S     400m  0.3  43  0.0 /bin/shibd -F -f -p /var/run/shibd.

    6287     1 root     S     256m  0.2  59  0.0 /bin/statusd

    The above command generates a report of processes every 10 seconds. The report provides the process names, their process ID (pid), status, CPU usage, and memory usage.

    The report continues to refresh and display in the CLI until you press q (quit).

    perf top

    The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space.

    FortiWeb# diagnose system perf    # or “perf top” in backend shell

    FortiWeb# diagnose system perf

       PerfTop:   69182 irqs/sec  kernel:96.4%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 64 CPUs)

    --------------------------------------------------------------------------------

        13.50%  [kernel]          [k] find_busiest_group

         3.20% [kernel]          [k] idle_cpu

         3.15%  [kernel]          [k] _raw_spin_lock

         2.44%  [kernel]          [k] __schedule

         2.42%  [kernel]          [k] rcu_sched_clock_irq

         2.07%  [kernel]          [k] _raw_spin_trylock

         1.95%  [kernel]          [k] native_irq_return_iret

  4. Kill processes

    Once you locate an offending PID from “diagnose system top”, you may want to terminate it. For example, in a test environment or when you fail to locate the cause when access to a server-policy always fails, you may try to kill proxyd or dnsproxyd.

    Under normal conditions, killing a process is not recommended.

    diagnose system kill 9 <pid>

    or

    Fn kill 9 <pid>

  5. Check if high CPU usage is caused by heavy traffic load

    Heavy traffic loads can cause sustained high CPU or RAM usage. If this is unusual, no action may be required, unless you are being subject to a DoS attack. Sustained heavy traffic load may indicate that you need a more powerful model of FortiWeb.

    You can check traffic load via GUI or debug logs in several ways:

    1) Monitor Total Connection per Second, Total Connections and Total HTTP Transaction, Throughput on the GUI dashboard.

       Total Connection per Second, Total Connections (also Concurrent Connection) are displayed directly in the widgets “System Resource” and “Policy Sessions”, whereas the current HTTP transaction per second is not displayed directly on GUI. You need to enable/add a widget named “HTTP Transactions” and calculate the TPS by dividing the total transaction in 5 minutes.

        Taking the screenshot below for example, the concurrent connection is 100000 and there are no new connections established per second, whereas there are nearly 6000000 transactions in the past 5 minutes - equal to 20000 transactions per second (TPS), so this might be the main cause why CPU usage reaches 10%.

    2) Check TCP connections in TIME_WAIT status

        TIME_WAIT connections cannot be displayed in dashboard widgets but also consume system connection/memory resources. You can also check connection in backend shell:

    /# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r

    199101 ESTABLISHED          #Concurrent connections

        251 LISTEN

          7 TIME_WAIT

          1 established)

          1 Foreign

    3) Examine traffic history in the traffic log. Go to Logs&Report > Log Access > Traffic.

       If massive traffic logs are generated in a short period, it indicates heavy traffic load.

  6. Check if high CPU usage is caused by Attacks

    A prolonged denial of service (DoS) or brute-force login attack (to name just a few) can bring your web servers to a standstill, if your FortiWeb appliance is not configured for it.

    In the FortiWeb appliance's web UI, you can watch for attacks in two ways:

    1) Monitor current HTTP traffic on the dashboard. Go to System > Status > Status and examine the attack event history graph in the Policy Summary widget.

    2) Examine attack history in the traffic log. Go to Logs&Report > Log Access > Attack.

    Before attacks occur, use the FortiWeb appliance's rich feature set to configure attack defenses.

  7. Check system and debug logs to see CPU resource status:

    1) Log&Report > Event > Filter > Action > check-resource

        Log example:

    CPU usage too high,CPU usage is 95, process proxyd

    2) Analyze NMON files with all relevant statistics

        NMON files include CPU, Mem, I/O statistics, you can do a comprehensive analysis from these relevant information.