Fortinet black logo

Administration Guide

Server policy intermittently inaccessible

Server policy intermittently inaccessible

  1. Check if networking connection is stable:
    • Ping continuously from a remote client to see if any failures or long response time;

    • Ping the back-end server from FortiWeb to see if any failures or long response time;

    • Visit the back-end server continuously from a remote client to see if any failures or long response time;

    • Visit the back-end server from FortiWeb to see if any failures or long response time when accessing the server-policy from remote client fails.

  2. Check if back-end servers’ status in server-pool are stable:
    • If server health check is ON, check Event logs to confirm the health check down/up events;

    • If server health check is OFF, check the logs on the back-end server (Apache/Nginx logs or other monitor system) if possible;

    • Visit the back-end server continuously from FortiWeb to see if any failures or long response time from time to time or when the connectivity issue occurs.

      You can use curl on FortiWeb back-end shell to visit the back-end server, and check the response time.

      Samples:

      curl -o /dev/null -s -w %{time_total}\\n HTTP://<back-end server_IP>:<port>

      curl -v HTTPs://<domain/IP>/ -A "check_HTTP" -so /dev/null --resolve <domain>:<port>:<IP> -k -w %{time_namelookup}::%{time_connect}::%{time_starttransfer}::%{time_total}::%{speed_download}"\n"

      You can run a script on FortiWeb back-end shell (upload the script via System > Maintenance > Backup&Restore > GUI File Download/Upload > Upload and chmod to add the execute permission) to visit the back-end server periodically and record the return code&response time. However, it’s not recommended when traffic is heavy.

  3. Check if FortiWeb system has resource shortage;
    • Check FortiWeb event logs or other external monitor tools (if available) to confirm if there is any traffic burst when the issue occurs;

    • Check FortiWeb event logs to confirm if there is any high CPU or Memory usage when the issue occurs; (Log&Report > Event > Filter > Action > check-resource)

      Find logs like:

      CPU usage too high,CPU usage is 64, process cmdbsvr

    • Check if the connection/session/throughput status are overloaded when the issue occurs:

      When connection per second or Throughput reaches the performance bottleneck, you may find that the system CPU usage is near to 100%, thus consider changing to a higher-end platform. But sometimes if the connection number of TIME_WAIT becomes very large, it means too many sockets may be occupied and new TCP connections can hardly be established, thus causing performance decrease or new request failures. In this case, you may consider adding

      diagnose policy total-conn-psec list

      diagnose policy total-session list

      diagnose policy total-traffic HTTP list

      Or

      /# netstat -antp | grep ESTABLISH | wc -l

      19094

      /# netstat -antp | grep TIME_WAIT | wc -l

      38688

      /# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r

      56338 TIME_WAIT

      33940 ESTABLISHED

      427 SYN_SENT

      251 LISTEN

      221 FIN_WAIT1

      196 FIN_WAIT2

      5 SYN_RECV

      1 established)

      1 Foreign

  4. Solution

    To alleviate or solve such issue, you can increase the maximum number of connection by adding IP addresses used to connect to the back-end servers:

    1. Add secondary IPs to the interface connected to the back-end server:

      Secondary IPs are necessary for both below methods.

      FortiWeb # sho sys interface port1

      config system interface

      edit "port3"

      set type physical

      set ip 10.13.4.254/24

      set allowaccess ping ssh snmp HTTP HTTPs FortiWeb-manager

      config secondaryip

      edit 1

      set ip 10.13.4.253/24

      next

      edit 2

      set ip 10.13.4.252/24

      next

      end

      end

    2. Method 1: Enable ip-src-balance or ip6-src-balance to allow FortiWeb to connect to back-end servers using multiple IPv4 addresses configured as above.

      This is a global option that affects all server policies. FortiWeb uses round-robin algorithm between all primary&secondary IPs to distribute connections to back-end servers:

      config system network-option

      set ip-src-balance enable

      set ip6-src-balance enable

      End

      Method 2: Enable client-real-ip and add available secondary IPs configured above to IP ranges, then traffic matching the specific policy will connect to back-end servers using these secondary IPs added to IP/IP Range:

      To ensure FortiWeb receives the server's response, configure FortiWeb as the back-end server’s gateway.

      This option is available only for Reverse Proxy mode.

      FortiWeb # show server-policy policy

      config server-policy policy

      edit "Test_Policy"

      ...

      set client-real-ip enable

      set real-ip-addr 10.13.4.253

      next

      end

  5. Check if there are system crash/coredump when the issue occurs.
  6. Check related logs and captured files for further analysis.

Server policy intermittently inaccessible

  1. Check if networking connection is stable:
    • Ping continuously from a remote client to see if any failures or long response time;

    • Ping the back-end server from FortiWeb to see if any failures or long response time;

    • Visit the back-end server continuously from a remote client to see if any failures or long response time;

    • Visit the back-end server from FortiWeb to see if any failures or long response time when accessing the server-policy from remote client fails.

  2. Check if back-end servers’ status in server-pool are stable:
    • If server health check is ON, check Event logs to confirm the health check down/up events;

    • If server health check is OFF, check the logs on the back-end server (Apache/Nginx logs or other monitor system) if possible;

    • Visit the back-end server continuously from FortiWeb to see if any failures or long response time from time to time or when the connectivity issue occurs.

      You can use curl on FortiWeb back-end shell to visit the back-end server, and check the response time.

      Samples:

      curl -o /dev/null -s -w %{time_total}\\n HTTP://<back-end server_IP>:<port>

      curl -v HTTPs://<domain/IP>/ -A "check_HTTP" -so /dev/null --resolve <domain>:<port>:<IP> -k -w %{time_namelookup}::%{time_connect}::%{time_starttransfer}::%{time_total}::%{speed_download}"\n"

      You can run a script on FortiWeb back-end shell (upload the script via System > Maintenance > Backup&Restore > GUI File Download/Upload > Upload and chmod to add the execute permission) to visit the back-end server periodically and record the return code&response time. However, it’s not recommended when traffic is heavy.

  3. Check if FortiWeb system has resource shortage;
    • Check FortiWeb event logs or other external monitor tools (if available) to confirm if there is any traffic burst when the issue occurs;

    • Check FortiWeb event logs to confirm if there is any high CPU or Memory usage when the issue occurs; (Log&Report > Event > Filter > Action > check-resource)

      Find logs like:

      CPU usage too high,CPU usage is 64, process cmdbsvr

    • Check if the connection/session/throughput status are overloaded when the issue occurs:

      When connection per second or Throughput reaches the performance bottleneck, you may find that the system CPU usage is near to 100%, thus consider changing to a higher-end platform. But sometimes if the connection number of TIME_WAIT becomes very large, it means too many sockets may be occupied and new TCP connections can hardly be established, thus causing performance decrease or new request failures. In this case, you may consider adding

      diagnose policy total-conn-psec list

      diagnose policy total-session list

      diagnose policy total-traffic HTTP list

      Or

      /# netstat -antp | grep ESTABLISH | wc -l

      19094

      /# netstat -antp | grep TIME_WAIT | wc -l

      38688

      /# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r

      56338 TIME_WAIT

      33940 ESTABLISHED

      427 SYN_SENT

      251 LISTEN

      221 FIN_WAIT1

      196 FIN_WAIT2

      5 SYN_RECV

      1 established)

      1 Foreign

  4. Solution

    To alleviate or solve such issue, you can increase the maximum number of connection by adding IP addresses used to connect to the back-end servers:

    1. Add secondary IPs to the interface connected to the back-end server:

      Secondary IPs are necessary for both below methods.

      FortiWeb # sho sys interface port1

      config system interface

      edit "port3"

      set type physical

      set ip 10.13.4.254/24

      set allowaccess ping ssh snmp HTTP HTTPs FortiWeb-manager

      config secondaryip

      edit 1

      set ip 10.13.4.253/24

      next

      edit 2

      set ip 10.13.4.252/24

      next

      end

      end

    2. Method 1: Enable ip-src-balance or ip6-src-balance to allow FortiWeb to connect to back-end servers using multiple IPv4 addresses configured as above.

      This is a global option that affects all server policies. FortiWeb uses round-robin algorithm between all primary&secondary IPs to distribute connections to back-end servers:

      config system network-option

      set ip-src-balance enable

      set ip6-src-balance enable

      End

      Method 2: Enable client-real-ip and add available secondary IPs configured above to IP ranges, then traffic matching the specific policy will connect to back-end servers using these secondary IPs added to IP/IP Range:

      To ensure FortiWeb receives the server's response, configure FortiWeb as the back-end server’s gateway.

      This option is available only for Reverse Proxy mode.

      FortiWeb # show server-policy policy

      config server-policy policy

      edit "Test_Policy"

      ...

      set client-real-ip enable

      set real-ip-addr 10.13.4.253

      next

      end

  5. Check if there are system crash/coredump when the issue occurs.
  6. Check related logs and captured files for further analysis.