Fortinet black logo

Administration Guide

Server-policy outage

Server-policy outage

  1. Check if all services on FortiWeb are not available, including the HTTPS/SSH service to the management portal, and the HTTP/HTTPS access to the server-policy;
  2. Check the system resources especially when the memory size is low (4G or below, or many configuration entries) or memory usage is abnormal (memory leak or OOM - out of memory);
  3. Check if reboot, crash or coredump occurred when the issue happened;
  4. Check if any new operation/configuration are performed/added/modified before the issue happened;

    Event logs can be checked for configuration change event, while detailed CLIs are not included.

  5. Check if there is any traffic (CPS/Throughput/Attack) burst or shift when the issue happened;

    Traffic burst usually leads to high CPU usage, so you can check the Event logs, nmon records, or 3rd party network monitoring history to confirm.

  6. Check if a high volume of logs generated or sent to FortiAnylazer or other outside log servers (may be CPU consuming)

Temporary Actions/Solution

  • Check the status of proxyd with ps | grep proxyd;

  • Execute exec session-cleanup (recommended), kill proxyd, or kill dnsproxyd (or fn kill proxyd on the front CLI) to restart proxyd or other processes.

  • Collect system and debug logs for further support analysis:
    • Most important system logs can be fetched by one-click download via GUI > System > Maintenance > Debug > Download:

      Please note that you need to enable GUI > System > Config > Feature Visibility > Debug before seeing such option:

    • Sometimes newly-added debug logs may not be included in the archive file downloaded through above method, then it’s better to check and download such logs via GUI > System > Maintenance > Backup & Restore > GUI File Download/Upload:

      Similarly, you needs to enable the GUI File Download/Upload via CLI:

      config system settings

      set enable-file-upload enable

      end

Server-policy outage

  1. Check if all services on FortiWeb are not available, including the HTTPS/SSH service to the management portal, and the HTTP/HTTPS access to the server-policy;
  2. Check the system resources especially when the memory size is low (4G or below, or many configuration entries) or memory usage is abnormal (memory leak or OOM - out of memory);
  3. Check if reboot, crash or coredump occurred when the issue happened;
  4. Check if any new operation/configuration are performed/added/modified before the issue happened;

    Event logs can be checked for configuration change event, while detailed CLIs are not included.

  5. Check if there is any traffic (CPS/Throughput/Attack) burst or shift when the issue happened;

    Traffic burst usually leads to high CPU usage, so you can check the Event logs, nmon records, or 3rd party network monitoring history to confirm.

  6. Check if a high volume of logs generated or sent to FortiAnylazer or other outside log servers (may be CPU consuming)

Temporary Actions/Solution

  • Check the status of proxyd with ps | grep proxyd;

  • Execute exec session-cleanup (recommended), kill proxyd, or kill dnsproxyd (or fn kill proxyd on the front CLI) to restart proxyd or other processes.

  • Collect system and debug logs for further support analysis:
    • Most important system logs can be fetched by one-click download via GUI > System > Maintenance > Debug > Download:

      Please note that you need to enable GUI > System > Config > Feature Visibility > Debug before seeing such option:

    • Sometimes newly-added debug logs may not be included in the archive file downloaded through above method, then it’s better to check and download such logs via GUI > System > Maintenance > Backup & Restore > GUI File Download/Upload:

      Similarly, you needs to enable the GUI File Download/Upload via CLI:

      config system settings

      set enable-file-upload enable

      end