Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

The maximum size of the CLI script action output is 16K characters. In cases where the output exceeds 16K, the email received will contain truncated output. To avoid this, it is recommended to limit the number of commands per script.

To define memory and CPU usage thresholds:

config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold	Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).
memory-use-threshold-extreme	Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).
memory-use-threshold-green	Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).
memory-use-threshold-red	Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs two CLI scripts to collect debug information, and then two email messages will be received with CLI output to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

Since the output in this example will exceed 16K, two scripts are used. The CLI scripts are run sequentially, and an email is sent out each time a script runs.

To create an automation stitch for high memory usage in the GUI:

1. Go to Security Fabric > Automation and click Create New.
2. Enter the stitch name (auto_high_memory).
3. Configure the trigger:
   1. Click Add Trigger.
   2. Click Create and select Conserve Mode.
   3. Enter the name, auto_high_memory.
   4. Click OK.
   5. Select the trigger in the list and click Apply.
4. Configure the first CLI Script action:
   1. Click Add Action.
   2. Click Create and select CLI Script.
   3. Enter the following:

      Name	high_memory_debug1
      Script	diagnose sys top 5 20 5 diagnose sys session full-stat get system performance status
      Administrator profile	Select a profile

      

   4. Click OK.
   5. Select the action in the list and click Apply.
5. Configure the first Email notification action:
   1. Click Add Action.
   2. Click Create and select Email.
   3. Enter the following:

      Name	auto_high_memory_email1
      To	Enter an email address
      Subject	CSF stitch alert: high_memory1
      Body	%%results%%

   4. Click OK.
   5. Select the action in the list and click Apply.
6. Configure the second CLI Script action:
   1. Click Add Action.
   2. Click Create and select CLI Script.
   3. Enter the following:

      Name	high_memory_debug2
      Script	diagnose sys session full-stat diagnose hardware sysinfo shm diagnose hardware sysinfo memory
      Administrator profile	Select a profile

      

   4. Click OK.
   5. Select the action in the list and click Apply.
7. Configure the second Email notification action:
   1. Click Add Action.
   2. Click Create and select Email.
   3. Enter the following:

      Name	auto_high_memory_email2
      To	Enter an email address
      Subject	CSF stitch alert: high_memory2
      Body	%%results%%

   4. Click OK.
   5. Select the action in the list and click Apply.
8. Click OK.

To create an automation stitch for high memory usage in the CLI:

1. Create the automation trigger:

   config system automation-trigger
       edit "auto_high_memory"
           set event-type low-memory
       next
   end

2. Create the automation actions:

   config system automation-action
       edit "high_memory_debug1"
           set action-type cli-script
           set script "diagnose sys top 5 20 5
   diagnose sys session full-stat 
   get system performance status"
           set output-size 10
           set timeout 0	
           set accprofile "super_admin"	
       next
       edit "auto_high_memory_email1"
           set action-type email
           set email-to "person@fortinet.com"
           set email-subject "CSF stitch alert: high_memory1"
           set message "%%results%%"
       next
          edit "high_memory_debug2"
           set action-type cli-script
           set script "diagnose sys session full-stat
   diagnose hardware sysinfo shm
   diagnose hardware sysinfo memory"
           set accprofile "super_admin"
       next
       edit "auto_high_memory_email2"
           set action-type email
           set email-to "person@fortinet.com"
           set email-subject "CSF stitch alert: high_memory2"
           set message "%%results%%"
       next
   end

3. Create the automation stitch:

   config system automation-stitch
       edit "auto_high_memory"
           set trigger "auto_high_memory"
           config actions
               edit 1
                   set action "high_memory_debug1"
                   set required enable
               next
               edit 2
                   set action "auto_high_memory_email1"
                   set required enable
               next
               edit 3
                   set action "high_memory_debug2"
                   set required enable
               next
               edit 4
                   set action "auto_high_memory_email2"
                   set required enable
               next
           end
       next
   end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory1
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0
...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib