Fortinet black logo

Administration Guide

Checking memory usage

  1. Use “diagnose debug memory” to check memory usage:

    This command will collect memory information via several different kinds of backend commands.

    FortiWeb# diagnose debug memory

    Tue Oct 26 17:42:56 UTC 2021

    17:42:56 up 5 days, 19:45, load average: 2.09, 1.78, 1.82

    init 1 shared 1528kB anonymous 112kB

    cmdbsvr 191 shared 17132kB anonymous 33688kB

    syslogd 873 shared 256kB anonymous 44kB

    klogd 874 shared 256kB anonymous 48kB

    hamain 875 shared 10632kB anonymous 6972kB

    hasync 876 shared 9328kB anonymous 6832kB

    ...

    ...

  2. After 6.4 release, the system will generate a regular monitoring file with a backend command “/bin/FortiWeb_get_memory_usage”, which includes the same output of “diagnose debug memory”.

    The regular output is recorded in /var/log/gui_upload/debug_memory.txt and can be downloaded via System > Maintenance > Backup&Restore. You can download it manually or use the one-click button to archive and download it.

    You can set the interval to record debug memory logs:

    config system global

    set debug-memory-interval 5 #default 5 minutes and the range is from 1 to 65535

    end

    /# more /var/log/gui_upload/debug_memory.txt

    Fri May 28 04:30:13 UTC 2021

    04:30:13 up 0 min, load average: 1.07, 0.26, 0.09

    init 1 shared 1376kB anonymous 104kB

    cmdbsvr 2293 shared 14044kB anonymous 29032kB

    syslogd 3457 shared 280kB anonymous 48kB

    ...

    ...

  3. Check current memory usage in backend shell:

    Please note that FortiWeb changes the way to login to the backend shell Refer to Part VI: Run backend-shell commands.

    free

    This command gives you a table of the total, used, free, shared, buffer/cache, and available RAM. It also shows the total amount of swap space configured, and how much is used and available. The default unit is KB.

    free= total – used – buff/cache

    /# free

    total used free shared buffers cached

    Mem: 130987692 4990340 125997352 16540 23576 129092

    -/+ buffers/cache: 4837672 126150020

    Swap: 0 0 0

    /proc/meminfo

    This is a virtual file that reports the amount of available and used memory. It contains real-time information about the system’s memory usage as well as the buffers and shared memory used by the kernel

    /# cat /proc/meminfo

    MemTotal: 28635360 kB

    MemFree: 25998836 kB

    MemAvailable: 26127368 kB

    Buffers: 201340 kB

    Cached: 192220 kB

    SwapCached: 0 kB

    Active: 1730772 kB

    Inactive: 164688 kB

    Active(anon): 1501972 kB

    Inactive(anon): 38064 kB

    Active(file): 228800 kB

    Inactive(file): 126624 kB

    Unevictable: 1164 kB

    Mlocked: 1164 kB

    SwapTotal: 0 kB

    SwapFree: 0 kB

    Dirty: 104 kB

    Writeback: 0 kB

    AnonPages: 1503120 kB

    Mapped: 89856 kB

    Shmem: 38164 kB

    KReclaimable: 22528 kB

    Slab: 94268 kB

    SReclaimable: 22528 kB

    SUnreclaim: 71740 kB

    KernelStack: 5536 kB

    PageTables: 12048 kB

    NFS_Unstable: 0 kB

    Bounce: 0 kB

    WritebackTmp: 0 kB

    CommitLimit: 14317680 kB

    Committed_AS: 5405984 kB

    VmallocTotal: 34359738367 kB

    VmallocUsed: 64024 kB

    VmallocChunk: 0 kB

    Percpu: 1984 kB

    DirectMap4k: 63424 kB

    DirectMap2M: 3082240 kB

    DirectMap1G: 28311552 kB

    /# top

    Some common usage: Press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage; Press “1” (number one) to check status of all logical processors.

    /# top

    Mem: 4919392K used, 126068300K free, 16348K shrd, 45984K buff, 134312K cached

    CPU: 0.1% usr 0.0% sys 0.0% nic 99.8% idle 0.0% io 0.0% irq 0.0% sirq

    Load average: 1.33 1.40 1.38 2/954 28663

    PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND

    6262 1 root S 9582m 7.4 55 0.0 /bin/proxyd

    6276 1 root S 224m 0.1 39 0.0 /bin/confd_ha

    6264 1 root S 6539m 5.1 24 0.0 /bin/bot_daemon

    6273 1 root S 2498m 1.9 60 0.0 /bin/garbage -o standalone

    6316 6238 root S 2098m 1.6 7 0.0 /bin/mysqld --defaults-file=/data/etc

    6251 1 root S 803m 0.6 12 0.0 /bin/monitord

    6269 1 root S 411m 0.3 17 0.0 /bin/sandboxd

    6271 1 root S 400m 0.3 41 0.0 /bin/shibd -F -f -p /var/run/shibd.pi

    6287 1 root S 256m 0.2 59 0.0 /bin/statusd

    6257 1 root S 245m 0.1 63 0.0 /bin/wvsd

    6244 1 root S 219m 0.1 36 0.0 /bin/logd

    6272 1 root S 202m 0.1 26 0.0 /bin/fortiviewd

    /# ps -l

    We usually focus on the RSS and VSZ values in ps output, which can be used to check the memory utilization for a specific process.

    RSS stands for Resident Set Size and shows how much RAM is utilized at present. It shows the entire stack of physically allocated memory so it is more important.

    VSZ is short for Virtual Memory Size. It’s the total amount of memory a process may hypothetically access. When a process is started, VSZ memory becomes RSS memory.

    If the RSS value increases continuously and does not decrease even if the traffic drops down, it might be a hint of memory leak and require further investigation.

    An example as below:

    At the beginning without any traffic, the VSZ is 10.3g VSZ, and RSS is 612m.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 6092968 122149324 78124 2741376 123905312

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 10.3g 612m 0:0 11:31 00:00:33 /bin/proxyd

    After traffic is pushed, both RSS and VSZ increase obviously.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 121216100 567704 78128 9199864 8769384

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 127g 88.9g 0:0 11:31 02:25:58 /bin/proxyd

    After traffic is stopped and concurrent connections are released, the RSS value may decrease obviously, but the VSZ often does not decrease.

    And, since some memory is still used, generally the RSS memory may not decrease to the initial value. As shown below, RSS is decreased from the peak value 88.9g to 1982m, but not the initial value 612m. It is normal and does not indicate a memory leak.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 9952592 119414496 77964 1616580 120133272

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 127g 1982m 0:0 11:31 02:39:38 /bin/proxyd

  1. Use “diagnose debug memory” to check memory usage:

    This command will collect memory information via several different kinds of backend commands.

    FortiWeb# diagnose debug memory

    Tue Oct 26 17:42:56 UTC 2021

    17:42:56 up 5 days, 19:45, load average: 2.09, 1.78, 1.82

    init 1 shared 1528kB anonymous 112kB

    cmdbsvr 191 shared 17132kB anonymous 33688kB

    syslogd 873 shared 256kB anonymous 44kB

    klogd 874 shared 256kB anonymous 48kB

    hamain 875 shared 10632kB anonymous 6972kB

    hasync 876 shared 9328kB anonymous 6832kB

    ...

    ...

  2. After 6.4 release, the system will generate a regular monitoring file with a backend command “/bin/FortiWeb_get_memory_usage”, which includes the same output of “diagnose debug memory”.

    The regular output is recorded in /var/log/gui_upload/debug_memory.txt and can be downloaded via System > Maintenance > Backup&Restore. You can download it manually or use the one-click button to archive and download it.

    You can set the interval to record debug memory logs:

    config system global

    set debug-memory-interval 5 #default 5 minutes and the range is from 1 to 65535

    end

    /# more /var/log/gui_upload/debug_memory.txt

    Fri May 28 04:30:13 UTC 2021

    04:30:13 up 0 min, load average: 1.07, 0.26, 0.09

    init 1 shared 1376kB anonymous 104kB

    cmdbsvr 2293 shared 14044kB anonymous 29032kB

    syslogd 3457 shared 280kB anonymous 48kB

    ...

    ...

  3. Check current memory usage in backend shell:

    Please note that FortiWeb changes the way to login to the backend shell Refer to Part VI: Run backend-shell commands.

    free

    This command gives you a table of the total, used, free, shared, buffer/cache, and available RAM. It also shows the total amount of swap space configured, and how much is used and available. The default unit is KB.

    free= total – used – buff/cache

    /# free

    total used free shared buffers cached

    Mem: 130987692 4990340 125997352 16540 23576 129092

    -/+ buffers/cache: 4837672 126150020

    Swap: 0 0 0

    /proc/meminfo

    This is a virtual file that reports the amount of available and used memory. It contains real-time information about the system’s memory usage as well as the buffers and shared memory used by the kernel

    /# cat /proc/meminfo

    MemTotal: 28635360 kB

    MemFree: 25998836 kB

    MemAvailable: 26127368 kB

    Buffers: 201340 kB

    Cached: 192220 kB

    SwapCached: 0 kB

    Active: 1730772 kB

    Inactive: 164688 kB

    Active(anon): 1501972 kB

    Inactive(anon): 38064 kB

    Active(file): 228800 kB

    Inactive(file): 126624 kB

    Unevictable: 1164 kB

    Mlocked: 1164 kB

    SwapTotal: 0 kB

    SwapFree: 0 kB

    Dirty: 104 kB

    Writeback: 0 kB

    AnonPages: 1503120 kB

    Mapped: 89856 kB

    Shmem: 38164 kB

    KReclaimable: 22528 kB

    Slab: 94268 kB

    SReclaimable: 22528 kB

    SUnreclaim: 71740 kB

    KernelStack: 5536 kB

    PageTables: 12048 kB

    NFS_Unstable: 0 kB

    Bounce: 0 kB

    WritebackTmp: 0 kB

    CommitLimit: 14317680 kB

    Committed_AS: 5405984 kB

    VmallocTotal: 34359738367 kB

    VmallocUsed: 64024 kB

    VmallocChunk: 0 kB

    Percpu: 1984 kB

    DirectMap4k: 63424 kB

    DirectMap2M: 3082240 kB

    DirectMap1G: 28311552 kB

    /# top

    Some common usage: Press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage; Press “1” (number one) to check status of all logical processors.

    /# top

    Mem: 4919392K used, 126068300K free, 16348K shrd, 45984K buff, 134312K cached

    CPU: 0.1% usr 0.0% sys 0.0% nic 99.8% idle 0.0% io 0.0% irq 0.0% sirq

    Load average: 1.33 1.40 1.38 2/954 28663

    PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND

    6262 1 root S 9582m 7.4 55 0.0 /bin/proxyd

    6276 1 root S 224m 0.1 39 0.0 /bin/confd_ha

    6264 1 root S 6539m 5.1 24 0.0 /bin/bot_daemon

    6273 1 root S 2498m 1.9 60 0.0 /bin/garbage -o standalone

    6316 6238 root S 2098m 1.6 7 0.0 /bin/mysqld --defaults-file=/data/etc

    6251 1 root S 803m 0.6 12 0.0 /bin/monitord

    6269 1 root S 411m 0.3 17 0.0 /bin/sandboxd

    6271 1 root S 400m 0.3 41 0.0 /bin/shibd -F -f -p /var/run/shibd.pi

    6287 1 root S 256m 0.2 59 0.0 /bin/statusd

    6257 1 root S 245m 0.1 63 0.0 /bin/wvsd

    6244 1 root S 219m 0.1 36 0.0 /bin/logd

    6272 1 root S 202m 0.1 26 0.0 /bin/fortiviewd

    /# ps -l

    We usually focus on the RSS and VSZ values in ps output, which can be used to check the memory utilization for a specific process.

    RSS stands for Resident Set Size and shows how much RAM is utilized at present. It shows the entire stack of physically allocated memory so it is more important.

    VSZ is short for Virtual Memory Size. It’s the total amount of memory a process may hypothetically access. When a process is started, VSZ memory becomes RSS memory.

    If the RSS value increases continuously and does not decrease even if the traffic drops down, it might be a hint of memory leak and require further investigation.

    An example as below:

    At the beginning without any traffic, the VSZ is 10.3g VSZ, and RSS is 612m.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 6092968 122149324 78124 2741376 123905312

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 10.3g 612m 0:0 11:31 00:00:33 /bin/proxyd

    After traffic is pushed, both RSS and VSZ increase obviously.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 121216100 567704 78128 9199864 8769384

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 127g 88.9g 0:0 11:31 02:25:58 /bin/proxyd

    After traffic is stopped and concurrent connections are released, the RSS value may decrease obviously, but the VSZ often does not decrease.

    And, since some memory is still used, generally the RSS memory may not decrease to the initial value. As shown below, RSS is decreased from the peak value 88.9g to 1982m, but not the initial value 612m. It is normal and does not indicate a memory leak.

    /# free

    total used free shared buff/cache available

    Mem: 130983668 9952592 119414496 77964 1616580 120133272

    Swap: 0 0 0

    /#

    /# ps -l | grep “/proxyd” | grep -v grep

    S 0 23653 1 127g 1982m 0:0 11:31 02:39:38 /bin/proxyd