Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:


Table of Contents

Administration Guide

Retrieving system logs in backend system

1. dmesg

Dmesg is used to examine or control the kernel ring buffer. It includes all important kernel information such as hardware loading and call trace information. Kernel level traffic debug logs will be also included in dmesg.

One can check such logs with “# dmesg” or “#dmesg | grep xxx” directly;

For further troubleshooting, you can archive all logs under the directory /var/log/dmesg/:

tar czf /var/log/gui_upload/dmesg.tar.gz /var/log/dmesg/

2. Apache error logs

If one failed to do some GUI related operation, please collect this logs for analysis:

/var/log# ls apache_logs/

error_log

3. CMDB logs

For configuration deployment issues, please collect cmdb logs for analysis:

# ls /var/log/cmdb/cmdb.log.*

cmdb/cmdb.log.0    cmdb/cmdb.log.155  cmdb/cmdb.log.211  cmdb/cmdb.log.44

#ls /var/log/dbg_cli/

4. /var/log/debug/

Some real-time logs will be generated and stored at /var/log/debug/:

/# ls /var/log/debug/

collect_tcpdump_para.txt    daemon_log_flag    proxyd_dbg

coredump_log_flag        dbsync_log        sample

crash.log            kernel.log        system-startup.log

crash_log_flag            kernel_log_flag    tmp

crl_updated_dbg        netstat_log_flag    daemon.log                nstd

5. /var/log/gui_upload/

1) Core, coredump and some real-time logs will be generated and stored at /var/log/gui_upload/:

/# ls /var/log/gui_upload/

core-proxyd-2141-1630609770    dlog_logd            ha_event_log            

core-proxyd-7794-1630610047    ints.txt                debug_disk.txtirq        

jeprof.out.51146.1630448785.heap    perf.data             kern.log

debug_out_d_cond_cpu.sh.txt           debug_out_d_mem.sh.txt    debug_out_d_net.sh.txt           

debug_out_d_proc.sh.txt

2) Some logs named as “debug_<function name>.txt” (or with the prefix “debug_out_d_” in some intermediate builds) are generated after 6.4.1.

  • Scripts in /var/log/debug/sample/ are samples to run in /var/log/outgoing;

  • Scripts in /var/log/outgoing/ are scripts actually run in /var/log/outgoing;

  • Currently these system information are collected:

    /# ls /var/log/debug/sample/        #script samples

    README         d_cond_cpu.sh  d_mem.sh       d_net.sh       d_proc.sh      first_flag

    /# ls /var/log/outgoing/        #scripts actually run

    d_cond_cpu.sh  d_mem.sh       d_net.sh       d_proc.sh

    /# ls -l /var/log/gui_upload/debug_out_d_*    (in new builds files are debug_<function name>.txt)

    -rw-r--r--    1 root     0            65018 Sep 28 18:03 /var/log/gui_upload/debug_out_d_cond_cpu.sh.txt

    -rw-r--r--    1 root     0           119859 Sep 28 18:03 /var/log/gui_upload/debug_out_d_mem.sh.txt

    -rw-r--r--    1 root     0            66371 Sep 28 18:03 /var/log/gui_upload/debug_out_d_net.sh.txt

    -rw-r--r--    1 root     0           126484 Sep 28 18:03

    /var/log/gui_upload/debug_out_d_proc.sh.txt

  • The information collected by these scripts mainly include:

    d_cond_cpu.sh: If the CPU usage more than 90% - date, top 10 daemons of CPU usage, perf top for 10 seconds

    d_mem.sh: date, free, /proc/meminfo, etc.

    d_net.sh: date, netstat -natpu, route -n

    d_proc.sh: date, top -b -n1, ps

  • The running interval for these scripts can be set with CLI:

    FortiWeb # show full system global

    config system global

      set debug-monitor-interval 5    #minutes

    End

    If the script is blocked for 30 sec, the system will kill it and call it in the next debug-monitor-interval.

  • If necessary, one can add scripts (shell or python) to this directory to collect system information; (NOT Recommended, because too many these manually-added tasks may impact system running & stability)

  • The size of “debug_<function name>.txt” is limited to 25MB. If the size gets greater, it will be moved to an .old file. And there are only two files rotated.

3) NMON logs are generated after 6.4.0.

    NMON (shorthand for Nigel's Monitor) is a system monitor tool that can collect system performance statistics including CPU, Mem, Disk, Net, etc.

  • NMON log files (with a suffix .nmon) are generated automatically and stored at /var/log/debug/tmp, and will be archived and can be downloaded via the method described in below section 10.2. The maximum number of .nmon files stored is 180.

  • A .nmon file is generated with a sampling interval of 5 minutes, and each time when system boots up, a new .nmon file will be generated. So generally only one .nmon file named “FortiWeb_220107_1734.nmon” (may be different on some previous builds) will be generated each day. Multiple .nmon files generated in one day indicate that system rebooted or crashed.

  • After processed by an nmon analyzer:

Retrieving system logs in backend system

1. dmesg

Dmesg is used to examine or control the kernel ring buffer. It includes all important kernel information such as hardware loading and call trace information. Kernel level traffic debug logs will be also included in dmesg.

One can check such logs with “# dmesg” or “#dmesg | grep xxx” directly;

For further troubleshooting, you can archive all logs under the directory /var/log/dmesg/:

tar czf /var/log/gui_upload/dmesg.tar.gz /var/log/dmesg/

2. Apache error logs

If one failed to do some GUI related operation, please collect this logs for analysis:

/var/log# ls apache_logs/

error_log

3. CMDB logs

For configuration deployment issues, please collect cmdb logs for analysis:

# ls /var/log/cmdb/cmdb.log.*

cmdb/cmdb.log.0    cmdb/cmdb.log.155  cmdb/cmdb.log.211  cmdb/cmdb.log.44

#ls /var/log/dbg_cli/

4. /var/log/debug/

Some real-time logs will be generated and stored at /var/log/debug/:

/# ls /var/log/debug/

collect_tcpdump_para.txt    daemon_log_flag    proxyd_dbg

coredump_log_flag        dbsync_log        sample

crash.log            kernel.log        system-startup.log

crash_log_flag            kernel_log_flag    tmp

crl_updated_dbg        netstat_log_flag    daemon.log                nstd

5. /var/log/gui_upload/

1) Core, coredump and some real-time logs will be generated and stored at /var/log/gui_upload/:

/# ls /var/log/gui_upload/

core-proxyd-2141-1630609770    dlog_logd            ha_event_log            

core-proxyd-7794-1630610047    ints.txt                debug_disk.txtirq        

jeprof.out.51146.1630448785.heap    perf.data             kern.log

debug_out_d_cond_cpu.sh.txt           debug_out_d_mem.sh.txt    debug_out_d_net.sh.txt           

debug_out_d_proc.sh.txt

2) Some logs named as “debug_<function name>.txt” (or with the prefix “debug_out_d_” in some intermediate builds) are generated after 6.4.1.

  • Scripts in /var/log/debug/sample/ are samples to run in /var/log/outgoing;

  • Scripts in /var/log/outgoing/ are scripts actually run in /var/log/outgoing;

  • Currently these system information are collected:

    /# ls /var/log/debug/sample/        #script samples

    README         d_cond_cpu.sh  d_mem.sh       d_net.sh       d_proc.sh      first_flag

    /# ls /var/log/outgoing/        #scripts actually run

    d_cond_cpu.sh  d_mem.sh       d_net.sh       d_proc.sh

    /# ls -l /var/log/gui_upload/debug_out_d_*    (in new builds files are debug_<function name>.txt)

    -rw-r--r--    1 root     0            65018 Sep 28 18:03 /var/log/gui_upload/debug_out_d_cond_cpu.sh.txt

    -rw-r--r--    1 root     0           119859 Sep 28 18:03 /var/log/gui_upload/debug_out_d_mem.sh.txt

    -rw-r--r--    1 root     0            66371 Sep 28 18:03 /var/log/gui_upload/debug_out_d_net.sh.txt

    -rw-r--r--    1 root     0           126484 Sep 28 18:03

    /var/log/gui_upload/debug_out_d_proc.sh.txt

  • The information collected by these scripts mainly include:

    d_cond_cpu.sh: If the CPU usage more than 90% - date, top 10 daemons of CPU usage, perf top for 10 seconds

    d_mem.sh: date, free, /proc/meminfo, etc.

    d_net.sh: date, netstat -natpu, route -n

    d_proc.sh: date, top -b -n1, ps

  • The running interval for these scripts can be set with CLI:

    FortiWeb # show full system global

    config system global

      set debug-monitor-interval 5    #minutes

    End

    If the script is blocked for 30 sec, the system will kill it and call it in the next debug-monitor-interval.

  • If necessary, one can add scripts (shell or python) to this directory to collect system information; (NOT Recommended, because too many these manually-added tasks may impact system running & stability)

  • The size of “debug_<function name>.txt” is limited to 25MB. If the size gets greater, it will be moved to an .old file. And there are only two files rotated.

3) NMON logs are generated after 6.4.0.

    NMON (shorthand for Nigel's Monitor) is a system monitor tool that can collect system performance statistics including CPU, Mem, Disk, Net, etc.

  • NMON log files (with a suffix .nmon) are generated automatically and stored at /var/log/debug/tmp, and will be archived and can be downloaded via the method described in below section 10.2. The maximum number of .nmon files stored is 180.

  • A .nmon file is generated with a sampling interval of 5 minutes, and each time when system boots up, a new .nmon file will be generated. So generally only one .nmon file named “FortiWeb_220107_1734.nmon” (may be different on some previous builds) will be generated each day. Multiple .nmon files generated in one day indicate that system rebooted or crashed.

  • After processed by an nmon analyzer: