Fortinet black logo

Troubleshooting Guide

Checking core files and basic coredump information

Copy Link
Copy Doc ID e6dc543a-b050-11ec-9fd1-fa163e15d75b:36917
Download PDF

Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

  1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

    FortiWeb# show full-configuration sys settings

    config system settings

    set enable-debug-log enable #enabled by default

    end

  2. Use “diagnose debug crashlog show” to check if any coredump files are generated.

    FortiWeb# diagnose debug <xxxlog> show
    xxx_log: coredumplog, crashlog, daemonlog, emerglog, kernlog, netstatlog

    FortiWeb# diagnose debug crashlog show

    core-proxyd-2141-1630609770

    core-proxyd-7794-1630610047

    core-proxyd-60152-1630609579

    You can also check if new core* or coredump* files are available in System > Maintenance > Backup & Restore > GUI File Download/Upload.

  3. Use “diagnose debug coredumplog show” to show basic stack information.

    FortiWeb# diagnose debug coredumplog show

    ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

    (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

    1 0x0000563f7b48584c in session_management_get_weight ()

    0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

    3 0x0000563f7b4b262d in ip_intelligence_session_init ()

    4 0x0000563f7b3343ff in pth_init_modinfo ()

    5 0x0000563f7b310797 in pt_service_http_init ()

    6 0x0000563f7b30959c in pt_service_init ()

    7 0x0000563f7b3837bb in pt_stream_create_service ()

    8 0x0000563f7b3842f1 in pt_stream_create ()

    9 0x0000563f7b38a3d4 in session_accept ()

    10 0x0000563f7b350cba in fd_epoll_poll ()

    11 0x0000563f7b39622d in _worker_loop ()

    0000012 0x0000563f7b3965f8 in worker_run ()

    13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

    14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

    (gdb)

    From above information from bug #770008, it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

    Note:

    Note:

    1. Kernel coredump files cannot be shown by “diagnose debug crashlog show” or “diagnose debug coredumplog show” on 7.0.1 and previous versions.

    You need to check if there are files named as “core-xx” & “coredump-xx” under System > Maintenance > Backup & Restore > GUI File Download/Upload.

    /var/log/gui_upload# ls -l

    -rw-r--r-- 1 root root 257583 Sep 28 02:50 core-2021-05-11-22_38

    -rw-r--r-- 1 root root 670162944 Sep 28 02:50 coredump-2021-05-11-22_38

    If yes, please refer to the section below to download them for further analysis.

    2. The format of core files are defined by:

    /# more /proc/sys/kernel/core_pattern

    /var/log/gui_upload/core-%e-%p-%t

    %e: daemon/process name

    %p: PID of the process

    %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

    E.g.:

    Epoch Converter - Unix Timestamp Converter

Actually you have another way to simply check the file generation date from GUI; just check the section below to find "Download core/coredump files".

Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

  1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

    FortiWeb# show full-configuration sys settings

    config system settings

    set enable-debug-log enable #enabled by default

    end

  2. Use “diagnose debug crashlog show” to check if any coredump files are generated.

    FortiWeb# diagnose debug <xxxlog> show
    xxx_log: coredumplog, crashlog, daemonlog, emerglog, kernlog, netstatlog

    FortiWeb# diagnose debug crashlog show

    core-proxyd-2141-1630609770

    core-proxyd-7794-1630610047

    core-proxyd-60152-1630609579

    You can also check if new core* or coredump* files are available in System > Maintenance > Backup & Restore > GUI File Download/Upload.

  3. Use “diagnose debug coredumplog show” to show basic stack information.

    FortiWeb# diagnose debug coredumplog show

    ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

    (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

    1 0x0000563f7b48584c in session_management_get_weight ()

    0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

    3 0x0000563f7b4b262d in ip_intelligence_session_init ()

    4 0x0000563f7b3343ff in pth_init_modinfo ()

    5 0x0000563f7b310797 in pt_service_http_init ()

    6 0x0000563f7b30959c in pt_service_init ()

    7 0x0000563f7b3837bb in pt_stream_create_service ()

    8 0x0000563f7b3842f1 in pt_stream_create ()

    9 0x0000563f7b38a3d4 in session_accept ()

    10 0x0000563f7b350cba in fd_epoll_poll ()

    11 0x0000563f7b39622d in _worker_loop ()

    0000012 0x0000563f7b3965f8 in worker_run ()

    13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

    14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

    (gdb)

    From above information from bug #770008, it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

    Note:

    Note:

    1. Kernel coredump files cannot be shown by “diagnose debug crashlog show” or “diagnose debug coredumplog show” on 7.0.1 and previous versions.

    You need to check if there are files named as “core-xx” & “coredump-xx” under System > Maintenance > Backup & Restore > GUI File Download/Upload.

    /var/log/gui_upload# ls -l

    -rw-r--r-- 1 root root 257583 Sep 28 02:50 core-2021-05-11-22_38

    -rw-r--r-- 1 root root 670162944 Sep 28 02:50 coredump-2021-05-11-22_38

    If yes, please refer to the section below to download them for further analysis.

    2. The format of core files are defined by:

    /# more /proc/sys/kernel/core_pattern

    /var/log/gui_upload/core-%e-%p-%t

    %e: daemon/process name

    %p: PID of the process

    %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

    E.g.:

    Epoch Converter - Unix Timestamp Converter

Actually you have another way to simply check the file generation date from GUI; just check the section below to find "Download core/coredump files".