Fortinet black logo

Administration Guide

Checking core files and basic coredump information

Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

  1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

    FortiWeb# show full-configuration sys settings

    config system settings

    set enable-debug-log enable #enabled by default

    end

  2. Use “diagnose debug crashlog show” to check if any coredump files are generated.

    FortiWeb# diagnose debug <xxxlog> show
    xxx_log: coredumplog, crashlog, daemonlog, emerglog, kernlog, netstatlog

    FortiWeb# diagnose debug crashlog show

    core-proxyd-2141-1630609770

    core-proxyd-7794-1630610047

    core-proxyd-60152-1630609579

    You can also check if new core* or coredump* files are available in System > Maintenance > Backup & Restore > GUI File Download/Upload.

  3. Use “diagnose debug coredumplog show” to show basic stack information.

    FortiWeb# diagnose debug coredumplog show

    ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

    (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

    1 0x0000563f7b48584c in session_management_get_weight ()

    0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

    3 0x0000563f7b4b262d in ip_intelligence_session_init ()

    4 0x0000563f7b3343ff in pth_init_modinfo ()

    5 0x0000563f7b310797 in pt_service_http_init ()

    6 0x0000563f7b30959c in pt_service_init ()

    7 0x0000563f7b3837bb in pt_stream_create_service ()

    8 0x0000563f7b3842f1 in pt_stream_create ()

    9 0x0000563f7b38a3d4 in session_accept ()

    10 0x0000563f7b350cba in fd_epoll_poll ()

    11 0x0000563f7b39622d in _worker_loop ()

    0000012 0x0000563f7b3965f8 in worker_run ()

    13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

    14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

    (gdb)

    From above information from bug #770008, it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

    Note: The format of core files are defined by:

    /# more /proc/sys/kernel/core_pattern

    /var/log/gui_upload/core-%e-%p-%t

    %e: daemon/process name

    %p: PID of the process

    %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

    E.g.:

    Epoch Converter - Unix Timestamp Converter

Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

  1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

    FortiWeb# show full-configuration sys settings

    config system settings

    set enable-debug-log enable #enabled by default

    end

  2. Use “diagnose debug crashlog show” to check if any coredump files are generated.

    FortiWeb# diagnose debug <xxxlog> show
    xxx_log: coredumplog, crashlog, daemonlog, emerglog, kernlog, netstatlog

    FortiWeb# diagnose debug crashlog show

    core-proxyd-2141-1630609770

    core-proxyd-7794-1630610047

    core-proxyd-60152-1630609579

    You can also check if new core* or coredump* files are available in System > Maintenance > Backup & Restore > GUI File Download/Upload.

  3. Use “diagnose debug coredumplog show” to show basic stack information.

    FortiWeb# diagnose debug coredumplog show

    ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

    (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

    1 0x0000563f7b48584c in session_management_get_weight ()

    0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

    3 0x0000563f7b4b262d in ip_intelligence_session_init ()

    4 0x0000563f7b3343ff in pth_init_modinfo ()

    5 0x0000563f7b310797 in pt_service_http_init ()

    6 0x0000563f7b30959c in pt_service_init ()

    7 0x0000563f7b3837bb in pt_stream_create_service ()

    8 0x0000563f7b3842f1 in pt_stream_create ()

    9 0x0000563f7b38a3d4 in session_accept ()

    10 0x0000563f7b350cba in fd_epoll_poll ()

    11 0x0000563f7b39622d in _worker_loop ()

    0000012 0x0000563f7b3965f8 in worker_run ()

    13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

    14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

    (gdb)

    From above information from bug #770008, it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

    Note: The format of core files are defined by:

    /# more /proc/sys/kernel/core_pattern

    /var/log/gui_upload/core-%e-%p-%t

    %e: daemon/process name

    %p: PID of the process

    %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

    E.g.:

    Epoch Converter - Unix Timestamp Converter