Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

   FortiWeb# show full-configuration sys settings

   config system settings

   set enable-debug-log enable #enabled by default

   end

2. Use “diagnose debug crashlog show” to check if any coredump files are generated.

   FortiWeb# diagnose debug <xxxlog> show
   xxx_log: coredumplog, crashlog, daemonlog, emerglog, kernlog, netstatlog

   FortiWeb# diagnose debug crashlog show

   core-proxyd-2141-1630609770

   core-proxyd-7794-1630610047

   core-proxyd-60152-1630609579

   You can also check if new core* or coredump* files are available in System > Maintenance > Backup & Restore > GUI File Download/Upload.

3. Use “diagnose debug coredumplog show” to show basic stack information.

   FortiWeb# diagnose debug coredumplog show

   ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

   (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

   1 0x0000563f7b48584c in session_management_get_weight ()

   0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

   3 0x0000563f7b4b262d in ip_intelligence_session_init ()

   4 0x0000563f7b3343ff in pth_init_modinfo ()

   5 0x0000563f7b310797 in pt_service_http_init ()

   6 0x0000563f7b30959c in pt_service_init ()

   7 0x0000563f7b3837bb in pt_stream_create_service ()

   8 0x0000563f7b3842f1 in pt_stream_create ()

   9 0x0000563f7b38a3d4 in session_accept ()

   10 0x0000563f7b350cba in fd_epoll_poll ()

   11 0x0000563f7b39622d in _worker_loop ()

   0000012 0x0000563f7b3965f8 in worker_run ()

   13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

   14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

   (gdb)

   From above information from bug #770008, it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

   Note: The format of core files are defined by:

   /# more /proc/sys/kernel/core_pattern

   /var/log/gui_upload/core-%e-%p-%t

   %e: daemon/process name

   %p: PID of the process

   %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

   E.g.:

   Epoch Converter - Unix Timestamp Converter