Fortinet black logo

Administration Guide

Home FortiNDR 7.2.3 Administration Guide
7.2.3
7.4.4
7.4.3
7.4.2
7.4.1
7.4.0
7.2.3
7.2.2
7.2.1
7.2.0
7.1.0
7.0.6
7.0.5
7.0.4
7.0.3
7.0.2
7.0.1
7.0.0

FortiNDR troubleshooting tips

FortiNDR troubleshooting tips

For more information about the CLI commands below, please see the FortiNDR CLI Reference Guide.

Best practices:
Recommendations CLI command

Comments
Reload all services and see if the issue is still reproducible exec reload

Turn off feature learning

exec learner off

If you loaded an interim build (other than GA) and are willing to wipe all db records exec db restore

Run exec reload to see if issue is still reproducible
If you loaded an interim build (other than GA) and cannot wipe all db records diagnose system db

Patches db at best efforts.
Retrieve and record all information get sys status

If you are seeing high CPU and MEM usage, please consider provisioning more resources.

Retrieve and record all information for VMs

diag sys vm

Observe for any FDS code other than 200, and if not 200, please check connections to FDN and license status.
Recommended Debug Setup:
  • A syslog server for FortiNDR events log as the GUI only has 1 days events.
  • A TFTP server for PCAP capture transfer.
General Debug Logs Retrieval

Scenario

CLI
Collect all crash logs from the first day FortiNDR started diagnose debug crashlog <crash_log_date>
Record kernel related logs from the bootup and save it to a file diagnose debug kernel display

File scanning related issues

The following troubleshooting tips are intended to diagnose the error message: File Not Accepted (Client side shows files are submitted but NDR does not have details of file).

To perform a general check:
  1. Check and record network conditions from the FortiNDR server to file submitting clients using the following CLI commands:
    • exec ping
    • exec traceroute
  2. Make sure all KDBs are updated. For example, no pending updates, no out of date db and no updating.
  3. Try submitting a lower throughput, (no archive file type, smaller file size) to see if it is still reproducible.
  4. Follow the PCAP dumping guide to dump files from port1 or port2 to make sure the traffic is there. Open dapture pcap with Wireshark to see if there are any redline/blacklines from Wireshark default filter setting which indicates bad network traffic quality. From previous troubleshooting experience, this is the most frequent cause of File Not Accepted.
Troubleshooting ICAP issues:
  1. After you reproduce the issue:
    1. Retrieve the latest ICAP server logs by running the CLI command: diag debug icap
    2. Save the server logs to a file.
  2. Usually you can resolve any outstanding issues by running the following CLI command: exec reload
Troubleshooting OFTP issues:
  1. From OFTP clients (usually FortiGate), record all traffic forward/AntiVirus Event logs from the Fortigate side.
  2. Refer to PCAP capturing guide, and save corresponding PCAPs.
Troubleshooting HTTP2 issues from FortiGate v7.0 onwards:

Recommendation

Run the following CLI command:

Record output and check for errors diagnose system csf global
Record output and make sure status is authorized diagnose system csf upstream
Collect logs diag debug enable and diagnose debug csfd 7

Manual Upload/API Submission/FortiSandbox Integration

For all issues:

Start with a single file upload and fetch results from the same subnet as directed from where the client resides. See Appendix A - API guide.

To verify the process is successful:

If a single file submit/fetch is working from the previous step. Run the following CLI commands:

  • diag debug enable

and

  • diagnose debug application 7

Record all output and look for any non 200 http code or stack traces.

File Submitted but not processed

Collect all the information from the process and record it using the following CLI commands:

  • diag debug enable

and

  • diagnose debug process <process_name>

Information for support tickets

If none of these recommendations work and you need to create a support ticket, please include the following information:

  1. PCAPs from Port1 or Port2 sniffer capturing. If the poc includes private traffic you do not want to share, provide a general analysis from NDR’s port1 or port2 from Wireshark. Include stats about the default filter, redlines and black line (tcp error).
  2. What actions were taken.
  3. Logs collected from your troubleshooting steps.
Previous
Next

FortiNDR troubleshooting tips

For more information about the CLI commands below, please see the FortiNDR CLI Reference Guide.

Best practices:
Recommendations CLI command

Comments
Reload all services and see if the issue is still reproducible exec reload

Turn off feature learning

exec learner off

If you loaded an interim build (other than GA) and are willing to wipe all db records exec db restore

Run exec reload to see if issue is still reproducible
If you loaded an interim build (other than GA) and cannot wipe all db records diagnose system db

Patches db at best efforts.
Retrieve and record all information get sys status

If you are seeing high CPU and MEM usage, please consider provisioning more resources.

Retrieve and record all information for VMs

diag sys vm

Observe for any FDS code other than 200, and if not 200, please check connections to FDN and license status.
Recommended Debug Setup:
  • A syslog server for FortiNDR events log as the GUI only has 1 days events.
  • A TFTP server for PCAP capture transfer.
General Debug Logs Retrieval

Scenario

CLI
Collect all crash logs from the first day FortiNDR started diagnose debug crashlog <crash_log_date>
Record kernel related logs from the bootup and save it to a file diagnose debug kernel display

File scanning related issues

The following troubleshooting tips are intended to diagnose the error message: File Not Accepted (Client side shows files are submitted but NDR does not have details of file).

To perform a general check:
  1. Check and record network conditions from the FortiNDR server to file submitting clients using the following CLI commands:
    • exec ping
    • exec traceroute
  2. Make sure all KDBs are updated. For example, no pending updates, no out of date db and no updating.
  3. Try submitting a lower throughput, (no archive file type, smaller file size) to see if it is still reproducible.
  4. Follow the PCAP dumping guide to dump files from port1 or port2 to make sure the traffic is there. Open dapture pcap with Wireshark to see if there are any redline/blacklines from Wireshark default filter setting which indicates bad network traffic quality. From previous troubleshooting experience, this is the most frequent cause of File Not Accepted.
Troubleshooting ICAP issues:
  1. After you reproduce the issue:
    1. Retrieve the latest ICAP server logs by running the CLI command: diag debug icap
    2. Save the server logs to a file.
  2. Usually you can resolve any outstanding issues by running the following CLI command: exec reload
Troubleshooting OFTP issues:
  1. From OFTP clients (usually FortiGate), record all traffic forward/AntiVirus Event logs from the Fortigate side.
  2. Refer to PCAP capturing guide, and save corresponding PCAPs.
Troubleshooting HTTP2 issues from FortiGate v7.0 onwards:

Recommendation

Run the following CLI command:

Record output and check for errors diagnose system csf global
Record output and make sure status is authorized diagnose system csf upstream
Collect logs diag debug enable and diagnose debug csfd 7

Manual Upload/API Submission/FortiSandbox Integration

For all issues:

Start with a single file upload and fetch results from the same subnet as directed from where the client resides. See Appendix A - API guide.

To verify the process is successful:

If a single file submit/fetch is working from the previous step. Run the following CLI commands:

  • diag debug enable

and

  • diagnose debug application 7

Record all output and look for any non 200 http code or stack traces.

File Submitted but not processed

Collect all the information from the process and record it using the following CLI commands:

  • diag debug enable

and

  • diagnose debug process <process_name>

Information for support tickets

If none of these recommendations work and you need to create a support ticket, please include the following information:

  1. PCAPs from Port1 or Port2 sniffer capturing. If the poc includes private traffic you do not want to share, provide a general analysis from NDR’s port1 or port2 from Wireshark. Include stats about the default filter, redlines and black line (tcp error).
  2. What actions were taken.
  3. Logs collected from your troubleshooting steps.
Previous
Next