Fortinet black logo

Administration Guide

Diagnosing debug flow

    Diagnosing debug flow

Debugging traffic flow at user level with diagnose commands

The most commonly used diagnose debug flow commands are combined as below:

Reset enabled diagnose settings, turn on debug log output with timestamp

diagnose debug reset

diagnose debug enable

diagnose debug timestamp enable

Add filters and start the flow trace

diagnose debug flow filter flow-detail 7 #Enables messages from each packet processing module and packet flow traces

diagnose debug flow filter HTTP-detail 7 #HTTP parser details

diagnose debug flow filter module-detail status on #Turn on details from modules processing the flow

diagnose debug flow filter server-ip 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode

diagnose debug flow filter client-ip 192.168.12.1 #The client IP

diagnose debug flow trace start

To stop output

diagnose debug flow trace stop

Diagnose debug disable

Debugging traffic flow at kernel level

Change the debug levels in the backend settings, then kernel level debug logs will be recorded in dmesg. This method is useful to track traffic flow processing in the system kernel.

1) /proc/tproxy/debug # for transparent mode.

  • echo "FFFF F" > proc/tproxy/debug: output logs to dmesg with a detailed level

  • echo "XXXX F" > proc/tproxy/debug: don’t forget to turn off debug logs

Use the same way to turn on debug logs for reverse-proxy and wccp mode.

Some details:

/var/log# more /proc/tproxy/debug

Debug modules : HOOK4 HOOK6 HASH POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv6

HASH : for tproxy hash

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

LOIP : for enable / disable local ip filter in hook4

PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd

Debug levels : 1 2 4 8

1 : for error message

2 : for data packet info

4 : for data following info

8 : for function entry/exit info

Current debug info : FFFF 15, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0

ex : echo "HOOK4 F" > debug > debug

ex : echo "PIP 1 10.200.2.1" > debug

Example:

[BEGIN] 9/13/2021 23:35:55

/# dmesg

[553897.203831] (tproxy) (/Chroot_Build/34/SVN_REPO_CHILD/FortiWEB/kernel/modules/tproxy/tproxy_policy.c:433) get vserver(240.0.0.29), vport(9781), dir(1)

[553897.203834] (tproxy) ====> get vserver(240.0.0.29), vport(9781), mark(1835264/1835264), incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80)

[553897.203836] (tproxy) (465) incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80) -ipid(63355) iptlen(60) seq(2348868809) ack_seq(0) syn(1) ack(0) fin(0) rst(0) psh(0)

[553897.203838] (tproxy) [fortiweb-tproxy] redirecting: proto 6 192.168.11.2:80 -> 240.0.0.29:9781, ipid(63355) iplen(60) mark: 1c0100

[553897.203855] (tproxy)

[553897.203855]

[553897.203855] ====> out to client : src:(192.168.11.2:80), dst:(192.168.11.1:48310)- seq(1319007036) ack_seq(2348868810) syn(1) ack(1) fin(0) rst(0) psh(0)

[553897.203856] (tproxy) [POST_ROUTING]: TO CLIENT OK, 192.168.11.2:80->192.168.11.1:48310, todevname:port3vlan101, flag 4000

2) /proc/rptproxy/debug #for reverse-proxy mode

/var/log# more /proc/rptproxy/debug

Debug modules : HOOK4 HOOK6 HASH POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv6

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

LOIP : for enable / disable local ip filter in hook4

PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd

Debug levels : 1 2 4 8

...

Current debug info : 0, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0

3) /proc/wproxy/debug #for wccp mode

/var/log# more /proc/wproxy/debug

Debug modules : HOOK4 HOOK6 POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv4

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

Debug levels : 1 2 4 8

...

Current debug info : 0, mbypass = 0, sysmode : 1

How to capture network packets in FortiWeb

Capturing network packets is a useful and direct method when troubleshooting network issues, including TCP connection establishment issues, SSL handshake issues or analyzing HTTP issues.

Usually it’s better to enable diagnose debug flow and capture packets at the same time, then analyze them together.

    Diagnosing debug flow

Debugging traffic flow at user level with diagnose commands

The most commonly used diagnose debug flow commands are combined as below:

Reset enabled diagnose settings, turn on debug log output with timestamp

diagnose debug reset

diagnose debug enable

diagnose debug timestamp enable

Add filters and start the flow trace

diagnose debug flow filter flow-detail 7 #Enables messages from each packet processing module and packet flow traces

diagnose debug flow filter HTTP-detail 7 #HTTP parser details

diagnose debug flow filter module-detail status on #Turn on details from modules processing the flow

diagnose debug flow filter server-ip 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode

diagnose debug flow filter client-ip 192.168.12.1 #The client IP

diagnose debug flow trace start

To stop output

diagnose debug flow trace stop

Diagnose debug disable

Debugging traffic flow at kernel level

Change the debug levels in the backend settings, then kernel level debug logs will be recorded in dmesg. This method is useful to track traffic flow processing in the system kernel.

1) /proc/tproxy/debug # for transparent mode.

  • echo "FFFF F" > proc/tproxy/debug: output logs to dmesg with a detailed level

  • echo "XXXX F" > proc/tproxy/debug: don’t forget to turn off debug logs

Use the same way to turn on debug logs for reverse-proxy and wccp mode.

Some details:

/var/log# more /proc/tproxy/debug

Debug modules : HOOK4 HOOK6 HASH POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv6

HASH : for tproxy hash

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

LOIP : for enable / disable local ip filter in hook4

PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd

Debug levels : 1 2 4 8

1 : for error message

2 : for data packet info

4 : for data following info

8 : for function entry/exit info

Current debug info : FFFF 15, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0

ex : echo "HOOK4 F" > debug > debug

ex : echo "PIP 1 10.200.2.1" > debug

Example:

[BEGIN] 9/13/2021 23:35:55

/# dmesg

[553897.203831] (tproxy) (/Chroot_Build/34/SVN_REPO_CHILD/FortiWEB/kernel/modules/tproxy/tproxy_policy.c:433) get vserver(240.0.0.29), vport(9781), dir(1)

[553897.203834] (tproxy) ====> get vserver(240.0.0.29), vport(9781), mark(1835264/1835264), incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80)

[553897.203836] (tproxy) (465) incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80) -ipid(63355) iptlen(60) seq(2348868809) ack_seq(0) syn(1) ack(0) fin(0) rst(0) psh(0)

[553897.203838] (tproxy) [fortiweb-tproxy] redirecting: proto 6 192.168.11.2:80 -> 240.0.0.29:9781, ipid(63355) iplen(60) mark: 1c0100

[553897.203855] (tproxy)

[553897.203855]

[553897.203855] ====> out to client : src:(192.168.11.2:80), dst:(192.168.11.1:48310)- seq(1319007036) ack_seq(2348868810) syn(1) ack(1) fin(0) rst(0) psh(0)

[553897.203856] (tproxy) [POST_ROUTING]: TO CLIENT OK, 192.168.11.2:80->192.168.11.1:48310, todevname:port3vlan101, flag 4000

2) /proc/rptproxy/debug #for reverse-proxy mode

/var/log# more /proc/rptproxy/debug

Debug modules : HOOK4 HOOK6 HASH POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv6

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

LOIP : for enable / disable local ip filter in hook4

PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd

Debug levels : 1 2 4 8

...

Current debug info : 0, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0

3) /proc/wproxy/debug #for wccp mode

/var/log# more /proc/wproxy/debug

Debug modules : HOOK4 HOOK6 POLICY

HOOK4 : for netfilter hook ipv4

HOOK6 : for netfilter hook ipv4

POLICY : for policy management

FFFF : for all above

XXXX : cleanup all above

PASS : for bypass this module in kernel path

Debug levels : 1 2 4 8

...

Current debug info : 0, mbypass = 0, sysmode : 1

How to capture network packets in FortiWeb

Capturing network packets is a useful and direct method when troubleshooting network issues, including TCP connection establishment issues, SSL handshake issues or analyzing HTTP issues.

Usually it’s better to enable diagnose debug flow and capture packets at the same time, then analyze them together.