Fortinet white logo
Fortinet white logo

Administration Guide

Explicit proxy logging

Explicit proxy logging

Explicit proxy traffic logging can be used to troubleshoot the HTTP proxy status for each HTTP transaction with the following:

  • Monitor HTTP header requests and responses in the UTM web filter log. This requires an SSL deep inspection profile to be configured in the corresponding firewall policy.

  • Log the explicit web proxy forward server name using set log-forward-server, which is disabled by default.

    config web-proxy global
        set log-forward-server {enable | disable}
    end
  • Log TCP connection failures in the traffic log when a client initiates a TCP connection to a remote host through the FortiGate and the remote host is unreachable.

  • Log HTTP transaction details

Basic configuration

The following FortiGate configuration is used in the three explicit proxy traffic logging use cases in this topic.

To configure the FortiGate:
  1. Configure the web proxy profile:

    config web-proxy profile
        edit "header"
            config headers
                edit 1
                    set name "test_request_header"
                    set action monitor-request
                next
                edit 2
                    set name "ETag"
                    set action monitor-response
                next
            end
        next
    end
  2. Enable forward server name logging in traffic:

    config web-proxy global
        set proxy-fqdn "100D.qa"
        set log-forward-server enable
    end
    
  3. Configure the web filter banned word table to block any HTTP response containing the text, works:

    config webfilter content
        edit 1
            set name "default"
            config entries
                edit "works"
                    set status enable
                    set action block
                next
            end
        next
    end
    
  4. Configure the web filter profile:

    config webfilter profile
        edit "header"
            set feature-set proxy
            config web
                set bword-table 1
            end
            config ftgd-wf
                unset options
            end
            set log-all-url enable
            set extended-log enable
            set web-extended-all-action-log enable
        next
    end
  5. Configure the web proxy forwarding server:

    config web-proxy forward-server
        edit "fgt-b"
            set ip 172.16.200.20
        next
    end
  6. Configure the firewall policy:

    config firewall policy
        edit 1
            set srcintf "port10"
            set dstintf "port9"
            set action accept
            set srcaddr "all"
            set dstaddr "all"
            set schedule "always"
            set service "ALL"
            set utm-status enable
            set inspection-mode proxy
            set webproxy-profile "header"
            set webproxy-forward-server "fgt-b"
            set ssl-ssh-profile "deep-inspection"
            set webfilter-profile "header"
            set logtraffic all
            set nat enable
        next
    end
Note

A firewall policy is used in this basic configuration example and the specific examples that follow. This feature also works for the explicit web proxy or transparent web proxy with proxy policies, and the configurations are similar:

  • Example 1: apply the web-proxy profile and webfilter profile to the proxy policy.
  • Example 2: apply the webproxy-forward-server.

Example 1: monitoring HTTP header requests

In this example, the user wants to monitor some HTTP headers in HTTP messages forwarded through a FortiGate proxy (either transparent or explicit proxy with a firewall policy in proxy mode or a proxy policy). When the monitored headers are detected, they will be logged in the UTM web filter log.

In the web proxy profile configuration, the following HTTP headers are monitored:

  • test_request_header: this is a user-customized HTTP header.

  • ETag: this is a HTTP header returned by the web server's 200 OK response.

Based on the web filter profile configuration, the monitored headers in the web proxy profile will only be logged when the HTTP response received by the FortiGate triggers a block action by the banned word table. The log-all-url, extended-log, and web-extended-all-action-log settings in the web filter profile must be enabled.

The following settings are required in the firewall policy:

  • set inspection-mode proxy

  • set webproxy-profile "header"

  • set ssl-ssh-profile "deep-inspection"

  • set webfilter-profile "header"

  • set logtraffic all

To verify the configuration:
  1. Send a HTTP request from the client:

    curl -kv https://172.16.200.33 -H "test_request_header: aaaaa"

    This command sends a HTTP request with the header test_request_header: aaaaa through the FortiGate. Since the response from the web server contains the word works, the response will be blocked by the web filter profile (header). During this process, two logs will be generated.

  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=19:01:19 eventtime=1681956079146481995 tz="-0700" logid="0314012288" type="utm" subtype="webfilter" eventtype="content" level="warning" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" reqtype="direct" url="https://172.16.200.33/" sentbyte=0 rcvdbyte=0 direction="incoming" action="blocked" banword="works" msg="URL was blocked because it contained banned word(s)." rawdata="[REQ] test_request_header=aaaaa||[RESP] Content-Type=text/html|ETag=\"34-5b23b9d3b67f4\""
    
    2: date=2023-04-19 time=19:01:19 eventtime=1681956079144896978 tz="-0700" logid="0319013317" type="utm" subtype="webfilter" eventtype="urlmonitor" level="notice" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" action="passthrough" reqtype="direct" url="https://172.16.200.33/" sentbyte=724 rcvdbyte=2769 direction="outgoing" msg="URL has been visited" ratemethod="ip" cat=255 rawdata="[REQ] test_request_header=aaaaa"

    Log 1 is for the blocked HTTP response that contains both monitored headers, test_request_header and ETag, and their values, aaaaa and 34-5b23b9d3b67f4, respectively. Log 2 is for the HTTP request passing through the FortiGate proxy that contains test_request_header and its aaaaa value in the rawdata field.

Example 2: logging the explicit web proxy forward server name

In this example, the user wants to see the name of the web proxy forward server in the traffic log when the traffic is forwarded by a web proxy forward server.

In the global web proxy settings, log-forward-server must be enabled.

The following settings are required in the firewall policy:

  • set inspection-mode proxy

  • set webproxy-forward-server "fgt-b"

  • set logtraffic all

When a HTTP request is sent through the FortiGate proxy, the request will be forwarded by the FortiGate to the upstream proxy (fgt-b), and the forward server's name will be logged in the traffic log.

To verify the configuration:
  1. Send a HTTP request from the client:

    curl -kv https://www.google.com
  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=19:51:33 eventtime=1681959093510003961 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=49762 srcintf="port10" srcintfrole="undefined" dstip=142.250.217.100 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="United States" sessionid=43292 proto=6 action="client-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=49762 duration=120 sentbyte=0 rcvdbyte=37729 sentpkt=0 rcvdpkt=33 appcat="unscanned" wanin=3779 wanout=682 lanin=879 lanout=36005 fwdsrv="fgt-b" utmaction="block" countssl=1 utmref=65506-14

Example 3: logging TCP connection failures

In this example, a client initiates a TCP connection to a remote network node through the FortiGate. The connection fails because the IP address or port of the remote node is unreachable. A Connection Failed message appears in the logs. In the firewall policy configuration, the inspection-mode can be set to either proxy or flow mode.

Note

Based on the basic FortiGate configuration used in examples 1 and 2, the forward server may need to be removed from the firewall policy if the forward server's TCP IP port is actually reachable. If the forward server proxy tries to set up back-to-back TCP connections with the downstream FortiGate and the remote server as in the case of deep-inspection, then when the client tries to connect to a remote node (even if the IP address or port is unreachable), the downstream FortiGate is able to establish a TCP connection with the upstream forward server, so there will be no Connection Failed message in the downstream FortiGate's log.

Note

Currently, the Connection Failed message in the downstream FortiGate's log is visible for the case when there is an unreachable TCP port only when explicit web proxy with a proxy policy is configured. Therefore, the following example that makes use of a firewall policy demonstrates this log message is only supported for the unreachable IP address case.

To verify the configuration:
  1. Send a HTTP request from the client to an unreachable IP:

    curl -kv https://172.16.200.34
  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=20:25:55 eventtime=1681961155100007061 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=52452 srcintf="port10" srcintfrole="undefined" dstip=172.16.200.34 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="Reserved" sessionid=44903 proto=6 action="server-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=52452 duration=20 sentbyte=180 rcvdbyte=164 sentpkt=3 rcvdpkt=3 appcat="unscanned" wanin=0 wanout=0 lanin=0 lanout=0 crscore=5 craction=262144 crlevel="low" msg="Connection Failed"

Example 4: logging HTTP transaction details

HTTP transaction details are logged in a traffic log when HTTP traffic is routed through a proxy. After an HTTP transaction is proxied through the FortiGate, traffic logs of the http-transaction subtype are generated in addition to the forward subtype log. HTTP transaction logs are based on each transaction, such as an HTTP request and response pair. When there are multiple HTTP transactions completed over the TCP connection there will be multiple http-transaction logs and only one forward traffic log.

HTTP transaction logging can be enabled in explicit-web proxy, transparent-web proxy, access-proxy, and proxy-mode firewall policies.

config firewall proxy-policy
    edit 1
        set proxy {explicit-web | transparent-web | access-proxy}
        logtraffic {utm | all}
        set log-http-transaction {enable | disable}
    next
end
config firewall policy
    edit 1
        set inspection-mode proxy
        logtraffic {utm | all}
        set log-http-transaction {enable | disable}
    next
end

One http-transaction log is generated for each HTTP transaction. A TCP connection can have multiple HTTP transactions, so there can be multiple http-transaction logs for one forward traffic log.

  • Explicit-web proxy logs:

    In the http-transaction logs (logs 2 and 3), transaction information such as httpmethod and statuscode are recorded.

    1: date=2024-05-21 time=20:06:17 eventtime=1716347177537010993 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=80 dstintf="port3" dstintfrole="undefined" sessionid=316483733 service="HTTP" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=12204 duration=30 wanin=760 rcvdbyte=760 wanout=211 lanin=163 sentbyte=163 lanout=36194 appcat="unscanned" utmaction="block" countav=1 utmref=65515-14
    
    2: date=2024-05-21 time=20:06:17 eventtime=1716347177536946272 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 dstip=172.16.200.44 dstport=80 sessionid=316483733 transid=50331679 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/eicar.com" agent="curl/7.68.0" duration=0 reqlength=86 resplength=392 rcvdbyte=760 sentbyte=163 scheme="http" hostname="172.16.200.44" resptype="cached" httpmethod="GET"statuscode="403" reqtime=1716347177 resptime=0 respfinishtime=1716347177 appcat="unscanned" utmaction="block" countav=1 utmref=65515-0
    3: date=2024-05-21 time=20:06:06 eventtime=1716347166400042072 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 dstip=172.16.200.44 dstport=80 sessionid=316483733 transid=50331678 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=77 resplength=368 rcvdbyte=368 sentbyte=77 scheme="http" hostname="172.16.200.44" resptype="normal" httpmethod="GET"statuscode="200" reqtime=1716347166 resptime=1716347166 respfinishtime=1716347166 appcat="unscanned"

    When the EICAR test file in the response is blocked by utm-av, utmref information referring to the corresponding utm-av log is included:

    # execute log detail 2 "65515-0"
    1 logs found.
    1 logs returned.
    1: date=2024-05-21 time=20:06:17 eventtime=1716347177536848145 tz="-0700" logid="0211008192" type="utm" subtype="virus" eventtype="infected" level="warning" vd="vdom1" policyid=1 poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" policytype="proxy-policy" msg="File is infected." action="blocked" service="HTTP" sessionid=316483733 transid=50331679 srcip=10.1.100.11 dstip=172.16.200.44 srcport=42694 dstport=80 srccountry="Reserved" dstcountry="Reserved" srcintf="port1" srcintfrole="undefined" dstintf="port3" dstintfrole="undefined" proto=6 direction="incoming" filename="eicar.com" quarskip="Quarantine-disabled" virus="EICAR_TEST_FILE" viruscat="Virus" dtype="av-engine" itype="infected" ref="https://fortiguard.com/encyclopedia/virus/2172" virusid=2172 url="http://172.16.200.44/eicar.com" profile="av" agent="curl/7.68.0" httpmethod="GET" analyticssubmit="false" crscore=50 craction=2 crlevel="critical"
  • Forward traffic log and http-transaction logs for transparent-web proxy policy, access-proxy proxy policy, and proxy-mode firewall policy:

    • Transparent-web proxy policy:

      1: date=2024-05-23 time=23:22:36 eventtime=1716531756508124889 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=34326 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=90954 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=2 policytype="proxy-policy" poluuid="1f986ef4-14c1-51ef-7d4f-fd482cefde73" trandisp="snat" transip=172.16.200.8 transport=34326 duration=0 wanin=3051 rcvdbyte=3051 wanout=618 lanin=842 sentbyte=842 lanout=4551 appid=40568 app="HTTPS.BROWSER" appcat="Web.Client" apprisk="medium" utmaction="allow" countssl=2 utmref=65516-0
      
      2: date=2024-05-23 time=23:22:36 eventtime=1716531756507534974 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=34326 dstip=172.16.200.44 dstport=443 sessionid=90954 transid=50331679 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="accept" policyid=2 policytype="proxy-policy" poluuid="1f986ef4-14c1-51ef-7d4f-fd482cefde73" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=787 scheme="https" hostname="172.16.200.44" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716531756 resptime=1716531756 respfinishtime=1716531756 appcat="unscanned"
    • Access-proxy proxy policy:

      1: date=2024-05-23 time=23:24:10 eventtime=1716531850437771247 tz="-0700" logid="0005000024" type="traffic" subtype="ztna" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=51076 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=91243 service="HTTPS" proxyapptype="http" proto=6 action="accept" policyid=3 policytype="proxy-policy" poluuid="2853b1ba-195e-51ef-2e7c-93b136b9505a" duration=0 gatewayid=1 realserverid=1 vip="vip-ztna" accessproxy="ztna" clientdevicemanageable="unknown" clientcert="no" wanin=2981 rcvdbyte=2981 wanout=712 lanin=824 sentbyte=824 lanout=2000 appcat="unscanned"
      
      2: date=2024-05-23 time=23:24:10 eventtime=1716531850436959338 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=51076 dstip=172.16.200.44 dstport=443 sessionid=91243 transid=50331680 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="accept" policyid=3 policytype="proxy-policy" poluuid="2853b1ba-195e-51ef-2e7c-93b136b9505a" url="https://a.ftnt.com/" agent="curl/7.68.0" duration=0 reqlength=36 resplength=185 rcvdbyte=2981 sentbyte=800 scheme="https" hostname="a.ftnt.com" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716531850 resptime=1716531850 respfinishtime=1716531850 appcat="unscanned"
    • Proxy-mode firewall policy:

      1: date=2024-05-23 time=23:32:11 eventtime=1716532331148593220 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=49424 srcintf="port1" srcintfrole="undefined" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" srccountry="Reserved" dstcountry="Reserved" sessionid=92757 proto=6 action="close" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" service="HTTPS" trandisp="snat" transip=172.16.200.8 transport=49424 duration=1 sentbyte=1406 rcvdbyte=5235 sentpkt=15 rcvdpkt=13 appcat="unscanned" wanin=3051 wanout=618 lanin=842 lanout=4551 utmaction="allow" countssl=2 utmref=65511-0
      
      2: date=2024-05-23 time=23:32:10 eventtime=1716532330081764186 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=49424 dstip=172.16.200.44 dstport=443 sessionid=92757 transid=50331681 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" url="https://172.16.200.44/" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=787 scheme="https" hostname="172.16.200.44" resptype="normal" reqtime=1716532330 resptime=1716532330 respfinishtime=1716532330 appcat="unscanned"

For HTTPS with explicit-web proxy, there is an additional piece of http-transaction log for each CONNECT request and response:

3: date=2024-05-21 time=20:34:44 eventtime=1716348884524284243 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=46030 dstip=172.16.200.44 dstport=443 sessionid=316483736 transid=50331683 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716348884 resptime=0 respfinishtime=1716348884 appcat="unscanned"

For HTTPS with certificate-inspection or no inspection, there is only one http-transaction log for each TCP connection because the encrypted HTTP messages are not decrypted:

  • Firewall policy:

    2: date=2024-05-23 time=21:38:56 eventtime=1716525535340969183 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=46462 dstip=172.16.200.44 dstport=443 sessionid=70593 transid=1 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" url="172.16.200.44" duration=0 reqlength=842 resplength=3100 rcvdbyte=3100 sentbyte=842 scheme="https" hostname="172.16.200.44" resptype="N/A" reqtime=1716525535 resptime=1716525535 respfinishtime=1716525535 appcat="unscanned"
  • Explicit-web proxy policy:

    2: date=2024-05-23 time=21:36:18 eventtime=1716525378802239534 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=56986 dstip=172.16.200.44 dstport=443 sessionid=1369348242 transid=50331673 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716525378 resptime=0 respfinishtime=1716525378 appcat="unscanned"

For SOCKS proxy, there is one http-transaction log for each HTTP transaction per TCP connection:

1: date=2024-05-23 time=22:50:34 eventtime=1716529833463518327 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63744 dstip=34.107.221.82 dstport=80 sessionid=1369348358 transid=117441403 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://detectportal.firefox.com/success.txt?ipv4" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0" duration=0 reqlength=305 resplength=216 rcvdbyte=7128 sentbyte=20143 scheme="http" hostname="detectportal.firefox.com" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716529833 resptime=1716529833 respfinishtime=1716529833 appcat="unscanned"
2: date=2024-05-23 time=22:49:40 eventtime=1716529779381260298 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63842 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=80 dstintf="port3" dstintfrole="undefined" sessionid=1369348430 service="webproxy" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=16135 duration=0 wanin=392 rcvdbyte=392 wanout=356 lanin=725 sentbyte=725 lanout=71832 appcat="unscanned" utmaction="block" countav=1 utmref=65517-14
3: date=2024-05-23 time=22:49:40 eventtime=1716529779381204800 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63842 dstip=172.16.200.44 dstport=80 sessionid=1369348430 transid=117441401 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/eicar.com" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0" duration=0 reqlength=356 resplength=392 rcvdbyte=392 sentbyte=725 scheme="http" hostname="172.16.200.44" resptype="cached" httpmethod="GET" statuscode="403" reqtime=1716529779 resptime=0 respfinishtime=1716529779 appcat="unscanned" utmaction="block" countav=1 utmref=65517-0

UTM logs that do not belong to an HTTP transaction are only associated with the forward traffic log, and not the http-transaction log:

Only the forward traffic log is associated to the utm-ssl log by the utmref.

  • forward traffic log:

    1: date=2024-05-23 time=22:01:43 eventtime=1716526903039789335 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=1369348243 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=1837 duration=0 wanin=3051 rcvdbyte=3051 wanout=618 lanin=960 sentbyte=960 lanout=4623 appcat="unscanned" utmaction="allow" countssl=2 utmref=65523-0
  • http-transaction logs:

    2: date=2024-05-23 time=22:01:43 eventtime=1716526903038941661 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 dstip=172.16.200.44 dstport=443 sessionid=1369348243 transid=50331675 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=936 scheme="https" hostname="172.16.200.44" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716526903 resptime=1716526903 respfinishtime=1716526903 appcat="unscanned"
    
    3: date=2024-05-23 time=22:01:43 eventtime=1716526903015933382 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 dstip=172.16.200.44 dstport=443 sessionid=1369348243 transid=50331674 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716526903 resptime=0 respfinishtime=1716526903 appcat="unscanned"

Explicit proxy logging

Explicit proxy logging

Explicit proxy traffic logging can be used to troubleshoot the HTTP proxy status for each HTTP transaction with the following:

  • Monitor HTTP header requests and responses in the UTM web filter log. This requires an SSL deep inspection profile to be configured in the corresponding firewall policy.

  • Log the explicit web proxy forward server name using set log-forward-server, which is disabled by default.

    config web-proxy global
        set log-forward-server {enable | disable}
    end
  • Log TCP connection failures in the traffic log when a client initiates a TCP connection to a remote host through the FortiGate and the remote host is unreachable.

  • Log HTTP transaction details

Basic configuration

The following FortiGate configuration is used in the three explicit proxy traffic logging use cases in this topic.

To configure the FortiGate:
  1. Configure the web proxy profile:

    config web-proxy profile
        edit "header"
            config headers
                edit 1
                    set name "test_request_header"
                    set action monitor-request
                next
                edit 2
                    set name "ETag"
                    set action monitor-response
                next
            end
        next
    end
  2. Enable forward server name logging in traffic:

    config web-proxy global
        set proxy-fqdn "100D.qa"
        set log-forward-server enable
    end
    
  3. Configure the web filter banned word table to block any HTTP response containing the text, works:

    config webfilter content
        edit 1
            set name "default"
            config entries
                edit "works"
                    set status enable
                    set action block
                next
            end
        next
    end
    
  4. Configure the web filter profile:

    config webfilter profile
        edit "header"
            set feature-set proxy
            config web
                set bword-table 1
            end
            config ftgd-wf
                unset options
            end
            set log-all-url enable
            set extended-log enable
            set web-extended-all-action-log enable
        next
    end
  5. Configure the web proxy forwarding server:

    config web-proxy forward-server
        edit "fgt-b"
            set ip 172.16.200.20
        next
    end
  6. Configure the firewall policy:

    config firewall policy
        edit 1
            set srcintf "port10"
            set dstintf "port9"
            set action accept
            set srcaddr "all"
            set dstaddr "all"
            set schedule "always"
            set service "ALL"
            set utm-status enable
            set inspection-mode proxy
            set webproxy-profile "header"
            set webproxy-forward-server "fgt-b"
            set ssl-ssh-profile "deep-inspection"
            set webfilter-profile "header"
            set logtraffic all
            set nat enable
        next
    end
Note

A firewall policy is used in this basic configuration example and the specific examples that follow. This feature also works for the explicit web proxy or transparent web proxy with proxy policies, and the configurations are similar:

  • Example 1: apply the web-proxy profile and webfilter profile to the proxy policy.
  • Example 2: apply the webproxy-forward-server.

Example 1: monitoring HTTP header requests

In this example, the user wants to monitor some HTTP headers in HTTP messages forwarded through a FortiGate proxy (either transparent or explicit proxy with a firewall policy in proxy mode or a proxy policy). When the monitored headers are detected, they will be logged in the UTM web filter log.

In the web proxy profile configuration, the following HTTP headers are monitored:

  • test_request_header: this is a user-customized HTTP header.

  • ETag: this is a HTTP header returned by the web server's 200 OK response.

Based on the web filter profile configuration, the monitored headers in the web proxy profile will only be logged when the HTTP response received by the FortiGate triggers a block action by the banned word table. The log-all-url, extended-log, and web-extended-all-action-log settings in the web filter profile must be enabled.

The following settings are required in the firewall policy:

  • set inspection-mode proxy

  • set webproxy-profile "header"

  • set ssl-ssh-profile "deep-inspection"

  • set webfilter-profile "header"

  • set logtraffic all

To verify the configuration:
  1. Send a HTTP request from the client:

    curl -kv https://172.16.200.33 -H "test_request_header: aaaaa"

    This command sends a HTTP request with the header test_request_header: aaaaa through the FortiGate. Since the response from the web server contains the word works, the response will be blocked by the web filter profile (header). During this process, two logs will be generated.

  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=19:01:19 eventtime=1681956079146481995 tz="-0700" logid="0314012288" type="utm" subtype="webfilter" eventtype="content" level="warning" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" reqtype="direct" url="https://172.16.200.33/" sentbyte=0 rcvdbyte=0 direction="incoming" action="blocked" banword="works" msg="URL was blocked because it contained banned word(s)." rawdata="[REQ] test_request_header=aaaaa||[RESP] Content-Type=text/html|ETag=\"34-5b23b9d3b67f4\""
    
    2: date=2023-04-19 time=19:01:19 eventtime=1681956079144896978 tz="-0700" logid="0319013317" type="utm" subtype="webfilter" eventtype="urlmonitor" level="notice" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" action="passthrough" reqtype="direct" url="https://172.16.200.33/" sentbyte=724 rcvdbyte=2769 direction="outgoing" msg="URL has been visited" ratemethod="ip" cat=255 rawdata="[REQ] test_request_header=aaaaa"

    Log 1 is for the blocked HTTP response that contains both monitored headers, test_request_header and ETag, and their values, aaaaa and 34-5b23b9d3b67f4, respectively. Log 2 is for the HTTP request passing through the FortiGate proxy that contains test_request_header and its aaaaa value in the rawdata field.

Example 2: logging the explicit web proxy forward server name

In this example, the user wants to see the name of the web proxy forward server in the traffic log when the traffic is forwarded by a web proxy forward server.

In the global web proxy settings, log-forward-server must be enabled.

The following settings are required in the firewall policy:

  • set inspection-mode proxy

  • set webproxy-forward-server "fgt-b"

  • set logtraffic all

When a HTTP request is sent through the FortiGate proxy, the request will be forwarded by the FortiGate to the upstream proxy (fgt-b), and the forward server's name will be logged in the traffic log.

To verify the configuration:
  1. Send a HTTP request from the client:

    curl -kv https://www.google.com
  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=19:51:33 eventtime=1681959093510003961 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=49762 srcintf="port10" srcintfrole="undefined" dstip=142.250.217.100 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="United States" sessionid=43292 proto=6 action="client-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=49762 duration=120 sentbyte=0 rcvdbyte=37729 sentpkt=0 rcvdpkt=33 appcat="unscanned" wanin=3779 wanout=682 lanin=879 lanout=36005 fwdsrv="fgt-b" utmaction="block" countssl=1 utmref=65506-14

Example 3: logging TCP connection failures

In this example, a client initiates a TCP connection to a remote network node through the FortiGate. The connection fails because the IP address or port of the remote node is unreachable. A Connection Failed message appears in the logs. In the firewall policy configuration, the inspection-mode can be set to either proxy or flow mode.

Note

Based on the basic FortiGate configuration used in examples 1 and 2, the forward server may need to be removed from the firewall policy if the forward server's TCP IP port is actually reachable. If the forward server proxy tries to set up back-to-back TCP connections with the downstream FortiGate and the remote server as in the case of deep-inspection, then when the client tries to connect to a remote node (even if the IP address or port is unreachable), the downstream FortiGate is able to establish a TCP connection with the upstream forward server, so there will be no Connection Failed message in the downstream FortiGate's log.

Note

Currently, the Connection Failed message in the downstream FortiGate's log is visible for the case when there is an unreachable TCP port only when explicit web proxy with a proxy policy is configured. Therefore, the following example that makes use of a firewall policy demonstrates this log message is only supported for the unreachable IP address case.

To verify the configuration:
  1. Send a HTTP request from the client to an unreachable IP:

    curl -kv https://172.16.200.34
  2. On the FortiGate, check the traffic logs:

    # execute log filter category 3
    1: date=2023-04-19 time=20:25:55 eventtime=1681961155100007061 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=52452 srcintf="port10" srcintfrole="undefined" dstip=172.16.200.34 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="Reserved" sessionid=44903 proto=6 action="server-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=52452 duration=20 sentbyte=180 rcvdbyte=164 sentpkt=3 rcvdpkt=3 appcat="unscanned" wanin=0 wanout=0 lanin=0 lanout=0 crscore=5 craction=262144 crlevel="low" msg="Connection Failed"

Example 4: logging HTTP transaction details

HTTP transaction details are logged in a traffic log when HTTP traffic is routed through a proxy. After an HTTP transaction is proxied through the FortiGate, traffic logs of the http-transaction subtype are generated in addition to the forward subtype log. HTTP transaction logs are based on each transaction, such as an HTTP request and response pair. When there are multiple HTTP transactions completed over the TCP connection there will be multiple http-transaction logs and only one forward traffic log.

HTTP transaction logging can be enabled in explicit-web proxy, transparent-web proxy, access-proxy, and proxy-mode firewall policies.

config firewall proxy-policy
    edit 1
        set proxy {explicit-web | transparent-web | access-proxy}
        logtraffic {utm | all}
        set log-http-transaction {enable | disable}
    next
end
config firewall policy
    edit 1
        set inspection-mode proxy
        logtraffic {utm | all}
        set log-http-transaction {enable | disable}
    next
end

One http-transaction log is generated for each HTTP transaction. A TCP connection can have multiple HTTP transactions, so there can be multiple http-transaction logs for one forward traffic log.

  • Explicit-web proxy logs:

    In the http-transaction logs (logs 2 and 3), transaction information such as httpmethod and statuscode are recorded.

    1: date=2024-05-21 time=20:06:17 eventtime=1716347177537010993 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=80 dstintf="port3" dstintfrole="undefined" sessionid=316483733 service="HTTP" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=12204 duration=30 wanin=760 rcvdbyte=760 wanout=211 lanin=163 sentbyte=163 lanout=36194 appcat="unscanned" utmaction="block" countav=1 utmref=65515-14
    
    2: date=2024-05-21 time=20:06:17 eventtime=1716347177536946272 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 dstip=172.16.200.44 dstport=80 sessionid=316483733 transid=50331679 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/eicar.com" agent="curl/7.68.0" duration=0 reqlength=86 resplength=392 rcvdbyte=760 sentbyte=163 scheme="http" hostname="172.16.200.44" resptype="cached" httpmethod="GET"statuscode="403" reqtime=1716347177 resptime=0 respfinishtime=1716347177 appcat="unscanned" utmaction="block" countav=1 utmref=65515-0
    3: date=2024-05-21 time=20:06:06 eventtime=1716347166400042072 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=42694 dstip=172.16.200.44 dstport=80 sessionid=316483733 transid=50331678 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=77 resplength=368 rcvdbyte=368 sentbyte=77 scheme="http" hostname="172.16.200.44" resptype="normal" httpmethod="GET"statuscode="200" reqtime=1716347166 resptime=1716347166 respfinishtime=1716347166 appcat="unscanned"

    When the EICAR test file in the response is blocked by utm-av, utmref information referring to the corresponding utm-av log is included:

    # execute log detail 2 "65515-0"
    1 logs found.
    1 logs returned.
    1: date=2024-05-21 time=20:06:17 eventtime=1716347177536848145 tz="-0700" logid="0211008192" type="utm" subtype="virus" eventtype="infected" level="warning" vd="vdom1" policyid=1 poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" policytype="proxy-policy" msg="File is infected." action="blocked" service="HTTP" sessionid=316483733 transid=50331679 srcip=10.1.100.11 dstip=172.16.200.44 srcport=42694 dstport=80 srccountry="Reserved" dstcountry="Reserved" srcintf="port1" srcintfrole="undefined" dstintf="port3" dstintfrole="undefined" proto=6 direction="incoming" filename="eicar.com" quarskip="Quarantine-disabled" virus="EICAR_TEST_FILE" viruscat="Virus" dtype="av-engine" itype="infected" ref="https://fortiguard.com/encyclopedia/virus/2172" virusid=2172 url="http://172.16.200.44/eicar.com" profile="av" agent="curl/7.68.0" httpmethod="GET" analyticssubmit="false" crscore=50 craction=2 crlevel="critical"
  • Forward traffic log and http-transaction logs for transparent-web proxy policy, access-proxy proxy policy, and proxy-mode firewall policy:

    • Transparent-web proxy policy:

      1: date=2024-05-23 time=23:22:36 eventtime=1716531756508124889 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=34326 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=90954 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=2 policytype="proxy-policy" poluuid="1f986ef4-14c1-51ef-7d4f-fd482cefde73" trandisp="snat" transip=172.16.200.8 transport=34326 duration=0 wanin=3051 rcvdbyte=3051 wanout=618 lanin=842 sentbyte=842 lanout=4551 appid=40568 app="HTTPS.BROWSER" appcat="Web.Client" apprisk="medium" utmaction="allow" countssl=2 utmref=65516-0
      
      2: date=2024-05-23 time=23:22:36 eventtime=1716531756507534974 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=34326 dstip=172.16.200.44 dstport=443 sessionid=90954 transid=50331679 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="accept" policyid=2 policytype="proxy-policy" poluuid="1f986ef4-14c1-51ef-7d4f-fd482cefde73" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=787 scheme="https" hostname="172.16.200.44" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716531756 resptime=1716531756 respfinishtime=1716531756 appcat="unscanned"
    • Access-proxy proxy policy:

      1: date=2024-05-23 time=23:24:10 eventtime=1716531850437771247 tz="-0700" logid="0005000024" type="traffic" subtype="ztna" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=51076 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=91243 service="HTTPS" proxyapptype="http" proto=6 action="accept" policyid=3 policytype="proxy-policy" poluuid="2853b1ba-195e-51ef-2e7c-93b136b9505a" duration=0 gatewayid=1 realserverid=1 vip="vip-ztna" accessproxy="ztna" clientdevicemanageable="unknown" clientcert="no" wanin=2981 rcvdbyte=2981 wanout=712 lanin=824 sentbyte=824 lanout=2000 appcat="unscanned"
      
      2: date=2024-05-23 time=23:24:10 eventtime=1716531850436959338 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=51076 dstip=172.16.200.44 dstport=443 sessionid=91243 transid=50331680 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="accept" policyid=3 policytype="proxy-policy" poluuid="2853b1ba-195e-51ef-2e7c-93b136b9505a" url="https://a.ftnt.com/" agent="curl/7.68.0" duration=0 reqlength=36 resplength=185 rcvdbyte=2981 sentbyte=800 scheme="https" hostname="a.ftnt.com" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716531850 resptime=1716531850 respfinishtime=1716531850 appcat="unscanned"
    • Proxy-mode firewall policy:

      1: date=2024-05-23 time=23:32:11 eventtime=1716532331148593220 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=49424 srcintf="port1" srcintfrole="undefined" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" srccountry="Reserved" dstcountry="Reserved" sessionid=92757 proto=6 action="close" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" service="HTTPS" trandisp="snat" transip=172.16.200.8 transport=49424 duration=1 sentbyte=1406 rcvdbyte=5235 sentpkt=15 rcvdpkt=13 appcat="unscanned" wanin=3051 wanout=618 lanin=842 lanout=4551 utmaction="allow" countssl=2 utmref=65511-0
      
      2: date=2024-05-23 time=23:32:10 eventtime=1716532330081764186 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=49424 dstip=172.16.200.44 dstport=443 sessionid=92757 transid=50331681 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" url="https://172.16.200.44/" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=787 scheme="https" hostname="172.16.200.44" resptype="normal" reqtime=1716532330 resptime=1716532330 respfinishtime=1716532330 appcat="unscanned"

For HTTPS with explicit-web proxy, there is an additional piece of http-transaction log for each CONNECT request and response:

3: date=2024-05-21 time=20:34:44 eventtime=1716348884524284243 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=46030 dstip=172.16.200.44 dstport=443 sessionid=316483736 transid=50331683 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716348884 resptime=0 respfinishtime=1716348884 appcat="unscanned"

For HTTPS with certificate-inspection or no inspection, there is only one http-transaction log for each TCP connection because the encrypted HTTP messages are not decrypted:

  • Firewall policy:

    2: date=2024-05-23 time=21:38:56 eventtime=1716525535340969183 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=46462 dstip=172.16.200.44 dstport=443 sessionid=70593 transid=1 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" policyid=1 policytype="policy" poluuid="1ab9cbbc-14c1-51ef-55cd-4b90ff9a117a" url="172.16.200.44" duration=0 reqlength=842 resplength=3100 rcvdbyte=3100 sentbyte=842 scheme="https" hostname="172.16.200.44" resptype="N/A" reqtime=1716525535 resptime=1716525535 respfinishtime=1716525535 appcat="unscanned"
  • Explicit-web proxy policy:

    2: date=2024-05-23 time=21:36:18 eventtime=1716525378802239534 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=56986 dstip=172.16.200.44 dstport=443 sessionid=1369348242 transid=50331673 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716525378 resptime=0 respfinishtime=1716525378 appcat="unscanned"

For SOCKS proxy, there is one http-transaction log for each HTTP transaction per TCP connection:

1: date=2024-05-23 time=22:50:34 eventtime=1716529833463518327 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63744 dstip=34.107.221.82 dstport=80 sessionid=1369348358 transid=117441403 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://detectportal.firefox.com/success.txt?ipv4" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0" duration=0 reqlength=305 resplength=216 rcvdbyte=7128 sentbyte=20143 scheme="http" hostname="detectportal.firefox.com" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716529833 resptime=1716529833 respfinishtime=1716529833 appcat="unscanned"
2: date=2024-05-23 time=22:49:40 eventtime=1716529779381260298 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63842 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=80 dstintf="port3" dstintfrole="undefined" sessionid=1369348430 service="webproxy" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=16135 duration=0 wanin=392 rcvdbyte=392 wanout=356 lanin=725 sentbyte=725 lanout=71832 appcat="unscanned" utmaction="block" countav=1 utmref=65517-14
3: date=2024-05-23 time=22:49:40 eventtime=1716529779381204800 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.143 srcport=63842 dstip=172.16.200.44 dstport=80 sessionid=1369348430 transid=117441401 srcuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" dstuuid="bdba900e-14c0-51ef-1328-c1b8329857ef" action="pending" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="http://172.16.200.44/eicar.com" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0" duration=0 reqlength=356 resplength=392 rcvdbyte=392 sentbyte=725 scheme="http" hostname="172.16.200.44" resptype="cached" httpmethod="GET" statuscode="403" reqtime=1716529779 resptime=0 respfinishtime=1716529779 appcat="unscanned" utmaction="block" countav=1 utmref=65517-0

UTM logs that do not belong to an HTTP transaction are only associated with the forward traffic log, and not the http-transaction log:

Only the forward traffic log is associated to the utm-ssl log by the utmref.

  • forward traffic log:

    1: date=2024-05-23 time=22:01:43 eventtime=1716526903039789335 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.16.200.44 dstport=443 dstintf="port3" dstintfrole="undefined" sessionid=1369348243 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" trandisp="snat" transip=172.16.200.8 transport=1837 duration=0 wanin=3051 rcvdbyte=3051 wanout=618 lanin=960 sentbyte=960 lanout=4623 appcat="unscanned" utmaction="allow" countssl=2 utmref=65523-0
  • http-transaction logs:

    2: date=2024-05-23 time=22:01:43 eventtime=1716526903038941661 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 dstip=172.16.200.44 dstport=443 sessionid=1369348243 transid=50331675 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=38 resplength=185 rcvdbyte=3051 sentbyte=936 scheme="https" hostname="172.16.200.44" resptype="normal" httpmethod="GET" statuscode="200" reqtime=1716526903 resptime=1716526903 respfinishtime=1716526903 appcat="unscanned"
    
    3: date=2024-05-23 time=22:01:43 eventtime=1716526903015933382 tz="-0700" logid="0006000026" type="traffic" subtype="http-transaction" level="notice" vd="vdom1" srcip=10.1.100.11 srcport=57928 dstip=172.16.200.44 dstport=443 sessionid=1369348243 transid=50331674 action="accept" policyid=1 policytype="proxy-policy" poluuid="1e1e0b2e-14c1-51ef-7b4c-6b789be487f2" url="https://172.16.200.44/" agent="curl/7.68.0" duration=0 reqlength=118 resplength=0 rcvdbyte=0 sentbyte=118 scheme="https" hostname="172.16.200.44" resptype="generated" httpmethod="CONNECT" statuscode="200" reqtime=1716526903 resptime=0 respfinishtime=1716526903 appcat="unscanned"