Improve log forwarding bandwidth efficiency

FortiAnalyzer supports a new option to allow log data to be compressed for bandwidth optimization when forwarding the logs to a remote server in FortiAnalyzer format.

Log messages will be compressed when this feature is enabled and both FortiAnalyzer devices support the log compression feature. When log forwarding is configured with compression enabled to a remote FortiAnalyzer that does not support compression, the logs will remain uncompressed.

To enable compression in log forwarding:

1. Go to System Settings > Log Forwarding, and click Create New.
2. Select FortiAnalyzer as the Remote Server Type, and configure the server settings for your remote FortiAnalyzer.
3. Set the Compression setting toggle to the ON position. It is set to OFF by default.
   
4. Click OK to save the log forwarding configuration.

Configuring log compression in the CLI

The following CLI setting has been added for log compression:

# set fwd-compression {enable|disable}

Following is an example of log forward configuration in the CLI:

config system log-forward

edit 3

set mode forwarding

set fwd-max-delay realtime

set server-name "demo"

set server-addr "10.2.125.244"

set fwd-reliable enable

set fwd-compression enable

set sync-metadata sf-topology interface-role device endusr-avatar

set signature 6723252594909515930

next

end

Diagnosing log forward compression

The log format is displayed in diagnose test application logfwd 3 and the compression ratio is displayed with diagnose test application logfwd 4.

To view the log format:

1. In the FortiAnalyzer CLI, enter the following command:
   

   diagnose test application logfwd 3

   The output will include information about the log format.

   #2: 244 => FortiAnalyzer @ 10.2.125.244:514 token=715983816682025708 Reliable Running Updt=1610129597

   tlvm-ver=2 logfwd-ver=1 logfmt=SiedLog compress

   Grp=ld-244 Qid=21 Updt=1610129598 Hash=1.115f51236d8e2a20.0.0

   - Dev-filter: FG100D3G00002901,FG100D3G00002900

To view the compression ratio:

1. In the FortiAnalyzer CLI, enter the following command:
   

   diagnose test application logfwd 4

   The output will include information about the compression ratio.

   ** Server#1: 244 ld-244 Qid=21 Connected bind: from 16m42s ago

   nmsg-sent=9978 nlog-sent=452083 send_timeout=0 send_err=1

   conn_err=9 msg_append_err=0 unreliable-errno=0

   nbytes-sent=22781160 compress-ratio=82.1%

   rate in last 5sec, 30sec, 60sec

   msg/sec: 6.0 6.0 5.9

   log/sec: 280.4 290.2 285.2

   In FortiAnalyzer 7.0.1, when log compression is enabled for the FortiAnalyzer log format, the FortiAnalyzer daemon will decide whether or not to compress the message based on the type of logs being forwarded. If all logs in the current buffer are in the lz4 format, then the compression will be skipped due to the compression efficiency being too low. Compression will only be performed when the buffer contains text logs or older compact logs. The compression ratio is not displayed when compression is not performed for the current server.

The remote analyzer with this feature displays received compressed forwarded logs in diagnose test application oftpd 7.

To view received compressed logs:

1. In the FortiAnalyzer CLI, enter the following command:
   

   diagnose test application oftpd 7

   The output will include information about received compressed logs within the log-forward gen2 stats section.

   FAZVM64 # diagnose test application oftpd 7

   Reliable logging stats:

   log=547 log(>4k)=36

   Reliable log-forward stats:

   log=0 log(>4k)=0 reg=0 ack=0 ack_back=0 thr=0 optcode_err=0

   Reliable log-forward gen2 stats:

   Connections:

   From FAZ-VMTM20009184 @ 10.2.125.245 sig.745f02f721e21529 Connected 5m22.181s ago

   Pos=1610387635.768239362.24429530.7 tlvm-ver=2 last_recv=1610387153 n_flushed=2457 n_compressed=2457

   Stats:

   add=1 del=0 replace=0

   inactive=0 expired=0

   Errors:

   conn=0 conn_info=0 discard=0

   epoll.add=0 epoll.del=0

   rcv_tlvm=0 rcv_oversize=0 parse_msg=0 build_resp=0

   Internal log-forward stats:

   queued=0 (max=2048) update=757 (now=759)

   errors

   fortilogd-not-running=0 no-init=0 socket=0 no-recv=0 unknown=0

   Internal-forward stats by source:

   dev-nonreliable : 0

   fwd-reliable : 2457

   fwd-nonreliable : 0

   dev-batch-upload : 0

   fct-batch-upload : 0

   dev-reliable : 0

   fwd-reliable-unencrypted : 0

   fwd-ha-isync : 0

   fwd-ha-isync-ack : 0

   dev-reliable-encrypted : 547

   fna-upload : 0

   faz-appevt : 0

   fct-siem : 0

   unknown : 0