Exporting Events from FortiSIEM

The following tools are provided:

phExportESEvent Tool
phExportEvent Tool
TestESSplitter Tool
phClickHouseCSVExport Tool

phExportESEvent Tool

Description: This tool exports events from Elasticsearch into a CSV file.

Usage: phExportESEvent <ESUrl> <ESPort> <ESDeploymentType> "<ESUser>" "<ESPassword>" <ESIndexName> <ReportingDevIp> <destDir> <splitThreads> <LogLevel>

Argument	Description
`ESUrl`	The Elasticsearch URL. Example, http://192.0.2.0.
`ESPort`	The Elasticsearch coordinating node port, e.g. 9200.
`ESType`	Provide the Elasticsearch type. 1: Native 2: AWS Elasticsearch Service 3: Elasticsearch Cloud
`ESUser`	Provide the Elasticsearch username. "" means no username.
`ESPassword`	Provide the Elasticsearch password. "" means no password.
`ESIndexName`	The name of the Elasticsearch index to be exported, for example, `fortisiem-event-2020.06.17-1`.
`ReportDevIp`	The IP address of the report device to be used to select events to export. "" means select all devices.
`destDir`	The export directory: `output_dir`.
`splitThreads`	The number of threads to be used for export, e.g., 10.
`logDevel`	The debug level for script output printing: `INFO` or `DEBUG`.

Notes:

Can be run from Supervisor or Worker.
Can be run as admin user.

Examples:

Native Elasticsearch Deployment Example

phExportESEvent https://192.0.2.0 9200 1 "Joe.123--test" "password" fortisiem-event-2021.08.05-1-000001 "192.0.2.4" /archive/ 10 INFO

AWS Elasticsearch Service Deployment Example

phExportESEvent https://search-eesna78-aaaa4ysukru3ui4ayaz2yya3km.us-east-1.es.amazonaws.com 443 2 "key" "secret" fortisiem-event-2021.09.29-1 "" /archive/ 10 INFO

Elasticsearch Cloud Deployment Example

phExportESEvent https://cpaagg33-d11e01.es.us-central1.gcp.cloud.es.io 9243 3 "elastic" "password" fortisiem-event-2021.10.01-1-000001 "" /archive/ 10 INFO

phExportEvent Tool

Description: This tool exports events from EventDB into a CSV file. The CSV file contains the following columns:

Customer Id (applicable to SP license)
Reporting Device IP
Reporting Device Name
Event Received Time
Raw Message

Usage: phExportEvent {--dest DESTINATION_DIR} {--starttime START_TIME | --relstarttime RELATIVE_START_TIME} {--endtime END_TIME | --relendtime RELATIVE_END_TIME} [--dev DEVICE_NAME] [--org ORGANIZATION_NAME] [-t TIME_ZONE]

Argument	Description
`DESTINATION_DIR`	Destination directory where the exported event files are saved.
`START_TIME`	Starting time of events to be exported. The format is YYYY-MM-DD HH:MM:SS {+\|-} TZ. If TZ is not given, the local time zone of the machine where the script is running will be used. Example: `2010-03-10 23:00:00 -8` means Pacific Standard Time, 23:00:00 03/10/2010. `2010-07-29 10:20:00 +5:30` means India Standard Time 10:20:00 07/29/2010.
`RELATIVE_ START_TIME`	This must be used together with `END_TIME`. Starting time of events to be exported is relative backwards to the end time, specified using `--endtime END_TIME`. The format is `{NUM}{d\|h\|m}` where `NUM` is the number of days or hours or minutes. For example, `-- relstarttime 5d` means the starting time is 5 days prior to the ending time.
`END_TIME`	Ending time of events to be exported. The format is the same as described for `START_TIME`.
`RELATIVE_END_ TIME`	This must be used together with `START_TIME`. Ending time of events to be exported is relative forward to the start time, specified using `START_TIME`. The format is the same that is used for RELATIVE_START_TIME.
`DEVICE_NAME`	Provide the host name or IP address of the device with the events to be exported. Use a comma-separated list to specify multiple IPs or host names, for example, `--dev 10.1.1.1,10.10.10.1,router1,router2`. Host name is case insensitive.
`ORGANIZATION_ NAME`	This is used only for Service Provider deployments. Provide the name of the organization with the events to be exported. To specify multiple organizations, enter a command for each organization, for example, `--org "Public Bank" --org "Private Bank"`. The organization name is case insensitive.
`TIME_ZONE`	Specifies the time zone used to format the event received time in the exported event files. The format is `{+\|-}TZ`, for example, `-8` means Pacific Standard Time, `+5:30` means India Standard Time.

Notes:

Can be run from Supervisor or Worker.
Can be run as admin user.

TestESSplitter Tool

Description: This tool exports events from ElasticSearch to a directory in FortiSIEM EventDB format.

Usage: TestESSplitter <ESBroker> <ESPort> <ESClusterType> <ESUser> <ESPassword> <IndexName> <destDir> <splitThreads> <logLevel>

Argument	Description
`ESBroker`	The IP of ElasticSearch Co-ordinator node.
`ESPort`	The port used for ElasticSearch.
`ESClusterType`	The ElasticSearch Cluster type. Values are "1" for Native, "2" for Amazon OpenSearch Service (previously known as Amazon Elasticsearch Service), and "3" for Elastic Cloud.
`ESUser`	The ElasticSearch username for authentication.
`ESPassword`	The ElasticSearch password for authentication.
`IndexName`	Provide an Index name. A new Index is created per day. Here is an example index name, `fortisiem-event-2021.05.14-2000-000001` where“fortisiem-event-2021.05.14” is the day and “2000” is the Organization ID. To find a list of indexes, run this command: `curl -XGET '10.10.2.5:9200/_cat/shards?v'` replacing `10.10.2.5` with the IP of a Co-ordinator node.
`destDir`	Destination directory where the exported events are saved in FortiSIEM eventDB format. Note: A trailing slash is mandatory. Example: `https://<destDir>/`.
`splitThreads`	Number of threads.
`logLevel`	INFO or DEBUG level log messages.

Notes:

Can be run from Supervisor or Worker.
Can be run as admin user.
This tool is located in /opt/phoenix/bin/.

Example:

[root@fsm]# /opt/phoenix/bin/TestESSplitter 10.10.2.5 "" "" fortisiem-event-2021.05.14-2000-000001 /root/output 10 INFO
 
[PH_MODULE_LOG_LEVEL_CHANGE]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phBaseProcess.cpp,[lineNumber]=675,[oldLogLevel]=2047,[newLogLevel]=424,[phLogDetail]=Module received log level change
[PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=global,[phLogDetail]=Module loaded local config successfully
[PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=phdatamanager,[phLogDetail]=Module loaded local config successfully
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phHttpClientPool.cpp,[lineNumber]=46,[phLogDetail]=phHttpClientPool: init hosts/port/auth/header=10.10.2.5/9200/:****/Content-Type: application/json
*   Trying 10.10.2.5...
* TCP_NODELAY set
* Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0)
> GET / HTTP/1.1
Host: 10.10.2.5:9200
Accept: */*
Content-Type: application/json
 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 530
< 
* Connection #0 to host 10.10.2.5 left intact
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1732,[phLogDetail]=Elastic init success: http://10.10.2.5:9200/
* Found bundle for host 10.10.2.5: 0x18f0870 [can pipeline]
* Re-using existing connection! (#0) with host 10.10.2.5
* Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0)
> GET /_cat/indices/fortisiem-event-2021.05.14-2000-000001?h=pri,rep,docs.count HTTP/1.1
Host: 10.10.2.5:9200
Accept: */*
Content-Type: application/json
…
…
…
…
 
< 
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 66 for index fortisiem-event-2021.05.14-2000-000001 slice 1 max 10
* Connection #0 to host 10.10.2.5 left intact
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 61 for index fortisiem-event-2021.05.14-2000-000001 slice 8 max 10
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 47737
< 
* Connection #0 to host 10.10.2.5 left intact
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 3 max 10
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 47178
< 
* Connection #0 to host 10.10.2.5 left intact
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 41910
< 
* Connection #0 to host 10.10.2.5 left intact
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 53258
< 
* Connection #0 to host 10.10.2.5 left intact
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 60587
< 
* Connection #0 to host 10.10.2.5 left intact
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 59 for index fortisiem-event-2021.05.14-2000-000001 slice 4 max 10
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 7 max 10
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 68 for index fortisiem-event-2021.05.14-2000-000001 slice 6 max 10
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 46 for index fortisiem-event-2021.05.14-2000-000001 slice 2 max 10
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds
[PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=TestESSplitter.cpp,[lineNumber]=82,[phLogDetail]=Events processed for split: 559 3.15

The result will be eventDB structured directories and files.

[root@fsm]# ls -l /root/output/
total 0
drwx------ 3 root root 22 May 14 15:25 CUSTOMER_2000
[root@fsm]# ls -l /root/output/CUSTOMER_2000/
total 0
drwx------ 3 root root 19 May 14 15:25 internal
[root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/
total 0
drwx------ 3 root root 37 May 14 15:25 18761
[root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/
total 4
drwx------ 12 root root 4096 May 14 15:25 450264-450287-168428094
[root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/450264-450287-168428094/
total 0
drwx------ 3 root root 18 May 14 15:25 seg-1-0-48-1620951010-1620971132
drwx------ 3 root root 18 May 14 15:25 seg-1-1-70-1620950470-1620971172
drwx------ 3 root root 18 May 14 15:25 seg-1-2-35-1620950916-1620971172
drwx------ 3 root root 18 May 14 15:25 seg-1-3-66-1620951819-1620969371
drwx------ 3 root root 18 May 14 15:25 seg-1-4-61-1620950830-1620970642
drwx------ 3 root root 18 May 14 15:25 seg-1-5-59-1620950830-1620971132
drwx------ 3 root root 18 May 14 15:25 seg-1-6-53-1620950482-1620970632
drwx------ 3 root root 18 May 14 15:25 seg-1-7-46-1620951278-1620971182
drwx------ 3 root root 18 May 14 15:25 seg-1-8-53-1620950470-1620970452
drwx------ 3 root root 18 May 14 15:25 seg-1-9-68-1620950650-1620971132

phClickHouseCSVExport Tool

Description: This tool exports events from ClickHouse into a CSV file. The file will contain these fields:

Event Receive Time
Reporting IP
Event Type
Raw Event Log.

Usage: phClickHouseCSVExport --starttime [Start Time] --endtime [End Time] --outfile [Output file] --deviceip [Reporting Device IP Address] --devicename [Reporting Device Name] --limit [Number of records to return] --debug --orgid [Organization ID (0 - 4294967295)] --orgname [Organization Name] --eventtype [Event Type]

Argument	Description
`--starttime [Start Time]`	Starting time of events to be exported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the Start Time with quotation marks. Example: `phClickHouseCSVExport --outfile /home/user/report.csv --starttime "2022-01-20 10:10:00" --endtime "2022-01-20 11:10:00"`
`--endtime [End Time]`	The end time of events to be exported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the End Time with quotation marks. Example: `phClickHouseCSVExport --outfile /home/user/report.csv --starttime "2022-01-20 10:10:00" --endtime "2022-01-20 11:10:00"`
`--outfile [Output file]`	The output file where the exported events are saved in FortiSIEM, CSV format.
`--deviceip [Reporting Device IP Address]`	Provide the IP address of the device with the events to be exported. Only one reporting device IP address is supported. For example, `--deviceip 10.1.1.1`.
`--devicename [Reporting Device Name]`	Provide the host name of the device with the events to be exported. For example, `--devicename router1`. Host name is case insensitive.
`--limit [Number of records to return]`	Provide the number of records to return, default is no limit.
`--debug`	Output SQL statement generated for this report.
`--orgid [Organization ID]`	Provide the ID of the organization with the events to be exported. The number can be from 0 to 4294967295.
`--orgname [Organization Name]`	This is used only for Service Provider deployments. Provide the name of the organization with the events to be exported. To specify multiple organizations, enter a command for each organization, for example, `--org "Public Bank" --orgname "Private Bank"`. The organization name is case insensitive.
`--eventtype [Event Type]`	Specify the event types to be exported.

Notes:

Can be run from Supervisor or Worker.
Can be run as admin user.