External resources for web filter

External Resources is a new feature introduced in FortiOS 6.0, which provides a capability to import an external blacklist which sits on an HTTP server. This feature helps FortiGate retrieve a dynamic URL/Domain Name/IP Address/Malware hash list from an external HTTP server periodically. FortiGate uses these external resources as Web Filter's remote categories, DNS Filter's remote categories, policy address objects or antivirus profile's malware definitions. If the external resource is updated, FortiGate objects will update dynamically.

External Resource are categorized into 4 types:

• URL list (Type=category)
• Domain Name List (Type=domain)
• IP Address list (Type=address)
• Malware hash list (Type=malware)

For Web Filter profile, it can use category type external resources. Category type external resources file is a URL entries list in a plain text file.

When a category type external resource is configured in Web Filter profile, it will be treated as a Remote Category. If the URL in a HTTP/HTTPS request matches the entry inside this external resource file, it will be treated as the Remote Category and follow the action configured for this category in Web Filter profile.

External resource type category also can be used in ssl-ssh-profile configuration for category-based SSL-Exempt. When a Remote Category is configured in ssl-ssh-profile SSL-Exempt, if a HTTPS request's URL matches in the Remote Category’s entry list, HTTPS request with destination for this URL can be exempted from SSL Deep Inspection.

External Resources File Format

External Resources File should follow the following requirements:

• The external resource file is a plain text format file and each URL list/IP Address/Domain Name occupies a single line.
• The file is limited to 10M, line is limited 128K (128 x 1024 entries), and the line length limit is 4K characters.
• The entries limited also follow table size limitation defined by CMDB per model.
• The external resource update period can be set to 1 minute, hourly, daily, weekly, or monthly (43200 min, 30 days).
• The external resource type as category (URL list) and domain (Domain Name list) share the category number range 192-221 (total 30 categories).
• There's no duplicated entry validation for external resources file (entry inside each file or inside different files).

For URL list (Type=category):

Scheme is optional, and will be truncated if found (http://, https:// is not needed).

Wildcard (*) is supported (from 6.2). It supports the '*' at beginning and ending of URL, and not in the middle of URL as follows:

+ support *.domain2.com, domain.com.* + not support: domain3.*.com

IDN (International Domain Name) and UTF encoding URL is supported (from 6.2).

IPv4,IPv6 format URL is supported. IPv6 in URL list must in [ ] form.

Configure External Resources from CLI

We can use CLI to configure the external resources files that is located on external HTTP Server. Under Global, configure the external resource file location and specify the resource type.

Web Filter will use category type external resources as Remote Categories. In the following example, it is configured a file Ext-Resource-Type-as-Category-1.txt as type as category, it will be treated in Web Filter as Remote Category, the category name configured as Ext-Resource-Type-as-Category-1 and category-id as 192:

config system external-resource
   edit "Ext-Resource-Type-as-Category-1"
     set type category  <----
     set category 192   <----
     set resource "http://172.16.200.66/external-resources/Ext-Resource-Type-as-Category-1.txt"
     set refresh-rate 1
   next
end   

Now in each VDOM, category type external resource can be used in Web Filter as Remote Cateogry. In the example above, URL list in "Ext-Resource-Type-as-Category-1.txt" file will be treated as remote category (category-id 192). Configure the action for this remote category in Web Filter profile and apply it in the policy:

config webfilter profile
    edit "webfilter"
        config ftgd-wf
            unset options
            config filters
                edit 1
                    set category 2
                    set action warning
                next
                ......
                
                edit 24
                    set category 192  <----
                    set action block
                next
                edit 25
                    set category 221
                    set action warning
                next
                edit 26
                    set category 193
                next
            end
        end
        set log-all-url enable
    next
end

config firewall policy
    edit 1
        set name "WebFilter"
        set srcintf "port10"
        set dstintf "port9"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set utm-status enable
        set logtraffic all
        set webfilter-profile "webfilter"
        set profile-protocol-options "protocol"
        set ssl-ssh-profile "protocols"
        set nat enable
    next
end

Configure External Resources from GUI

To configure, edit, or view the Entries for external resources from GUI:

1. Go to Global > Security Fabric > Fabric Connectors:

   

2. Go to Global > Security Fabric > Fabric Connectors.
3. Click Create New, and in the Threat Feeds section, select FortiGuard Category.

   

4. Enter the resource name, URI location of the resource file, resource authentication credential, and Refresh Rate.

   

5. Click OK.
6. After a few minutes, double-click the Threat Feeds Object you just configured. It is shown in the Edit page.
7. Click View Entries to view the entry list in the external resources file:

   

8. Go to VDOM > Security Profiles > Web Filter. The configured external resources is shown and configured in each Web Filter Profile:

   

Log Example

If an HTTP/HTTPS request URL is matched in remote category's entry list, it will override its original FortiGuard URL rating and be treated as a remote category.

Go to VDOM > Log & Report > Web Filter:

CLI Example:

1: date=2019-01-18 time=15:49:15 logid="0316013056" type="utm" subtype="webfilter" eventtype="ftgd_blk" level="warning" vd="vdom1" eventtime=1547855353 policyid=1 sessionid=88922 srcip=10.1.100.18 srcport=39886 srcintf="port10" srcintfrole="undefined" dstip=216.58.193.67 dstport=443 dstintf="port9" dstintfrole="undefined" proto=6 service="HTTPS" hostname="www.fortinet.com" profile="webfilter" action="blocked" reqtype="direct" url="/" sentbyte=752 rcvdbyte=10098 direction="outgoing" msg="URL belongs to a denied category in policy" method="domain" cat=192 catdesc="Ext-Resource-Type-as-Category-1"

Remote Category in ssl-ssh-profile category-based SSL-Exempt

Remote category can be applied in ssl-ssh-profile category-based SSL-Exempt.

Go to VDOM > Security Profiles > SSL/SSH Inspection:

HTTPS request URLs matched in this remote category will be exempted from SSL deep inspection.

Log example:

3: date=2019-01-18 time=16:06:21 logid="0345012688" type="utm" subtype="webfilter" eventtype="ssl-exempt" level="information" vd="vdom1" eventtime=1547856379 policyid=1 sessionid=90080 srcip=10.1.100.18 srcport=39942 srcintf="port10" srcintfrole="undefined" dstip=216.58.193.67 dstport=443 dstintf="port9" dstintfrole="undefined" proto=6 service="HTTPS" hostname="www.fortinet.com" profile="webfilter" action="passthrough" reqtype="direct" url="/" sentbyte=517 rcvdbyte=0 direction="outgoing" msg="The SSL session was exempted." method="domain" cat=192 catdesc="Ext-Resource-Type-as-Category-1" urlsource="exempt_type_user_cat"

Local Category and Remote Category Priority

Web Filter can have both local category and remote category at the same time. There's no duplication check between local category URL override and remote category resource file. For example, a URL like www.example.com may be shown both in remote category entry list and in FortiGate's local category URL override configuration. We recommend avoiding this scenario since FortiGate does not check for duplicates. However, if a URL is duplicated in both local category and remote category, it is rated as local category.