Traffic Shaping Priority Queueing (PRIQ)

This traffic shaping document describes Priority Queueing (PRIQ), Type of Service (ToS) priority, and Quality of Service (QoS). It also explains the following:

• Why traffic shaping only occurs when traffic approaches the configured capacity on a given interface.
• Why you should configure the FortiGate unit to preemptively drop excess packets.
• How priority queues work on the FortiGate.
• The difference between ToS-based priority and global ToS priority.
• Why you must enable traffic shaping for ALL firewall policies to get expected results.
• How firewall policy priorities and ToS policies affect each other.
• Why traffic shaper priorities only effect per port egress queueing.

Any CLI commands and GUI references in this article have been tested for both FortiOS 5.2.5 and FortiOS 5.4, and any differences between versions will be documented.

How traffic shaping really works

One of the most common misconceptions with traffic shaping on your FortiGate is that setting a “priority” will ensure that high priority traffic will download faster than low priority traffic. This perfectly reasonable expectation does not fully encapsulate what “priority” means in FortiOS, which needs to be taken into consideration. Traffic shaping will only begin to take effect when an interface with traffic shaping configured reaches its capacity. Until this threshold is reached all traffic is treated equally. As the interface experiences high traffic levels that reach its threshold, you will begin to notice a variation in traffic flow or download speeds.

A screenshot of a shaper at capacity in the FortiView > Traffic Shaping section (FortiOS 5.4).

Before you begin

There are a few things you need to know about Traffic Shaping and priority queueing before we begin:

• Packets are prioritized based on their priority value.
• The priority value is based on whether you have configured Type of Service (ToS) priority and/or Firewall policy priority.
• The total priority value then determines which queue the packet is placed in, out of six queue options.

  Also, remember that only per port egress queueing works!

Other considerations that affect which queue is used include:

• Whether the traffic is through traffic or originates on the FortiGate.
• Whether traffic shaping is enabled in all your firewall policies.

Traffic shaping methods

When deciding how to configure QoS techniques, it can be helpful to know when FortiGate units employ each technique in the overall traffic processing flow, and the considerations that follow.

Dropping excess packets early on

As traffic arrives (ingress) and departs (egress) on an interface, the FortiGate unit begins to process the traffic. In later phases of network processing — such as enforcing maximum bandwidth on sessions handled by a security policy — if the current rate for the destination interface or traffic regulated by that policy is too high, the FortiGate unit may drop the packet. Time spent on prior processing — like web filtering, decryption, or IPS — is wasted on these dropped packets.

You can prevent wasted effort on ingress by configuring the FortiGate unit to preemptively drop excess packets when they are received at the source interface, before most other traffic processing is performed:

config system interface
    edit <interface_name>
        set inbandwidth <rate_int>
        set outbandwidth <rate_int>
    next
end

Where <rate_int> is set to the bandwidth limit in Kb/s, excess packets will be dropped. If the inbandwidth <rate_int> is set to 0, then the rate is not limited.

As with ingress, if you set the rate to 0 (zero) that means you are setting the rate to unlimited. Rate limiting traffic accepted by the interface enables you to restrict incoming traffic to rates that, while no longer the full capacity of the interface, at the traffic shaping point in the processing are more likely to result in acceptable rates of outgoing traffic per destination interface or all security policies. This conserves FortiGate processing resources for those packets likely to be viable (to the point of egress).

This diagram shows how excess packets going from WAN 1 can be intercepted and dropped at the source interface.

How priority queuing works

After packet acceptance, the FortiGate unit classifies traffic and may apply traffic policing at additional points during processing. It may also apply QoS techniques, such as prioritization and traffic shaping. Traffic shaping consists of a mixture of traffic policing to enforce bandwidth limits, and priority queue adjustment to assist packets in achieving the guaranteed rate.

If you have configured prioritization, the FortiGate unit prioritizes egressing packets by distributing them among FIFO (first in, first out) queues associated with each possible priority number. Each physical interface has six priority queues. Virtual interfaces use the priority queues of the physical interface to which they are bound.

Each physical interface’s six queues are queue 0 to queue 5, where queue 0 is the highest priority queue. However, you may observe that your traffic uses only a subset of those six queues. For example, some traffic may always use a certain queue number. Queuing may also vary by the packet rate or mixture of services. Some queue numbers may only be used by through traffic for which you have configured traffic shaping in the security policy that applies to that traffic session.

This diagram illustrates the following description:

• Administrative access traffic will always use queue 0.
• Traffic matching security policies without traffic shaping may use queue 0, queue 1, or queue 2. The queue is selected based on the priority value you have configured for packets with that ToS (Type of Service) bit value, if you have configured ToS-based priorities.
• Traffic matching security policies with traffic shaping enabled in the policy may use any queue. The queue is selected based on whether the packet rate is currently below the guaranteed bandwidth (queue 0), or above the guaranteed bandwidth. Packets at rates greater than the maximum bandwidth limit are dropped.
• If the global tos-based-priority is low (3) and the priority in a traffic-shaper is medium (2), when a packet flows through a policy that refers to the shaper, the packet will be assigned the priority defined by the shaper. In this case, medium (2).

Types of priority

Prioritization and traffic shaping behavior vary based on the configuration, service type, traffic volume, and whether the traffic is through traffic or originates at the FortiGate unit itself.

Packets can be assigned a priority in one of three ways:

• On entering ingress – for packets flowing through the firewall.
• Upon generation – for packets generated by the firewall (including packets generated due to AV proxying).
• On passing through a firewall policy – for packets passing through a firewall policy that has a traffic shaper defined.

Ingress priority and priority for generated packets is controlled via two different CLI settings:

config system global
    set traffic-priority-level {high|medium|low}
end

config system tos-based-priority
    edit 1
        set tos [0-15]
        set priority (high|medium|low)
    next
end

Type of Service (ToS) priority

Type of Service is an 8-bit field in the IP header that allows you to determine how the IP datagram should be delivered, using the following criteria: Delay, Throughput, Priority, Reliability, and Cost. The criteria help gateways pick the best way to route datagrams.

A router maintains a ToS value for each route in its routing table. The lowest priority ToS is 0, and the highest is 7 (when bits 3, 4, and 5 are all set to 1). There are four other bits that are seldom used or reserved that are not included here.

Together these bits are the tos variable of the tos-based-priority command. The router tries to match the ToS of the datagram to the ToS on one of the available routes to the destination. If there is no match, then the datagram is sent over a zero ToS route. Using increased quality may increase the cost of delivery, because better performance may consume limited network resources.

Each bit represents the priority as per RFC 1349:

• 1000 – minimize delay
• 0100 – maximize throughput
• 0010 – maximize reliability
• 0001 – minimize monetary cost

The tos value is set in the CLI using the commands:

config system tos-based-priority
    edit <sequence_number>
        set tos [0-15]
        set priority [high | medium | low]
    next
end

Where tos is the value of the type of service bit in the IP datagram header with a value between 0 and 15, and priority is the priority of this type of service.

ToS Priority	Setting
High	1
Medium	2
Low	3

These priority levels conform to the firewall traffic shaping priorities, as defined in RFC 1349.

Firewall policy priority

All traffic shapers are enabled within a security policy, including the Application Control shapers. As such, the shapers take effect after any DoS detection policies, and before any routing or packet scanning occurs.

The shaper you select for the security policy (shared shaper) will affect the traffic in the direction defined in the policy. For example, if the source port is lan and the destination is wan 1, the shaping affects the flow in this direction only — affecting the upload speed of the outbound traffic.

By selecting Shared Traffic Shaper Reverse Direction, you can define the traffic shaper for the policy in the opposite direction to affect the download speed of the inbound traffic. In this example, from wan 1 to lan.

config firewall policy
    edit <policy_number>
        ...
        set traffic-shaper <shaper_name>
        set per-ip-shaper <shaper_name>
        set traffic-shaper-reverse <shaper_name>
    next
end

In a firewall policy you can enable traffic shaping and set the firewall priority to high, medium, or low:

Firewall Policy Priority	Setting
High (default)	1
Medium	2
Low	3

Since all security policies are set to “high” priority by default, it is necessary to set some traffic at a lower priority to get results. When you enable traffic shaping, and change the priority to medium or low it will override the default setting.

To have proper QoS using the FortiGate, the firewall policy you create between your incoming interface and outgoing interface should include two interfaces. For example, a LAN to WAN 1 policy.

Important: Make sure that ALL the firewall policies that use these two interfaces for communication have traffic shaping enabled!

In versions of FortiOS 5.2 and earlier, you must enable traffic shaping at the policy level for each individual policy:

A screenshot of a FortiOS 5.2 Security Policy with all types of traffic shaping enabled, under Policy & Objects > Policy > IPv4.

This is no longer necessary in FortiOS 5.4, as the new Traffic Shaping Policies allow you to apply traffic shaping globally to any traffic matching your criteria. The criteria must specify a source, a destination, a service, and the outgoing interface:

A screenshot of a FortiOS 5.4 traffic shaping policy, under Policy & Objects > Traffic Shaping Policy.

How do these priorities affect each other?

The global or ingress ToS-based priority value is combined with the firewall policy priority value:

Global priority (0, 1, 2) + policy priority (1, 2, 3) = total priority (queue number).

Let’s take a look at some examples:

• If we assume a default ingress priority of low (2) and a firewall policy priority of low (3), then the resulting priority is 5.
• If the packet flowing through results in a rate that is less than the guaranteed bandwidth, then the priority is set to 0 regardless of the priority in the firewall policy.
• If the packet flowing through results in a rate that’s above the maximum bandwidth, then the packet is dropped.
• If the packet flowing through results in a rate that is between the guaranteed and the maximum bandwidth, then the packet priority is increased by the priority from the policy. Therefore, assuming a default ingress priority of high (0) and a firewall policy of high (1), then the resulting priority is 1.
• When a packet is sent to the egress device, it is attached to a queue based on the packet priority. For example, priority 0 is attached to queue 1, and so on. If the queue is full, then the packet is dropped.

Important: Shaper priority only affects per port egress queueing. Thus, if there are two streams of traffic — with one egressing over port 1 and one egressing over port 2 — then the priority has no effect whatsoever. Both streams will continue to run at full speed.

Traffic passing through the FortiGate

The method a FortiGate unit uses to determine the priority queue for traffic passing through the FortiGate unit depends on whether you have enabled Traffic Shaping. Packets may or may not use a priority queue directly or indirectly derived from the type of service (ToS) bit — sometimes used instead with differentiated services — in the packet’s IP header.

If Traffic Shaping is not enabled in the security policy, the FortiGate unit neither limits nor guarantees bandwidth. Traffic shaping for that session uses the priority queue determined by matching the ToS bit in its header with your configured values:

config system global
    set traffic-priority-level {high | medium | low}
end

or, if you have configured a priority specifically for that TOS bit value:

config system tos-based-priority
    edit <id_int>
        set tos [0-15]
        set priority {high | medium | low}
    next
end

Where tos is the value of the ToS bit in the packet’s IP header, and high has a value of 0 and low is 2. Priority values configured in the second location will override the global ToS-based priority. In other words, packet priority = ToS-based priority.

For example, you might specify that packets with a ToS bit value of 2 should use queue 0, the highest priority queue:

config system tos-based-priority
    edit 15
        set tos 2
        set priority high
    next
end

If Traffic Shaping is enabled in the security policy using shared traffic shapers, the FortiGate unit may instead or also subject packets to traffic policing or priority queue increases in an effort to meet bandwidth guarantees configured in the shaper:

config firewall shaper traffic-shaper
    edit <shaper_name>
        ...
        set priority {high | medium | low }
        set maximum bandwidth <rate>
        set guaranteed-bandwidth <rate>
    next
end

Where high has a priority value of 1 and low is 3, and <rate> is the bandwidth limit in kilobits per second.

Traffic queueing as as the packet rate increases.

• If the current packet rate is less than the guaranteed bandwidth, packets use priority queue 0. In other words, packet priority = 0.
• If the current packet rate is greater than the guaranteed bandwidth, but less than maximum bandwidth, the FortiGate unit assigns a priority queue by adding the numerical value of the security policy-based priority, where high has a priority value of 0 and low is 2. Because the two values are added, depending on the configured ToS-based priorities, packets in this category could use queues from queue 1 to queue 5. In other words, packet priority = ToS-based priority + security policy-based priority. For example, if you have enabled Traffic Shaping in the security policy, and the security policy’s Traffic Priority is Low (value 3), and the priority normally applied to packets with that ToS bit is medium (value 1), then packets have a total packet priority of 4, and use priority queue 4.
• If the current packet rate exceeds the maximum bandwidth, excess packets are dropped.

Traffic originating at the FortiGate

Security policies do not apply to administrative access traffic to the FortiGate through IPsec tunnel negotiations. Consequently, FortiGates do not apply traffic shaping to these types of traffic. These typesof traffic use the highest priority queue, queue 0. In other words, packet priority = 0.

Exceptions to this rule include traffic types with connections that are related to a session governed by a security policy. For example, if you have enabled FortiGuard AntiVirus scanning, traffic from the sender technically terminates at the FortiGate proxy that scans that traffic type; the FortiGate unit initiates a second connection that transmits scanned content to its destination. Because the second connection’s traffic is technically originating from the FortiGate proxy, and therefore the FortiGate unit itself, it uses the highest priority queue, queue 0. However, this connection is logically associated with through traffic, and is therefore subject to possible bandwidth enforcement and guarantees in its governing security policy. In this way, it behaves partly like other through traffic.

Egress queueing

Shaper priority only affects per port egress queueing, so if you have two streams of traffic, like one egressing over Port1 and one egressing over Port2, then priority has no effect whatsoever. Both streams will continue to run at full speed.

To make any difference to the order in which packets egress the interface, there must be packets of a lower priority queued on the egress interface. This usually happens when there is an imbalance between the packet rates on the interfaces.

For example, if the LAN is 1Gb, but the WAN is only 100MB. In this scenario the priority of the traffic egressing the WAN is very important, but the traffic egressing the LAN is rendered irrelevant (as it would take 10 WAN links to drive traffic at a high enough rate to cause queuing interference on the LAN interface).

This was tested by performing a debug on the kernel to determine when priority would take effect. In this case, by counting how many times the egress interface had more than one packet in the queue. Two simultaneous 500MB downloads via HTTP were performed, with one policy set to a high priority and one set to a low priority. Results showed that there was more than one packet in the egress queue only 23 times. With over 600,000 packets egressing over that interface, altering the priority of 23 does not make a practical difference to the relative speed of downloads.

Resources

In the FortiOS Handbook, you may be interested in checking out the following Traffic Shaping sections:

• Important Considerations for more information.
• Troubleshooting for diagnose commands.
• Traffic Shaper Monitor for how to view traffic shaping results.