diagnose debug jemalloc proxyd

The diagnose debug jemalloc proxyd command provides debugging tools for analyzing memory allocation and detecting potential leaks in proxyd. It offers options for generating memory dump files, inspecting object pools, and identifying long-lived objects that may indicate memory retention issues.

Specifically, the diagnose debug jemalloc proxyd pdump-leaks command dumps objects that have existed for a specified duration, helping to identify potential memory leaks by listing persistent objects in proxyd's object pools. Each object pool allocates fixed-size memory chunks for tasks such as packet handling, session tracking, and connection management. By monitoring object lifespan, FortiWeb detects abnormal memory retention that may indicate a leak.

To maintain efficiency and prevent excessive output, the command limits the number of dumped entries to 100 per object pool per working thread.

Syntax

diagnose debug jemalloc proxyd dump
diagnose debug jemalloc proxyd pdump
diagnose debug jemalloc proxyd pdump-leaks <minutes>

dump	Generates a memory dump file for proxyd using the jemalloc allocator.
pdump	Dumps object pool details, including active allocations.
pdump-leaks <minutes>	Identifies and logs object pool entries that have persisted beyond the expected lifespan, aiding in memory leak detection. If <minutes> is not provided, the default is 30 minutes.

Examples

/var/log/gui_upload# cat proxyd-objpool-3488-1732325050.txt
                    objpool name:            total  incsize  objsize nodesize lock     nalloced       nfreed
           worker_run-1-packet_t:          4136000      500     4096     4136   no         1000          963
Total age > 2 min: 37
          worker_run-1-session_t:           188000      500      336      376   no          500          499
session age 3 min, policy server_policy1
Total age > 2 min: 1
        worker_run-1-pt_stream_t:           280000      500      520      560   no          500          498
stream age 3 min, dir 1, policy server_policy1
stream age 3 min, dir 0, policy server_policy1
Total age > 2 min: 2
           worker_run-0-packet_t:          2068000      500     4096     4136   no          500          500
Total age > 2 min: 0
          worker_run-0-session_t:           188000      500      336      376   no          500          500
Total age > 2 min: 0
       worker_run-1-connection_t:           536000     1000      496      536   no         1000          998
connection age 3 min, (192.168.2.1:14162->192.168.2.2:80)
connection age 3 min, (10.159.28.221:56012->10.159.28.226:443)
Total age > 2 min: 2
        worker_run-0-pt_stream_t:           280000      500      520      560   no          500          500
Total age > 2 min: 0
       worker_run-1-pt_service_t:           948000      500     1856     1896   no          500          498
service age 3, type 0, policy server_policy1
service age 3, type 0, policy server_policy1
Total age > 2 min: 2
       worker_run-1-pt_substream:           320000      500      600      640   no          500          498
substream 321e0528 age 3 min, status 0, peer 321e07a8, policy server_policy1
substream 321e07a8 age 3 min, status 0, peer 321e0528, policy server_policy1
Total age > 2 min: 2
       worker_run-0-connection_t:           536000     1000      496      536   no         1000         1000
Total age > 2 min: 0
       worker_run-0-pt_service_t:           948000      500     1856     1896   no          500          500
Total age > 2 min: 0
       worker_run-0-pt_substream:           320000      500      600      640   no          500          500
Total age > 2 min: 0