Skip to content

Glaber Path Profiler — Programming Guide

Fork-local wall-time profiler for attributing time spent in hot-path stages (history syncer first). Use it when top/htop are useless because the process is blocked on IPC, SQL, or ClickHouse rather than burning CPU.

Piece Path
Public API include/glb_prof.h
Implementation src/libs/glb_prof/glb_prof.c
Library src/libs/glb_prof/libglbprof.a
Example instrumentation src/libs/zbxcachehistory/dbcache.c (sync_server_history)
Enable + periodic dump src/zabbix_server/dbsyncer/dbsyncer.c

Related but different: stock Zabbix zbx_prof_* (include/zbxprof.h) — mutex/rwlock/processing spans dumped to the log via -R prof_enable. Prefer glb_prof for named hot-path stages.


Goals and cost model

  • When disabled: glb_prof_enter / glb_prof_leave are a single TLS flag check + unlikely branch. No clock reads, no string work.
  • When enabled: one zbx_timespec() on enter and on leave; fixed-size TLS counters (one slot per section).
  • Scope: per fork/thread only. There is no shared aggregation or trapper export yet — results are printed with LOG_INF on flush.
  • What is measured: wall time, not CPU time. That is intentional for IO/wait bottlenecks.

Basic usage

C++ (preferred) — RAII scope

Like std::lock_guard: destructor calls leave on any exit path (return, throw, break).

#include "glb_prof.h"

int foo()
{
    glb_prof_scope prof(GLB_PROF_VC_GET_REMOTE);
    // or: GLB_PROF_SCOPE(GLB_PROF_VC_GET_REMOTE);

    if (fail_early)
        return FAIL;   /* leave runs automatically */

    return SUCCEED;
}

glb_prof_sql_scope sql(sql_text); does the same for classified SQL helpers.

C — manual enter/leave

#include "glb_prof.h"

glb_prof_enable();

glb_prof_enter(GLB_PROF_HIST_WRITE);
/* ... work to measure ... */
glb_prof_leave(GLB_PROF_HIST_WRITE);

glb_prof_flush("history syncer #3");

Pair enter/leave carefully around early returns, or wrap the body in a single exit path. In C++ prefer glb_prof_scope.

API

Call Role
glb_prof_enable() Turn profiling on for this fork; reset counters and start a new window.
glb_prof_disable() Turn off; clear state. Enter/leave become almost free again.
glb_prof_scope / GLB_PROF_SCOPE C++ RAII guard — enter on construct, leave on destroy (preferred).
glb_prof_enter / glb_prof_leave Manual start/stop (C, or when RAII is awkward).
glb_prof_sql_enter / glb_prof_sql_leave Classify SQL text and time it (used by DB layer).
glb_prof_flush(info) Print LOG_INF stats for the current window, then reset totals.

In C++, prefer glb_prof_scope so early return cannot skip leave. Manual enter/leave remain for C code (dbcache.c, etc.).


Adding a new section

Sections are defined once in include/glb_prof.h via GLB_PROF_SECTIONS. The enum (GLB_PROF_*) and the printable name table are generated from that list; names are bound by enum id ([GLB_PROF_FOO] = "foo"), not by array order.

  1. Append one line to the list:
#define GLB_PROF_SECTIONS               \
    X(IPC_RECV,     "ipc_recv")     \
    /* ... existing ... */              \
    X(IDLE,         "idle")         \
    X(CH_CURL_WAIT,     "ch_curl_wait")     /* new */
  1. Instrument the code:
glb_prof_enter(GLB_PROF_CH_CURL_WAIT);
/* curl_easy_perform(...) or equivalent */
glb_prof_leave(GLB_PROF_CH_CURL_WAIT);
  1. Rebuild. No other files need a parallel name array.

Naming tips

  • Enum token: short, upper-case role (HIST_WRITE, TRIGGERS_SQL).
  • Print name: stable, lower-case, snake_case (hist_write) — this is what operators grep in logs.
  • Prefer functional stages over raw function names when the same stage is called from several places.

Semantics and pitfalls

Nesting different sections

Different sections may overlap or nest. Each accumulates its own wall time. The flush line accounted can therefore exceed 100% of window if stages overlap. That is expected — use per-section % to see where time goes; do not assume the percentages form a partition unless you instrument non-overlapping stages.

Same section entered twice

A second enter for a section that is already open is ignored (outermost start is kept). One leave closes it. Prefer non-reentrant use of a given section id.

Unbalanced leave

leave with no matching open enter is a no-op.

Flush while a section is still open

Time elapsed so far in the open section is attributed to the current window; the section stays open and continues into the next window. Safe to flush from the main loop without unwinding nested work.

Idle vs busy

GLB_PROF_IDLE is used around intentional sleep when the syncer has nothing to do. High idle % means underload; high hist_write / triggers_sql with low idle means a real bottleneck.

If a process never enters GLB_PROF_IDLE (typical for event-loop workers), flush derives idle as max(0, window − busy), where busy is the sum of all other section totals. Overlapping sections can make busy exceed the window; derived idle is then 0. Summary lines busy / idle / accounted are always printed against wall-clock window length.


Reading the log output

History syncer flushes about every 30 seconds (PROF_INTERVAL) when profiling is enabled via -R prof_enable, with a label like history syncer #N. Profiling is off by default.

Example:

=== glb_prof history syncer #3 (window 5.012s) ===
  ipc_recv           calls=812      total=  0.390s  avg=   0.480ms  max=   4.100ms    7.8%
  hist_write         calls=41       total=  2.870s  avg=  70.000ms  max= 310.000ms   57.2%
  idle               calls=1200     total=  0.800s  avg=   0.667ms  max=   1.200ms   15.9%
  accounted                             total=  4.500s                        89.7% of window
Field Meaning
window Wall time since last flush (or since enable).
calls Completed enter/leave pairs in the window.
total Sum of wall time in the section.
avg / max Per-call average and max (ms).
% total / window — share of wall time, not CPU%.
accounted Sum of section totals including idle (may be > or < window when stages overlap).
busy Sum of all non-idle section totals.
idle Measured GLB_PROF_IDLE, or derived window − busy when idle was never entered.

How to use it

  1. Find the section with the largest % (or high max with modest % for latency spikes).
  2. Drill into that code path — optionally add finer child sections (e.g. split hist_write into buffer flush vs HTTP wait).
  3. Compare several syncer forks (#1, #2, …); load is sharded and bottlenecks may be uneven.

Enabling in a new worker

Prefer runtime enable (-R prof_enable=…) over hard-coding glb_prof_enable() at process start. If you need a local flush loop:

ZBX_THREAD_ENTRY(my_worker_thread, args)
{
    /* ... */
    /* do not enable by default — wait for RTC / explicit enable */

    for (;;)
    {
        /* hot path with glb_prof_enter/leave around stages */

        if (interval_elapsed)
        {
            char info[64];

            zbx_snprintf(info, sizeof(info), "%s #%d", process_name, process_num);
            glb_prof_flush(info);
        }
    }
}

Link $(top_builddir)/src/libs/glb_prof/libglbprof.a into the binary if it is not already linked (server and proxy already include it).

Runtime control already wires prof_enable / prof_disable to glb_prof_* in daemon.c. Keep workers cheap when disabled (enter/leave are a single branch).


Current history syncer sections

Instrumented in sync_server_history():

Section Typical meaning
ipc_recv Pull metrics from preprocessing → syncer IPC
config_cache Load item config for the batch
prepare_history Prepare history / inventory / proxy nodata side effects
vc_get_local / vc_get_remote Value-cache fetch (local map vs IPC)
vc_add_local / vc_add_remote Value-cache add (local vs remote)
state_keeper Item error / state IPC
trigger_timers Fetch or reschedule trigger timers
triggers_sql Trigger recalculation + SQL transaction (outer stage)
hist_write History module write (e.g. ClickHouse)
trends Trend update
inventory_sql Inventory SQL transaction (outer stage)
sql_select / sql_insert / sql_update / sql_delete Classified DB statements (from zbx_db_vselect / zbx_db_vexecute)
sql_begin / sql_commit / sql_rollback Transaction control
sql_other Unclassified SQL
idle Sleep when the batch was empty

SQL sections nest under outer stages such as triggers_sql; that overlap is intentional.


Build notes

  • Library is built as src/libs/glb_prof (Makefile.am + configure.ac entry).
  • After adding the directory to the build system, regenerate with the usual autoconf / ./config.status / make flow if Makefiles are missing.
  • Header is C and C++ friendly (extern TLS flag + macros); implementation is compiled with the project’s C++ toolchain like other Glaber C units.