Glaber Path Profiler — Programming Guide
Fork-local wall-time profiler for attributing time spent in hot-path stages (history syncer first). Use it when top/htop are useless because the process is blocked on IPC, SQL, or ClickHouse rather than burning CPU.
| Piece | Path |
|---|---|
| Public API | include/glb_prof.h |
| Implementation | src/libs/glb_prof/glb_prof.c |
| Library | src/libs/glb_prof/libglbprof.a |
| Example instrumentation | src/libs/zbxcachehistory/dbcache.c (sync_server_history) |
| Enable + periodic dump | src/zabbix_server/dbsyncer/dbsyncer.c |
Related but different: stock Zabbix zbx_prof_* (include/zbxprof.h) — mutex/rwlock/processing spans dumped to the log via -R prof_enable. Prefer glb_prof for named hot-path stages.
Goals and cost model
- When disabled:
glb_prof_enter/glb_prof_leaveare a single TLS flag check + unlikely branch. No clock reads, no string work. - When enabled: one
zbx_timespec()on enter and on leave; fixed-size TLS counters (one slot per section). - Scope: per fork/thread only. There is no shared aggregation or trapper export yet — results are printed with
LOG_INFon flush. - What is measured: wall time, not CPU time. That is intentional for IO/wait bottlenecks.
Basic usage
C++ (preferred) — RAII scope
Like std::lock_guard: destructor calls leave on any exit path (return, throw, break).
#include "glb_prof.h"
int foo()
{
glb_prof_scope prof(GLB_PROF_VC_GET_REMOTE);
// or: GLB_PROF_SCOPE(GLB_PROF_VC_GET_REMOTE);
if (fail_early)
return FAIL; /* leave runs automatically */
return SUCCEED;
}
glb_prof_sql_scope sql(sql_text); does the same for classified SQL helpers.
C — manual enter/leave
#include "glb_prof.h"
glb_prof_enable();
glb_prof_enter(GLB_PROF_HIST_WRITE);
/* ... work to measure ... */
glb_prof_leave(GLB_PROF_HIST_WRITE);
glb_prof_flush("history syncer #3");
Pair enter/leave carefully around early returns, or wrap the body in a single exit path. In C++ prefer glb_prof_scope.
API
| Call | Role |
|---|---|
glb_prof_enable() |
Turn profiling on for this fork; reset counters and start a new window. |
glb_prof_disable() |
Turn off; clear state. Enter/leave become almost free again. |
glb_prof_scope / GLB_PROF_SCOPE |
C++ RAII guard — enter on construct, leave on destroy (preferred). |
glb_prof_enter / glb_prof_leave |
Manual start/stop (C, or when RAII is awkward). |
glb_prof_sql_enter / glb_prof_sql_leave |
Classify SQL text and time it (used by DB layer). |
glb_prof_flush(info) |
Print LOG_INF stats for the current window, then reset totals. |
In C++, prefer glb_prof_scope so early return cannot skip leave. Manual enter/leave remain for C code (dbcache.c, etc.).
Adding a new section
Sections are defined once in include/glb_prof.h via GLB_PROF_SECTIONS. The enum (GLB_PROF_*) and the printable name table are generated from that list; names are bound by enum id ([GLB_PROF_FOO] = "foo"), not by array order.
- Append one line to the list:
#define GLB_PROF_SECTIONS \
X(IPC_RECV, "ipc_recv") \
/* ... existing ... */ \
X(IDLE, "idle") \
X(CH_CURL_WAIT, "ch_curl_wait") /* new */
- Instrument the code:
glb_prof_enter(GLB_PROF_CH_CURL_WAIT);
/* curl_easy_perform(...) or equivalent */
glb_prof_leave(GLB_PROF_CH_CURL_WAIT);
- Rebuild. No other files need a parallel name array.
Naming tips
- Enum token: short, upper-case role (
HIST_WRITE,TRIGGERS_SQL). - Print name: stable, lower-case, snake_case (
hist_write) — this is what operators grep in logs. - Prefer functional stages over raw function names when the same stage is called from several places.
Semantics and pitfalls
Nesting different sections
Different sections may overlap or nest. Each accumulates its own wall time. The flush line accounted can therefore exceed 100% of window if stages overlap. That is expected — use per-section % to see where time goes; do not assume the percentages form a partition unless you instrument non-overlapping stages.
Same section entered twice
A second enter for a section that is already open is ignored (outermost start is kept). One leave closes it. Prefer non-reentrant use of a given section id.
Unbalanced leave
leave with no matching open enter is a no-op.
Flush while a section is still open
Time elapsed so far in the open section is attributed to the current window; the section stays open and continues into the next window. Safe to flush from the main loop without unwinding nested work.
Idle vs busy
GLB_PROF_IDLE is used around intentional sleep when the syncer has nothing to do. High idle % means underload; high hist_write / triggers_sql with low idle means a real bottleneck.
If a process never enters GLB_PROF_IDLE (typical for event-loop workers), flush derives idle as max(0, window − busy), where busy is the sum of all other section totals. Overlapping sections can make busy exceed the window; derived idle is then 0. Summary lines busy / idle / accounted are always printed against wall-clock window length.
Reading the log output
History syncer flushes about every 30 seconds (PROF_INTERVAL) when profiling is enabled via -R prof_enable, with a label like history syncer #N. Profiling is off by default.
Example:
=== glb_prof history syncer #3 (window 5.012s) ===
ipc_recv calls=812 total= 0.390s avg= 0.480ms max= 4.100ms 7.8%
hist_write calls=41 total= 2.870s avg= 70.000ms max= 310.000ms 57.2%
idle calls=1200 total= 0.800s avg= 0.667ms max= 1.200ms 15.9%
accounted total= 4.500s 89.7% of window
| Field | Meaning |
|---|---|
window |
Wall time since last flush (or since enable). |
calls |
Completed enter/leave pairs in the window. |
total |
Sum of wall time in the section. |
avg / max |
Per-call average and max (ms). |
% |
total / window — share of wall time, not CPU%. |
accounted |
Sum of section totals including idle (may be > or < window when stages overlap). |
busy |
Sum of all non-idle section totals. |
idle |
Measured GLB_PROF_IDLE, or derived window − busy when idle was never entered. |
How to use it
- Find the section with the largest
%(or highmaxwith modest%for latency spikes). - Drill into that code path — optionally add finer child sections (e.g. split
hist_writeinto buffer flush vs HTTP wait). - Compare several syncer forks (
#1,#2, …); load is sharded and bottlenecks may be uneven.
Enabling in a new worker
Prefer runtime enable (-R prof_enable=…) over hard-coding glb_prof_enable() at process start. If you need a local flush loop:
ZBX_THREAD_ENTRY(my_worker_thread, args)
{
/* ... */
/* do not enable by default — wait for RTC / explicit enable */
for (;;)
{
/* hot path with glb_prof_enter/leave around stages */
if (interval_elapsed)
{
char info[64];
zbx_snprintf(info, sizeof(info), "%s #%d", process_name, process_num);
glb_prof_flush(info);
}
}
}
Link $(top_builddir)/src/libs/glb_prof/libglbprof.a into the binary if it is not already linked (server and proxy already include it).
Runtime control already wires prof_enable / prof_disable to glb_prof_* in daemon.c. Keep workers cheap when disabled (enter/leave are a single branch).
Current history syncer sections
Instrumented in sync_server_history():
| Section | Typical meaning |
|---|---|
ipc_recv |
Pull metrics from preprocessing → syncer IPC |
config_cache |
Load item config for the batch |
prepare_history |
Prepare history / inventory / proxy nodata side effects |
vc_get_local / vc_get_remote |
Value-cache fetch (local map vs IPC) |
vc_add_local / vc_add_remote |
Value-cache add (local vs remote) |
state_keeper |
Item error / state IPC |
trigger_timers |
Fetch or reschedule trigger timers |
triggers_sql |
Trigger recalculation + SQL transaction (outer stage) |
hist_write |
History module write (e.g. ClickHouse) |
trends |
Trend update |
inventory_sql |
Inventory SQL transaction (outer stage) |
sql_select / sql_insert / sql_update / sql_delete |
Classified DB statements (from zbx_db_vselect / zbx_db_vexecute) |
sql_begin / sql_commit / sql_rollback |
Transaction control |
sql_other |
Unclassified SQL |
idle |
Sleep when the batch was empty |
SQL sections nest under outer stages such as triggers_sql; that overlap is intentional.
Build notes
- Library is built as
src/libs/glb_prof(Makefile.am+configure.acentry). - After adding the directory to the build system, regenerate with the usual
autoconf/./config.status/makeflow if Makefiles are missing. - Header is C and C++ friendly (
externTLS flag + macros); implementation is compiled with the project’s C++ toolchain like other Glaber C units.