Server process profiling (glb_prof)

For bottleneck analysis (history syncer, lock wait, SQL, value cache, and so on) Glaber provides the built-in glb_prof profiler. It is off by default; enable and disable it without restarting the server via runtime control (-R).

While profiling is on, the process periodically writes log lines such as === glb_prof history syncer #1 (window …) === with cumulative time per stage. For a history syncer the dump is typically about every 30 seconds.

Run the commands as a user that can use the runtime-control socket (usually root or the service user). The binary is glaber_server (on older installs it may be zabbix_server). Pass a config if needed: -c /etc/glaber/glaber_server.conf.

See also general troubleshooting.

Enable and disable

Enable for history syncer #1

glaber_server -R 'prof_enable=history syncer,1'

Disable

glaber_server -R 'prof_disable=history syncer,1'

Other examples

All history syncers:

glaber_server -R 'prof_enable=history syncer'

By process PID:

ps aux | grep '[h]istory syncer'
glaber_server -R 'prof_enable=<pid>'

All server processes (noisy — use only for a short diagnostic window):

glaber_server -R prof_enable
glaber_server -R prof_disable

Collect results from the log

Typical log path: /var/log/glaber/glaber_server.log (older installs: /var/log/zabbix/zabbix_server.log).

grep 'glb_prof' /var/log/glaber/glaber_server.log

or live:

tail -f /var/log/glaber/glaber_server.log | grep --line-buffered glb_prof

Attach several dump windows (about 1–2 minutes under load) for the process under investigation. After collecting data, disable profiling (prof_disable) so it does not flood the log.

Each active section in a dump reports total time, share of the window, call count, average, and maximum.

Profiling sections

Names in the log match the Log name column. Not every section appears in every process — only those whose code paths ran while profiling was on.

History sync / processing stages

Log name	Description
ipc_recv	Receiving a batch from IPC ( → processing queue)
config_cache	Config-cache work while preparing/processing values
prepare_history	Preparing history before write and related logic
state_keeper	State updates via state keeper
trigger_timers	Trigger timer processing
triggers_sql	SQL related to trigger/event handling and persistence
hist_write	Writing history to the external store (e.g. ClickHouse)
trends	Trend calculation and write
inventory_sql	Inventory update SQL

Proxy poller / preprocessing / IPC

Log name	Description
proxy_process	One proxy-poller cycle (network, config, data/tasks). SQL and DC show up separately via sql_* / locks
preproc_process	Metric batch work in glb_preproc_worker
ipc_metrics_wait	Wait while the metrics IPC buffer is exhausted (starvation sending to ->preproc or ->processing). While waiting, a dump is written about every 30s

Enable examples:

glaber_server -R 'prof_enable=proxy poller,1'
glaber_server -R 'prof_enable=glb_preproc_worker,1'

Value cache

Log name	Description
vc_get_local	Read from the local value-cache shard
vc_get_remote	Read via the remote/IPC value-cache keeper path
vc_add_local	Write to the local value-cache shard
vc_add_remote	Write via the remote/IPC value-cache keeper path

SQL (auto-classified queries)

Wall time of SQL execution in the DB layer. Type is taken from the start of the query text.

Log name	Description
sql_select	SELECT / WITH …
sql_insert	INSERT
sql_update	UPDATE
sql_delete	DELETE
sql_begin	BEGIN
sql_commit	COMMIT
sql_rollback	ROLLBACK
sql_other	Other SQL statements

These counters can overlap in wall time with the stage sections above (for example triggers_sql and sql_update): a stage is a logical code block; sql_* is time spent in the DB driver.

Locks

Log name	Description
lock_wait	Waiting until any profiled lock is acquired (mutex or rwlock)
locked_rd	Holding a rwlock in read mode (rdlock)
locked_wr	Holding a rwlock in write mode (wrlock)
locked_mutex	Holding a plain mutex (always exclusive)

lock_wait is time before acquire; locked_* is time after acquire until unlock. Rd/wr vs mutex differ by primitive type, not by “strength”.

Other

Log name	Description
idle	Process idle time. If GLB_PROF_IDLE was not measured, derived as window − busy (remaining wall time)