Problem events export — design & implementation plan
Related: EVENTS_EXPORT.md (legacy Zabbix file-export behaviour).
Owner process: problems_keeper. Problem create / recover / close / acknowledge / suppress / messages are its responsibility. Export is implemented as a HistoryModule write channel.
Sinks:
- §5 File export (
events_file) - §6 Syslog export (
events_syslog) - §7 ClickHouse export (
write_eventsonclickhouse)
1. Verdict: data available today
glb_problem_t alone is not enough for useful export. Hostnames/groups are the obvious gap; recovery event id and an emission hook are also missing.
1.1 What problems_keeper already has
| Field | On glb_problem_t |
Notes |
|---|---|---|
Problem eventid |
Yes | Snowflake via keeper id_generator |
Trigger / objectid |
Yes | |
clock (problem time) |
Yes | From trigger notify |
name |
Yes | Macro-expanded after zbx_add_event |
severity |
Yes | |
tags |
Yes | {tag,value} |
hostids |
Yes | IDs only |
Recovered flag / r_clock |
Yes | Private r_clock; no recovery event id |
| Ack / suppress / symptom / cause | Yes | Needed for lifecycle export types |
1.2 Gaps
| Needed | Available? | Gap |
|---|---|---|
| Host technical / visible names | No (hostids only) | Resolve via DC / config cache |
| Host groups | No | Resolve via DC / DB |
ns |
Always 0 today |
Emit 0 in v1 or plumb later |
Recovery eventid |
Not on glb_problem_t |
Capture from close_trigger_event() |
| Message text on updates | Parsed in trapper JSON, not kept on glb_problem_update_t |
Carry message into export payload |
| Outbound export hook | None | Emit on every relevant state transition |
Bottom line: keeper has problem identity and most semantics. It needs enrichment (hosts/groups), recovery id at close, message text on MESSAGE updates, and a single emission API for all lifecycle types.
2. Goals
- Export all problem-related lifecycle events from problems_keeper (not only PROBLEM/OK).
- Wire format: few structured index fields + one unstructured JSON payload (
data) so new event kinds do not require schema churn in every sink. - Modular sinks via HistoryModule only (
Name;{json}), including file, syslog, ClickHouse. - Treat problem events as a sibling history write channel (same pattern as trends).
- Convert the history subsystem interface to C++ as part of making events a first-class channel.
Non-goals for v1:
- Re-enable legacy
ExportDir/ExportType=eventsas the primary API (optional shim only). - Full UI for browsing exported events.
- Perfect classic Zabbix NDJSON parity as the internal protocol (sinks may still project a classic-compatible view if needed).
3. Export protocol draft
3.1 Design principles
- Envelope is stable. Only a small set of structured fields is required for routing, indexing, and time-ordering.
- Payload is extensible. Type-specific content lives in an unstructured JSON object (
data). Adding a new lifecycle event means adding atypevalue and documentingdatakeys — not changing the envelope or every module’s C struct. - Problem content is not flattened into the envelope. Snapshot / diff of the problem goes inside
data(e.g.data.problem), so sinks that only care abouttype+ time do not depend on problem schema. - One NDJSON line / one syslog line / one ClickHouse row = one envelope.
3.2 Envelope (structured fields)
Every exported record:
| Field | Type | Required | Meaning |
|---|---|---|---|
eventid |
uint64 | yes | Problem event id this record concerns (stable correlation key across lifecycle) |
clock |
int | yes | Unix seconds of this lifecycle event |
ns |
int | yes | Nanoseconds; 0 until plumbed |
type |
string | yes | Lifecycle event type (see below) |
data |
object | yes | Unstructured JSON payload for this type (may be {}) |
Wire example (NDJSON line):
{"eventid":12345,"clock":1710000000,"ns":0,"type":"problem.create","data":{...}}
Notes:
- For recovery,
eventidremains the problem id. Recovery event id (if any) belongs indata(e.g.data.r_eventid), not as a second top-level id. This keeps filtering “everything about problem X” trivial. - Optional future envelope fields (only if clearly universal):
source(problems_keeper),userid. Prefer putting actor info indatauntil proven necessary.
3.3 Event types
type is a stable string. Use dotted names so families sort/filter cleanly.
type |
When emitted | Typical data contents |
|---|---|---|
problem.create |
New problem created | Full problem snapshot (+ enriched hosts/groups) |
problem.recover |
Problem recovered (trigger OK / correlation / cleanup close that creates recovery) | r_eventid, recovery clock, optional problem snapshot or minimal refs |
problem.close |
Manual close (ack action CLOSE) | userid, r_eventid, reason/source |
problem.acknowledge |
Acknowledged | userid, optional message |
problem.unacknowledge |
Unacknowledged | userid |
problem.message |
Message added | userid, message |
problem.severity |
Severity changed | userid, old_severity, new_severity |
problem.suppress |
Suppressed (by ack) | userid, suppress_from, suppress_until |
problem.unsuppress |
Unsuppressed | userid |
problem.rank_cause |
Ranked to cause | userid, related ids |
problem.rank_symptom |
Ranked to symptom | userid, cause_eventid |
problem.suppress_maintenance |
Suppressed because host(s) in maintenance | hostids / flags |
problem.suppress_trigger |
Suppressed because trigger non-functional / UNKNOWN | triggerid / flags |
problem.unsuppress_maintenance |
Maintenance suppress cleared | |
problem.unsuppress_trigger |
Trigger-state suppress cleared | |
action.execute |
Reserved: action/escalation step executed against the problem | actionid, esc_step, media/operation summary |
action.notify |
Reserved: notification sent | channel, subject/body refs, status |
Existing keeper actions map from glb_problem_update_t:
PROBLEM_UPDATE_* |
Export type |
|---|---|
| (create path) | problem.create |
| (recover path) | problem.recover |
CLOSE |
problem.close (and/or problem.recover — pick one primary; recommend emit problem.close for manual, problem.recover for automatic) |
ACKNOWLEDGE |
problem.acknowledge |
UNACKNOWLEDGE |
problem.unacknowledge |
MESSAGE |
problem.message |
SEVERITY |
problem.severity |
SUPPRESS |
problem.suppress |
UNSUPPRESS |
problem.unsuppress |
RANK_TO_CAUSE |
problem.rank_cause |
RANK_TO_SYMPTOM |
problem.rank_symptom |
New functionality later = new type string + documented data keys. No envelope change.
3.4 data conventions
- Always a JSON object.
- Prefer nested objects over flat mega-keys:
data.problem— snapshot of problem fields when useful (name,severity,objectid,tags,hosts,groups,acknowledged,suppressed, …).data.change— diff-style fields for updates (old_severity,new_severity,message, …).data.actor—{ "userid": … }when a user initiated the change.- Enrichment (hostnames, groups) is done once in problems_keeper and placed into
data.problem.hosts/data.problem.groupsfor types that need a snapshot. - Sinks must tolerate unknown keys and unknown
typevalues (forward-compatible). - Classic Zabbix PROBLEM/OK projection (if needed for an old consumer) is a sink-side adapter from
problem.create/problem.recover|close, not the internal protocol.
Example — create:
{
"eventid": 12345,
"clock": 1710000000,
"ns": 0,
"type": "problem.create",
"data": {
"problem": {
"name": "High CPU utilization",
"severity": 3,
"objectid": 1001,
"tags": [{"tag": "scope", "value": "performance"}],
"hosts": [{"hostid": 10, "host": "web01", "name": "Web server 01"}],
"groups": ["Linux servers"]
}
}
}
Example — severity change:
{
"eventid": 12345,
"clock": 1710000100,
"ns": 0,
"type": "problem.severity",
"data": {
"actor": {"userid": 7},
"change": {"old_severity": 3, "new_severity": 4}
}
}
Example — message:
{
"eventid": 12345,
"clock": 1710000200,
"ns": 0,
"type": "problem.message",
"data": {
"actor": {"userid": 7},
"change": {"message": "Investigating with netops"}
}
}
3.5 In-process record (C++)
struct glb_history_event_t {
uint64_t eventid = 0; // problem id
int clock = 0;
int ns = 0;
std::string type; // e.g. "problem.create"
std::string data_json; // serialized JSON object for `data`
};
Keeper builds type + a JSON object (via zbx_json or equivalent), then calls the history API. Modules receive the envelope; they do not parse problem internals unless they choose to.
4. Architecture (HistoryModule only)
4.1 Decision
Configuration and loading are only via:
HistoryModule=<name>;{json}
There is no separate EventsModule (or other parallel loader). Event sinks are history modules that register the events write channel. ClickHouse opts in with "write_events":true in its existing JSON.
Do not add events to write_types=dbl,uint,… — those are item value types. Events are a separate write channel, same idea as trends.
4.2 Write channels
GLB_MODULE_API_HISTORY_WRITE // item values
GLB_MODULE_API_HISTORY_WRITE_TRENDS // trends
GLB_MODULE_API_HISTORY_WRITE_EVENTS // NEW: problem lifecycle events
GLB_MODULE_API_HISTORY_READ…
GLB_MODULE_API_DESTROY
Public API:
int glb_history_add_events(const std::vector<glb_history_event_t> &events);
// or span/pointer+count equivalent
problems_keeper
│ any lifecycle change
▼
build glb_history_event_t (envelope + data_json, enrich once)
│
▼
glb_history_add_events() // HistoryModule fan-out
├─ events_file → NDJSON files
├─ events_syslog → syslog
├─ clickhouse → events table (if write_events)
└─ worker → optional later (put_events)
4.3 Emission points
| Keeper path | type |
|---|---|
create_add_new_problem |
problem.create |
| auto recover | problem.recover |
PROBLEM_UPDATE_CLOSE |
problem.close |
other PROBLEM_UPDATE_* |
matching type from §3.3 |
| maintenance / trigger suppress toggles | problem.suppress_* / problem.unsuppress_* |
| future action/escalation hooks | action.* |
Export is best-effort: sink failures must not roll back problem processing.
4.4 Enrichment
Resolve hostnames/groups in problems_keeper (or a shared helper) when building data.problem for types that include a snapshot. Cache by hostid; tolerate deleted hosts with empty names.
Event-only modules (events_file, events_syslog) register only the events write channel (+ destroy). ClickHouse registers it when "write_events":true.
Sink-specific configuration and behaviour: §5 File export, §6 Syslog export, §7 ClickHouse export.
Optional: if legacy ExportDir is set with events enabled, synthesize an internal events_file module. New installs use HistoryModule=events_file;….
5. File export
Module name: events_file.
Purpose: append problem lifecycle envelopes to local NDJSON files for offline shipping, SIEM pickup, or debugging — replacement for the old ExportDir problems writer.
5.1 Configuration
HistoryModule=events_file;{
"dir":"/var/lib/zabbix/export",
"file_size":"1G",
"prefix":"problems"
}
| JSON key | Meaning | Default |
|---|---|---|
dir |
Output directory (must exist, writable) | required |
file_size |
Rotate when current file would exceed this size | 1G |
prefix |
File name prefix | problems |
5.2 Behaviour
- One NDJSON line per envelope (
eventid,clock,ns,type,data). - File name pattern:
{dir}/{prefix}-problems-keeper-{n}.ndjson(single writer in the problems_keeper process; avoid per-history-syncer files). - Rotation: rename to
.oldwhen size limit is hit, then open a new file (same idea as legacyzbxexport). - Best-effort: I/O errors drop the record and log (rate-limited); do not fail problem processing.
- Optional legacy shim: map
ExportDir+ExportType=eventsto this module.
5.3 Implementation notes
- New backend:
src/libs/glb_history/history_events_file.cpp. - Reuse rotation/open ideas from
src/libs/zbxexport/export.cwhere useful; do not resurrect dbsyncer-side problem file openers as the primary path.
6. Syslog export
Module name: events_syslog.
Purpose: stream the same envelopes to the local syslog / journal for real-time ops tooling without a local file consumer.
6.1 Configuration
HistoryModule=events_syslog;{
"ident":"glaber-events",
"facility":"local0"
}
| JSON key | Meaning | Default |
|---|---|---|
ident |
openlog identity |
glaber-events |
facility |
Syslog facility (local0…local7, user, daemon, …) |
local0 |
6.2 Behaviour
openlog(ident, LOG_PID|LOG_NDELAY, facility)at module init.- One syslog message = one envelope JSON object (same protocol as file).
- Priority mapping (initial proposal): derive from
typeand/ordata.problem.severitywhen present; recoveries and messages default toLOG_INFO; high/disaster create →LOG_ERR. Exact mapping is still an open question (§12). - No local buffering beyond syslogd; failures logged rate-limited.
- Best-effort only.
6.3 Implementation notes
- New backend:
src/libs/glb_history/history_events_syslog.cpp. - Event-only: implements
add_events(+ destroy), ignores history/trends.
7. ClickHouse export
Module: existing clickhouse HistoryModule, extended with an events write path.
Purpose: long-term archival of problem lifecycle events alongside item history/trends, using the same module connection and flush machinery.
7.1 Configuration
HistoryModule=clickhouse;{
"url":"http://127.0.0.1:8123",
"username":"default",
"password":"password",
"dbname":"glaber",
"write_types":"dbl,str,log,uint,text",
"write_events":true
}
| JSON key | Meaning | Default |
|---|---|---|
write_events |
Register events write channel and insert into events table | false (no behaviour change for existing installs) |
| other keys | Unchanged ClickHouse history module options | see existing module |
7.2 Example table schema (illustrative only)
Note: The schema below is purely an example for discussion and early prototyping. It is not optimised or prepared for searching problems by host, group, tag, or name. Indexes, extracted columns,
ORDER BY, and possibly separate dictionaries/materialized views for those dimensions are subject to change later. Do not treat this DDL as a stable contract.
CREATE TABLE glaber.events
(
day Date,
clock DateTime,
ns UInt32,
eventid UInt64,
type LowCardinality(String),
data String -- JSON object (full unstructured payload)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(day)
ORDER BY (eventid, clock, type)
TTL day + INTERVAL 24 MONTH;
Properties of this example:
- Envelope fields are columns;
datastays a JSON string so newtypevalues need no ALTER. - Suitable for append-only archival and “all events for
eventid” style reads. - Not suited yet for efficient filters like “problems on host X”, “tag=Y”, “group=Z”, or “name contains …” — those would require a later schema revision (e.g. materialized columns, arrays, secondary indexes, or a search-oriented table).
7.3 Behaviour
- Batch inserts on the same flush timer as history when
write_eventsis enabled. - Writes are best-effort relative to problems_keeper (same as other sinks).
- Reads of events from ClickHouse are optional for v1 (write/archival first).
7.4 Implementation notes
- Extend
history_clickhousebackend withadd_events. - Add example DDL under
database/clickhouse/schema.sqlwith the same “example / subject to change” comment. - Later: materialized views or a revised table for host/group/tag/name search — tracked as follow-up, not MVP.
8. History interface today — analysis
8.1 Current shape
| Piece | Location | Language / style |
|---|---|---|
| Public API | include/glb_history.h |
Mixed: C functions + C++ types (ZBX_DC_HISTORY class, std::vector in get API) |
| Internal callbacks | src/libs/glb_history/history.h |
C function pointers + void *data |
| Dispatcher | src/libs/glb_history/history.c |
C file using C++ features; fan-out via API_CALLBACKS[] |
| ClickHouse / worker | history_clickhouse.c, history_worker.c |
C-with-classes / C++ in .c units |
| Registration | glb_register_callback() in modules API |
Untyped void (*)(void) casts |
Problems:
- Not a real C++ interface — modules are C callbacks with manual casts; no ownership, no virtual API, hard to extend safely.
- Hybrid already hurts — headers pull C++ (
vector, classes) while files are named.cand still use C callback tables. ZBX_DC_HISTORYis a class in a “C” header with incomplete serialize notes;trend_tremains a C struct with rawchar *.- Adding events on the old callback table repeats the same untyped pattern.
- Init discovery is
strstr(name, "clickhouse")/"worker"— fragile; C++ registry maps cleanly to module names.
8.2 Target C++ interface (draft)
// include/glb_history.hpp (name TBD)
struct GlbHistoryEvent {
uint64_t eventid = 0;
int clock = 0;
int ns = 0;
std::string type;
std::string data_json;
};
class GlbHistoryBackend {
public:
virtual ~GlbHistoryBackend() = default;
virtual int add_history(ZBX_DC_HISTORY *history, int history_num) { return SUCCEED; }
virtual int add_trend(trend_t *trend) { return SUCCEED; }
virtual int add_events(const std::vector<GlbHistoryEvent> &events) { return SUCCEED; }
virtual int get_history(/*…*/) { return FAIL; }
virtual int get_history_aggregates_json(/*…*/) { return FAIL; }
virtual int get_trends_json(/*…*/) { return FAIL; }
virtual int get_trends_aggregates_json(/*…*/) { return FAIL; }
};
// Facade used by server code
int glb_history_init(char **history_modules, char **error);
int glb_history_add_history(ZBX_DC_HISTORY *history, int history_num);
int glb_history_add_trend(trend_t *trend);
int glb_history_add_events(const std::vector<GlbHistoryEvent> &events);
// … gets unchanged at call sites initially
Loader:
using GlbHistoryBackendFactory = std::function<std::unique_ptr<GlbHistoryBackend>(const std::string &json_params)>;
// register "clickhouse", "worker", "events_file", "events_syslog"
Fan-out: keep current semantics — all backends that implement write get the batch; first success for reads.
Compatibility: keep glb_history_*.h wrappers calling the C++ facade so existing .c call sites compile during migration.
9. Code layout (proposed)
include/
glb_history.h # thin C-compatible wrappers (temporary)
glb_history.hpp # C++ facade + GlbHistoryEvent + GlbHistoryBackend
objects/processing/glb_history_event.hpp
src/libs/glb_history/
history.cpp # was history.c — init, fan-out, registry
history_backend.hpp # GlbHistoryBackend
history_clickhouse.cpp # + add_events
history_worker.cpp # optional put_events later
history_events_file.cpp # NEW
history_events_syslog.cpp # NEW
src/zabbix_server/glb_problems_keeper/
problems_export.hpp/cpp # build envelope + data_json, enrich, add_events
problems_storage.hpp # emit on lifecycle paths
glb_problem_update_t.hpp # carry message text into update path
database/clickhouse/schema.sql # events table (envelope + data String)
conf/zabbix_server.conf.full # HistoryModule event sink docs
10. Implementation plan (checklist)
Phase A — Protocol & HistoryModule decisions (done in this doc)
- [x] Envelope:
eventid,clock,ns,type,data. - [x] Lifecycle
typecatalog (create/recover/close/ack/message/severity/suppress/rank + reserved action.*). - [x] HistoryModule-only configuration (no EventsModule).
- [x] Events as separate write channel (not
write_types).
Phase B — Convert history interface to C++
Prerequisite / parallel track for clean add_events support.
- [ ] Inventory all
glb_history_*call sites (server, dbsyncer, trapper, proxy if any). - [ ] Introduce
glb_history.hppwithGlbHistoryBackend+GlbHistoryEvent. - [ ] Move dispatcher from
history.c→history.cpp; replaceAPI_CALLBACKSfan-out for history withstd::vector<std::unique_ptr<GlbHistoryBackend>>(or dual-register during transition). - [ ] Rename/port
history_clickhouse.c→.cppimplementingGlbHistoryBackend. - [ ] Rename/port
history_worker.c→.cppimplementingGlbHistoryBackend. - [ ] Named factory registry (
clickhouse,worker, …) instead ofstrstr. - [ ] Keep
include/glb_history.hwrappers so existing callers compile. - [ ] Remove untyped
glb_register_callback(HISTORY_*)usage from history modules once backends own registration via factory. - [ ] Build + smoke-test history write/read/trends with ClickHouse only (no behaviour change).
- [ ] Document module authoring: implement backend, register factory, parse JSON params.
Phase C — Events channel + sinks
- [ ] Add
add_eventstoGlbHistoryBackend+glb_history_add_eventsfan-out. - [ ] Implement
events_filebackend. - [ ] Implement
events_syslogbackend. - [ ] ClickHouse:
write_eventsflag +glaber.eventstable (envelope columns +dataString). - [ ] Extend loader for
events_file/events_syslognames.
Phase D — problems_keeper emission
- [ ] Host/group enricher (DC-backed cache).
- [ ] Return/capture
recovery_event_idfrom close path intodata. - [ ] Extend
glb_problem_update_t(or parallel payload) to carry message text forproblem.message. - [ ] Helper:
emit_problem_event(type, eventid, clock, ns, data_object). - [ ] Emit on create / recover / close / all
PROBLEM_UPDATE_*/ suppress toggles. - [ ] Best-effort: never fail keeper transaction because of export.
- [ ] Tests: envelope JSON for create, recover, severity, message.
Phase E — Hardening
- [ ] Stop unused legacy
problems-*.ndjsonopens in dbsyncer/main/taskmanager when new path is used. - [ ] Optional
ExportDir→ syntheticevents_fileshim. - [ ] Metrics: exported / dropped / enrich failures.
- [ ] Update
conf/zabbix_server.conf.full+ link from EVENTS_EXPORT.md. - [ ] Delete commented
zbx_export_eventsonce parity is accepted.
Phase F — Follow-ups
- [ ] Worker
put_eventsprotocol. - [ ] Emit
action.execute/action.notifyfrom escalator/alerter (or via keeper). - [ ] Real
nsthrough trigger → keeper. - [ ] ClickHouse materialized views for common
datafields. - [ ] Drop C wrappers / finish
trend_tC++ cleanup when safe.
11. MVP
- Finish Phase B enough that backends are C++ classes (even if clickhouse/worker are thin adapters first).
GlbHistoryEvent+glb_history_add_events.- Emit at least
problem.createandproblem.recover/problem.closewith enricheddata.problemwhere applicable. HistoryModule=events_file;{…}andevents_syslog;{…}end-to-end.- Optionally ClickHouse
write_eventsin the same slice (schema fits the envelope naturally).
Success criteria: PROBLEM→OK yields two NDJSON envelopes with stable eventid, distinct type, and JSON data; ack/message types can land immediately after with the same envelope; keeper survives sink failures.
12. Open questions
- Manual close: emit only
problem.close, or alsoproblem.recover? - Should suppress-by-maintenance/trigger emit export events (can be noisy)?
- Syslog priority: derive from
type, fromdata.problem.severity, or fixedLOG_INFO? - ClickHouse: store
dataasStringvs nativeJSONtype (depends on CH version)? - Classic NDJSON adapter: required for MVP or drop in favour of the new envelope only?
- When revising ClickHouse schema for host/group/tag/name search: separate table vs materialized columns on
data?
13. Summary
| Topic | Conclusion |
|---|---|
| Protocol | Structured eventid + clock + ns + type; unstructured data JSON for everything else |
| Event coverage | All problem lifecycle ops (+ reserved action.*); extensible via new type values |
| Config | HistoryModule only |
| API | New history write channel add_events (sibling of history/trends) |
| History code | Convert callback-table hybrid to C++ GlbHistoryBackend + factory registry |
| ClickHouse | Opt-in write_events; example envelope+data schema only — not search-optimised, subject to change (§7) |
| File / syslog | Dedicated HistoryModules (§5, §6) |
| Owner | problems_keeper builds and emits; modules only transport/store |