Skip to content

Problem events export — design & implementation plan

Related: EVENTS_EXPORT.md (legacy Zabbix file-export behaviour).

Owner process: problems_keeper. Problem create / recover / close / acknowledge / suppress / messages are its responsibility. Export is implemented as a HistoryModule write channel.

Sinks:


1. Verdict: data available today

glb_problem_t alone is not enough for useful export. Hostnames/groups are the obvious gap; recovery event id and an emission hook are also missing.

1.1 What problems_keeper already has

Field On glb_problem_t Notes
Problem eventid Yes Snowflake via keeper id_generator
Trigger / objectid Yes
clock (problem time) Yes From trigger notify
name Yes Macro-expanded after zbx_add_event
severity Yes
tags Yes {tag,value}
hostids Yes IDs only
Recovered flag / r_clock Yes Private r_clock; no recovery event id
Ack / suppress / symptom / cause Yes Needed for lifecycle export types

1.2 Gaps

Needed Available? Gap
Host technical / visible names No (hostids only) Resolve via DC / config cache
Host groups No Resolve via DC / DB
ns Always 0 today Emit 0 in v1 or plumb later
Recovery eventid Not on glb_problem_t Capture from close_trigger_event()
Message text on updates Parsed in trapper JSON, not kept on glb_problem_update_t Carry message into export payload
Outbound export hook None Emit on every relevant state transition

Bottom line: keeper has problem identity and most semantics. It needs enrichment (hosts/groups), recovery id at close, message text on MESSAGE updates, and a single emission API for all lifecycle types.


2. Goals

  1. Export all problem-related lifecycle events from problems_keeper (not only PROBLEM/OK).
  2. Wire format: few structured index fields + one unstructured JSON payload (data) so new event kinds do not require schema churn in every sink.
  3. Modular sinks via HistoryModule only (Name;{json}), including file, syslog, ClickHouse.
  4. Treat problem events as a sibling history write channel (same pattern as trends).
  5. Convert the history subsystem interface to C++ as part of making events a first-class channel.

Non-goals for v1:

  • Re-enable legacy ExportDir / ExportType=events as the primary API (optional shim only).
  • Full UI for browsing exported events.
  • Perfect classic Zabbix NDJSON parity as the internal protocol (sinks may still project a classic-compatible view if needed).

3. Export protocol draft

3.1 Design principles

  • Envelope is stable. Only a small set of structured fields is required for routing, indexing, and time-ordering.
  • Payload is extensible. Type-specific content lives in an unstructured JSON object (data). Adding a new lifecycle event means adding a type value and documenting data keys — not changing the envelope or every module’s C struct.
  • Problem content is not flattened into the envelope. Snapshot / diff of the problem goes inside data (e.g. data.problem), so sinks that only care about type + time do not depend on problem schema.
  • One NDJSON line / one syslog line / one ClickHouse row = one envelope.

3.2 Envelope (structured fields)

Every exported record:

Field Type Required Meaning
eventid uint64 yes Problem event id this record concerns (stable correlation key across lifecycle)
clock int yes Unix seconds of this lifecycle event
ns int yes Nanoseconds; 0 until plumbed
type string yes Lifecycle event type (see below)
data object yes Unstructured JSON payload for this type (may be {})

Wire example (NDJSON line):

{"eventid":12345,"clock":1710000000,"ns":0,"type":"problem.create","data":{...}}

Notes:

  • For recovery, eventid remains the problem id. Recovery event id (if any) belongs in data (e.g. data.r_eventid), not as a second top-level id. This keeps filtering “everything about problem X” trivial.
  • Optional future envelope fields (only if clearly universal): source (problems_keeper), userid. Prefer putting actor info in data until proven necessary.

3.3 Event types

type is a stable string. Use dotted names so families sort/filter cleanly.

type When emitted Typical data contents
problem.create New problem created Full problem snapshot (+ enriched hosts/groups)
problem.recover Problem recovered (trigger OK / correlation / cleanup close that creates recovery) r_eventid, recovery clock, optional problem snapshot or minimal refs
problem.close Manual close (ack action CLOSE) userid, r_eventid, reason/source
problem.acknowledge Acknowledged userid, optional message
problem.unacknowledge Unacknowledged userid
problem.message Message added userid, message
problem.severity Severity changed userid, old_severity, new_severity
problem.suppress Suppressed (by ack) userid, suppress_from, suppress_until
problem.unsuppress Unsuppressed userid
problem.rank_cause Ranked to cause userid, related ids
problem.rank_symptom Ranked to symptom userid, cause_eventid
problem.suppress_maintenance Suppressed because host(s) in maintenance hostids / flags
problem.suppress_trigger Suppressed because trigger non-functional / UNKNOWN triggerid / flags
problem.unsuppress_maintenance Maintenance suppress cleared
problem.unsuppress_trigger Trigger-state suppress cleared
action.execute Reserved: action/escalation step executed against the problem actionid, esc_step, media/operation summary
action.notify Reserved: notification sent channel, subject/body refs, status

Existing keeper actions map from glb_problem_update_t:

PROBLEM_UPDATE_* Export type
(create path) problem.create
(recover path) problem.recover
CLOSE problem.close (and/or problem.recover — pick one primary; recommend emit problem.close for manual, problem.recover for automatic)
ACKNOWLEDGE problem.acknowledge
UNACKNOWLEDGE problem.unacknowledge
MESSAGE problem.message
SEVERITY problem.severity
SUPPRESS problem.suppress
UNSUPPRESS problem.unsuppress
RANK_TO_CAUSE problem.rank_cause
RANK_TO_SYMPTOM problem.rank_symptom

New functionality later = new type string + documented data keys. No envelope change.

3.4 data conventions

  • Always a JSON object.
  • Prefer nested objects over flat mega-keys:
  • data.problem — snapshot of problem fields when useful (name, severity, objectid, tags, hosts, groups, acknowledged, suppressed, …).
  • data.change — diff-style fields for updates (old_severity, new_severity, message, …).
  • data.actor — { "userid": … } when a user initiated the change.
  • Enrichment (hostnames, groups) is done once in problems_keeper and placed into data.problem.hosts / data.problem.groups for types that need a snapshot.
  • Sinks must tolerate unknown keys and unknown type values (forward-compatible).
  • Classic Zabbix PROBLEM/OK projection (if needed for an old consumer) is a sink-side adapter from problem.create / problem.recover|close, not the internal protocol.

Example — create:

{
  "eventid": 12345,
  "clock": 1710000000,
  "ns": 0,
  "type": "problem.create",
  "data": {
    "problem": {
      "name": "High CPU utilization",
      "severity": 3,
      "objectid": 1001,
      "tags": [{"tag": "scope", "value": "performance"}],
      "hosts": [{"hostid": 10, "host": "web01", "name": "Web server 01"}],
      "groups": ["Linux servers"]
    }
  }
}

Example — severity change:

{
  "eventid": 12345,
  "clock": 1710000100,
  "ns": 0,
  "type": "problem.severity",
  "data": {
    "actor": {"userid": 7},
    "change": {"old_severity": 3, "new_severity": 4}
  }
}

Example — message:

{
  "eventid": 12345,
  "clock": 1710000200,
  "ns": 0,
  "type": "problem.message",
  "data": {
    "actor": {"userid": 7},
    "change": {"message": "Investigating with netops"}
  }
}

3.5 In-process record (C++)

struct glb_history_event_t {
    uint64_t    eventid = 0;     // problem id
    int         clock = 0;
    int         ns = 0;
    std::string type;            // e.g. "problem.create"
    std::string data_json;       // serialized JSON object for `data`
};

Keeper builds type + a JSON object (via zbx_json or equivalent), then calls the history API. Modules receive the envelope; they do not parse problem internals unless they choose to.


4. Architecture (HistoryModule only)

4.1 Decision

Configuration and loading are only via:

HistoryModule=<name>;{json}

There is no separate EventsModule (or other parallel loader). Event sinks are history modules that register the events write channel. ClickHouse opts in with "write_events":true in its existing JSON.

Do not add events to write_types=dbl,uint,… — those are item value types. Events are a separate write channel, same idea as trends.

4.2 Write channels

GLB_MODULE_API_HISTORY_WRITE              // item values
GLB_MODULE_API_HISTORY_WRITE_TRENDS       // trends
GLB_MODULE_API_HISTORY_WRITE_EVENTS       // NEW: problem lifecycle events
GLB_MODULE_API_HISTORY_READ…
GLB_MODULE_API_DESTROY

Public API:

int glb_history_add_events(const std::vector<glb_history_event_t> &events);
// or span/pointer+count equivalent
problems_keeper
    │  any lifecycle change
    ▼
build glb_history_event_t (envelope + data_json, enrich once)
    │
    ▼
glb_history_add_events()          // HistoryModule fan-out
    ├─ events_file      → NDJSON files
    ├─ events_syslog    → syslog
    ├─ clickhouse       → events table (if write_events)
    └─ worker           → optional later (put_events)

4.3 Emission points

Keeper path type
create_add_new_problem problem.create
auto recover problem.recover
PROBLEM_UPDATE_CLOSE problem.close
other PROBLEM_UPDATE_* matching type from §3.3
maintenance / trigger suppress toggles problem.suppress_* / problem.unsuppress_*
future action/escalation hooks action.*

Export is best-effort: sink failures must not roll back problem processing.

4.4 Enrichment

Resolve hostnames/groups in problems_keeper (or a shared helper) when building data.problem for types that include a snapshot. Cache by hostid; tolerate deleted hosts with empty names.

Event-only modules (events_file, events_syslog) register only the events write channel (+ destroy). ClickHouse registers it when "write_events":true.

Sink-specific configuration and behaviour: §5 File export, §6 Syslog export, §7 ClickHouse export.

Optional: if legacy ExportDir is set with events enabled, synthesize an internal events_file module. New installs use HistoryModule=events_file;….


5. File export

Module name: events_file.

Purpose: append problem lifecycle envelopes to local NDJSON files for offline shipping, SIEM pickup, or debugging — replacement for the old ExportDir problems writer.

5.1 Configuration

HistoryModule=events_file;{
  "dir":"/var/lib/zabbix/export",
  "file_size":"1G",
  "prefix":"problems"
}
JSON key Meaning Default
dir Output directory (must exist, writable) required
file_size Rotate when current file would exceed this size 1G
prefix File name prefix problems

5.2 Behaviour

  • One NDJSON line per envelope (eventid, clock, ns, type, data).
  • File name pattern: {dir}/{prefix}-problems-keeper-{n}.ndjson (single writer in the problems_keeper process; avoid per-history-syncer files).
  • Rotation: rename to .old when size limit is hit, then open a new file (same idea as legacy zbxexport).
  • Best-effort: I/O errors drop the record and log (rate-limited); do not fail problem processing.
  • Optional legacy shim: map ExportDir + ExportType=events to this module.

5.3 Implementation notes

  • New backend: src/libs/glb_history/history_events_file.cpp.
  • Reuse rotation/open ideas from src/libs/zbxexport/export.c where useful; do not resurrect dbsyncer-side problem file openers as the primary path.

6. Syslog export

Module name: events_syslog.

Purpose: stream the same envelopes to the local syslog / journal for real-time ops tooling without a local file consumer.

6.1 Configuration

HistoryModule=events_syslog;{
  "ident":"glaber-events",
  "facility":"local0"
}
JSON key Meaning Default
ident openlog identity glaber-events
facility Syslog facility (local0…local7, user, daemon, …) local0

6.2 Behaviour

  • openlog(ident, LOG_PID|LOG_NDELAY, facility) at module init.
  • One syslog message = one envelope JSON object (same protocol as file).
  • Priority mapping (initial proposal): derive from type and/or data.problem.severity when present; recoveries and messages default to LOG_INFO; high/disaster create → LOG_ERR. Exact mapping is still an open question (§12).
  • No local buffering beyond syslogd; failures logged rate-limited.
  • Best-effort only.

6.3 Implementation notes

  • New backend: src/libs/glb_history/history_events_syslog.cpp.
  • Event-only: implements add_events (+ destroy), ignores history/trends.

7. ClickHouse export

Module: existing clickhouse HistoryModule, extended with an events write path.

Purpose: long-term archival of problem lifecycle events alongside item history/trends, using the same module connection and flush machinery.

7.1 Configuration

HistoryModule=clickhouse;{
  "url":"http://127.0.0.1:8123",
  "username":"default",
  "password":"password",
  "dbname":"glaber",
  "write_types":"dbl,str,log,uint,text",
  "write_events":true
}
JSON key Meaning Default
write_events Register events write channel and insert into events table false (no behaviour change for existing installs)
other keys Unchanged ClickHouse history module options see existing module

7.2 Example table schema (illustrative only)

Note: The schema below is purely an example for discussion and early prototyping. It is not optimised or prepared for searching problems by host, group, tag, or name. Indexes, extracted columns, ORDER BY, and possibly separate dictionaries/materialized views for those dimensions are subject to change later. Do not treat this DDL as a stable contract.

CREATE TABLE glaber.events
(
    day Date,
    clock DateTime,
    ns UInt32,
    eventid UInt64,
    type LowCardinality(String),
    data String           -- JSON object (full unstructured payload)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(day)
ORDER BY (eventid, clock, type)
TTL day + INTERVAL 24 MONTH;

Properties of this example:

  • Envelope fields are columns; data stays a JSON string so new type values need no ALTER.
  • Suitable for append-only archival and “all events for eventid” style reads.
  • Not suited yet for efficient filters like “problems on host X”, “tag=Y”, “group=Z”, or “name contains …” — those would require a later schema revision (e.g. materialized columns, arrays, secondary indexes, or a search-oriented table).

7.3 Behaviour

  • Batch inserts on the same flush timer as history when write_events is enabled.
  • Writes are best-effort relative to problems_keeper (same as other sinks).
  • Reads of events from ClickHouse are optional for v1 (write/archival first).

7.4 Implementation notes

  • Extend history_clickhouse backend with add_events.
  • Add example DDL under database/clickhouse/schema.sql with the same “example / subject to change” comment.
  • Later: materialized views or a revised table for host/group/tag/name search — tracked as follow-up, not MVP.

8. History interface today — analysis

8.1 Current shape

Piece Location Language / style
Public API include/glb_history.h Mixed: C functions + C++ types (ZBX_DC_HISTORY class, std::vector in get API)
Internal callbacks src/libs/glb_history/history.h C function pointers + void *data
Dispatcher src/libs/glb_history/history.c C file using C++ features; fan-out via API_CALLBACKS[]
ClickHouse / worker history_clickhouse.c, history_worker.c C-with-classes / C++ in .c units
Registration glb_register_callback() in modules API Untyped void (*)(void) casts

Problems:

  1. Not a real C++ interface — modules are C callbacks with manual casts; no ownership, no virtual API, hard to extend safely.
  2. Hybrid already hurts — headers pull C++ (vector, classes) while files are named .c and still use C callback tables.
  3. ZBX_DC_HISTORY is a class in a “C” header with incomplete serialize notes; trend_t remains a C struct with raw char *.
  4. Adding events on the old callback table repeats the same untyped pattern.
  5. Init discovery is strstr(name, "clickhouse") / "worker" — fragile; C++ registry maps cleanly to module names.

8.2 Target C++ interface (draft)

// include/glb_history.hpp (name TBD)

struct GlbHistoryEvent {
    uint64_t    eventid = 0;
    int         clock = 0;
    int         ns = 0;
    std::string type;
    std::string data_json;
};

class GlbHistoryBackend {
public:
    virtual ~GlbHistoryBackend() = default;

    virtual int add_history(ZBX_DC_HISTORY *history, int history_num) { return SUCCEED; }
    virtual int add_trend(trend_t *trend) { return SUCCEED; }
    virtual int add_events(const std::vector<GlbHistoryEvent> &events) { return SUCCEED; }

    virtual int get_history(/*…*/) { return FAIL; }
    virtual int get_history_aggregates_json(/*…*/) { return FAIL; }
    virtual int get_trends_json(/*…*/) { return FAIL; }
    virtual int get_trends_aggregates_json(/*…*/) { return FAIL; }
};

// Facade used by server code
int glb_history_init(char **history_modules, char **error);
int glb_history_add_history(ZBX_DC_HISTORY *history, int history_num);
int glb_history_add_trend(trend_t *trend);
int glb_history_add_events(const std::vector<GlbHistoryEvent> &events);
// … gets unchanged at call sites initially

Loader:

using GlbHistoryBackendFactory = std::function<std::unique_ptr<GlbHistoryBackend>(const std::string &json_params)>;
// register "clickhouse", "worker", "events_file", "events_syslog"

Fan-out: keep current semantics — all backends that implement write get the batch; first success for reads.

Compatibility: keep glb_history_*.h wrappers calling the C++ facade so existing .c call sites compile during migration.


9. Code layout (proposed)

include/
  glb_history.h                 # thin C-compatible wrappers (temporary)
  glb_history.hpp               # C++ facade + GlbHistoryEvent + GlbHistoryBackend
  objects/processing/glb_history_event.hpp

src/libs/glb_history/
  history.cpp                   # was history.c — init, fan-out, registry
  history_backend.hpp           # GlbHistoryBackend
  history_clickhouse.cpp        # + add_events
  history_worker.cpp            # optional put_events later
  history_events_file.cpp       # NEW
  history_events_syslog.cpp     # NEW

src/zabbix_server/glb_problems_keeper/
  problems_export.hpp/cpp       # build envelope + data_json, enrich, add_events
  problems_storage.hpp          # emit on lifecycle paths
  glb_problem_update_t.hpp      # carry message text into update path

database/clickhouse/schema.sql  # events table (envelope + data String)

conf/zabbix_server.conf.full    # HistoryModule event sink docs

10. Implementation plan (checklist)

Phase A — Protocol & HistoryModule decisions (done in this doc)

  • [x] Envelope: eventid, clock, ns, type, data.
  • [x] Lifecycle type catalog (create/recover/close/ack/message/severity/suppress/rank + reserved action.*).
  • [x] HistoryModule-only configuration (no EventsModule).
  • [x] Events as separate write channel (not write_types).

Phase B — Convert history interface to C++

Prerequisite / parallel track for clean add_events support.

  • [ ] Inventory all glb_history_* call sites (server, dbsyncer, trapper, proxy if any).
  • [ ] Introduce glb_history.hpp with GlbHistoryBackend + GlbHistoryEvent.
  • [ ] Move dispatcher from history.c → history.cpp; replace API_CALLBACKS fan-out for history with std::vector<std::unique_ptr<GlbHistoryBackend>> (or dual-register during transition).
  • [ ] Rename/port history_clickhouse.c → .cpp implementing GlbHistoryBackend.
  • [ ] Rename/port history_worker.c → .cpp implementing GlbHistoryBackend.
  • [ ] Named factory registry (clickhouse, worker, …) instead of strstr.
  • [ ] Keep include/glb_history.h wrappers so existing callers compile.
  • [ ] Remove untyped glb_register_callback(HISTORY_*) usage from history modules once backends own registration via factory.
  • [ ] Build + smoke-test history write/read/trends with ClickHouse only (no behaviour change).
  • [ ] Document module authoring: implement backend, register factory, parse JSON params.

Phase C — Events channel + sinks

  • [ ] Add add_events to GlbHistoryBackend + glb_history_add_events fan-out.
  • [ ] Implement events_file backend.
  • [ ] Implement events_syslog backend.
  • [ ] ClickHouse: write_events flag + glaber.events table (envelope columns + data String).
  • [ ] Extend loader for events_file / events_syslog names.

Phase D — problems_keeper emission

  • [ ] Host/group enricher (DC-backed cache).
  • [ ] Return/capture recovery_event_id from close path into data.
  • [ ] Extend glb_problem_update_t (or parallel payload) to carry message text for problem.message.
  • [ ] Helper: emit_problem_event(type, eventid, clock, ns, data_object).
  • [ ] Emit on create / recover / close / all PROBLEM_UPDATE_* / suppress toggles.
  • [ ] Best-effort: never fail keeper transaction because of export.
  • [ ] Tests: envelope JSON for create, recover, severity, message.

Phase E — Hardening

  • [ ] Stop unused legacy problems-*.ndjson opens in dbsyncer/main/taskmanager when new path is used.
  • [ ] Optional ExportDir → synthetic events_file shim.
  • [ ] Metrics: exported / dropped / enrich failures.
  • [ ] Update conf/zabbix_server.conf.full + link from EVENTS_EXPORT.md.
  • [ ] Delete commented zbx_export_events once parity is accepted.

Phase F — Follow-ups

  • [ ] Worker put_events protocol.
  • [ ] Emit action.execute / action.notify from escalator/alerter (or via keeper).
  • [ ] Real ns through trigger → keeper.
  • [ ] ClickHouse materialized views for common data fields.
  • [ ] Drop C wrappers / finish trend_t C++ cleanup when safe.

11. MVP

  1. Finish Phase B enough that backends are C++ classes (even if clickhouse/worker are thin adapters first).
  2. GlbHistoryEvent + glb_history_add_events.
  3. Emit at least problem.create and problem.recover/problem.close with enriched data.problem where applicable.
  4. HistoryModule=events_file;{…} and events_syslog;{…} end-to-end.
  5. Optionally ClickHouse write_events in the same slice (schema fits the envelope naturally).

Success criteria: PROBLEM→OK yields two NDJSON envelopes with stable eventid, distinct type, and JSON data; ack/message types can land immediately after with the same envelope; keeper survives sink failures.


12. Open questions

  1. Manual close: emit only problem.close, or also problem.recover?
  2. Should suppress-by-maintenance/trigger emit export events (can be noisy)?
  3. Syslog priority: derive from type, from data.problem.severity, or fixed LOG_INFO?
  4. ClickHouse: store data as String vs native JSON type (depends on CH version)?
  5. Classic NDJSON adapter: required for MVP or drop in favour of the new envelope only?
  6. When revising ClickHouse schema for host/group/tag/name search: separate table vs materialized columns on data?

13. Summary

Topic Conclusion
Protocol Structured eventid + clock + ns + type; unstructured data JSON for everything else
Event coverage All problem lifecycle ops (+ reserved action.*); extensible via new type values
Config HistoryModule only
API New history write channel add_events (sibling of history/trends)
History code Convert callback-table hybrid to C++ GlbHistoryBackend + factory registry
ClickHouse Opt-in write_events; example envelope+data schema only — not search-optimised, subject to change (§7)
File / syslog Dedicated HistoryModules (§5, §6)
Owner problems_keeper builds and emits; modules only transport/store