Workers

The idea

To extend Glaber's metrics processing so-called workers are used.

A worker is a simple program on any language that is using stdin/stdout and JSON to interact with Glaber and use any mission specific protocol to link Glaber with outside world.

For example, there is a syslog worker[link] - it accept Syslog network traffic, parses Syslog protocol and prints out data in JSONL format via stdout.

Unlike ExternalScripts, a worker is a long-running program that is expected to initialize once and then process lots of requests in the loop. This eliminates quite an overhead required for launching short-lived scripts.

Workers are launched and controlled by the Glaber server, so there is no need to demonize them and pay attention if they are running or not.

Workers are launched in an isolated forked process and do not affect Glaber is they have problems related to memory leaks, segmentation faults and etc. Glaber automatically restarts workers time to time, also it will bring up workers that failed and exited.

Workers automatically restarted if the worker executable file has changed making it easy to update workers. Workers might be killed as well, if needed, this will cause Glaber to re-run them.

Usage

Workers are used in many parts of Glaber:

1. history interface to ease implementing of new history storage engines
2. pinger interface to implement high-efficiency ICMP service
3. Server workers for user-created means of processing of incoming data

Implemtenting workers

A worker might be implemented on any language the system supports either Sript (interpretable) or Binary ones. Worker must be executable by user under which Glaber server is running (typically it is 'zabbix', on new installs it is 'glaber').

If an executable needs some special permissions, you might want to set +s bit or extended rights via extended linux permissions. For example, glb_map worker used for icmp checks needs sudo flag or PCAP extended permission to be able to use RAW sockets for efficient icmp packet crafting and PCAP permission for capturing responses.

A typical worker structure is:

Init();

While( new_data = GetNewDataFromSpecificProtocol()) {

    ProcessData(new_data);

    json = ConvertDataToJSON(new_data);
    Print(stdout, json);
}

Worker source code examples

Please explore workers in golang in Glaber project, C worker syslog_tools project

Performance and high-load

When Glaber server runs a worker, it increases stdin and stdout buffer sizes to maximum system possible values (about 10 Megabytes in recent systems). It's has been tested in production and works fine on data rates up to 30K NVPS.

Server does it's best to read out data from the worker as fast as possible. However on really high rates of data coming from worker, it's advised to implement worker-side stdout buffering and move data emitting to a separate thread or a kind of co-routine a language might have to avoid blocks.

This measure might be also useful if there are bursts when worker outputs data in high volumes for a short period of time and it's critical not to block worker on stdout operations during such a bursts.