Event API

This is the low-level API for interacting directly with events generated from the Arborist Manager. This document is intended for developers writing their own Arborist observers(s).

Overview

The Event API consists of an initial subscription to an event type (or class of events) via the tree-api, and a ZMQ socket to receive them afterwards. Messages are passed using msgpack for speed. Both libraries enjoy rich support across many different languages, so you should be able to easily interact with Arborist using the language of your choice.

Nomenclature in this document matches that used by Msgpack. ie, a Map is an associative array, Nil is null, etc.

Creating a Subscription

First, reference the tree-api documentation to connect to the Manager and create a new subscription object via the subscribe API.

On a successful subscription, the Manager will return a UUID that represents the specified criteria, and is used as the topic for connecting to the ZMQ socket.

If you locally keep track of generated subscription UUIDs, you can reduce load on the manager by sharing and reusing them across any number of subscribers, thanks to the nature of ZMQ pubsub.

Read more about subscription topics here and at the ZMQ Guide.

Connecting

Arborist uses a local ZMQ socket by default, at ipc:///tmp/arborist_events.sock. If configured to listen on an external address, you can connect to it from anywhere -- including broadcast domains, multicast, UDP, or a more traditional TCP destination. (Future features are planned to only allow trusted connections via ZMQ/ZAUTH. For the time being, you'll have to rely on external firewalling to limit access.)

Connections to the Manager API are of the ZMQ SUB typed socket. Other ZMQ socket types will error. The connection will block until events are available for processing. By default, the Manager only opens a local IPC socket. Remote connections require Manager configuration changes, such as:

---
arborist:
  event_api_url: tcp://10.3.0.75:5012

Event Structure

Raw events coming in off the wire are opaque MsgPack binary, and must be decoded to a Map.

All events have two top level keys:

key value
type String. The event label.
data Map. The payload of the event. The exact content depends on the event type.

Additionally, events of the node category also contain an identifier key, the unique node label that is generated the event.

Node Events

For reference, these are the possible node state transitions. indirect transitions are those that are caused by other node events.

For the majority of node events, the payload is the serialized state of the node. Omitting the entire payload here for the sake of brevity, but for each node type we'll call out some specifics.

node.acked

An acked event is issued once per state transition, when a node in a down state is acknowledged by a human. The payload is the serialized node.

The ack key in the payload may be of particular interest, which contains the who/why/when of how the acknowledgement event took place.

"status": "acked",
"ack": {
  "message": "okay!",
  "sender": "mahlon",
  "via": null,
  "time": "2017-10-26T16:14:40+00:00"
}

node.disabled

Issued once per state transition, when a node is in an up, unknown, or already acked state, and it is acknowledged by a human. The payload is the serialized node.

The ack key in the payload may be of particular interest, which contains the who/why/when of how the disabled event took place.

"status": "disabled",
"ack": {
  "message": "okay!",
  "sender": "mahlon",
  "via": null,
  "time": "2017-10-26T16:14:40+00:00"
}

node.down

Issued once per state transition, when a monitor check fails. The payload is the serialized node.

The errors key in the payload may be of particular interest. It's a Map of monitors that are signalling down and their individual error messages.

"errors": {
  "tcpport": "Connection refused"
}

node.quieted

Issued once per state transition, when a dependency of the node fails it's monitor check. The dependency can be a primary (a direct parent in the tree layout), or a secondary (cutting across the tree, an explicit dependency called out in the Node DSL).

The quieted_reasons key in the payload may be of particular interest, which contains what nodes caused the quieted event, and if they were a primary or secondary dependency.

"quieted_reasons": {
  "primary": "Parent disabled: mahlon-workstation"
}

node.unknown

Issued once per state transition. A node is in an unknown state when it hasn't had a monitor check, or when it needs retesting after its dependencies transition to up.

node.up

Issued once per state transition, when a monitor check succeeds. The payload is the serialized node.

node.warn

Issued once per state transition, when a monitor check succeeds, but indicates a future problem. The payload is the serialized node.

The warnings key in the payload may be of particular interest. It's a Map of monitors that are signalling down and their individual warning messages.

"warnings": {
  "disk": "Mount at /var is 87% full"
}

node.delta

This event is generated every time a node is updated, regardless of state transition. The payload is a Map, keyed for each attribute that was updated. The values are arrays - the first element was the last attribute value, the second element is what it was changed to.

{
  "thing": [ false, true ],
  "another": [ 1, 2 ]
}

node.update

This event is generated every time a node is updated, regardless of state transition. The payload is the current state of the serialized node.

System Events

Events generated by the system work a little differently than node events - they are a more traditional ZMQ topic, and don't require creating a Arborist Subscription request to use them. Just indicate what you're filtering on the ZMQ SUB socket directly.

require 'arborist/client'
client = Arborist::Client.new        # Create a client object
client.event_api.subscribe( 'sys.' ) # Subscribe to all available system events.

sys.node_added

Generated when a node is created within the running Manager via the graft API call. The payload is a single keypair Map, with the new node identifier.

{
  "node": "new_identifier"
}

sys.node_removed

Generated when a node is deleted within the running Manager via the prune API call. The payload is a single keypair Map, with the old node identifier.

{
  "node": "old_identifier"
}

sys.heartbeat

The heartbeat event is only emitted if the configuration setting heartbeat_frequency is set. The value is in milliseconds. Using this event, clients can detect when the manager last started, and can issue re-subscriptions to node events if necessary.

{
  "start_time": "2017-10-30T16:36:46-0700",
  "version: "0.1.0",
  "runid": "614ec7569b2e675a28672c11507754ec"
}

Example

This example uses the Arborist client library to demonstrate a fire hose -- attaching a subscription object to the root node (that sees all node events), without any search criteria (all events match.)

The concepts here are the same regardless of your programming language.

If you're using Ruby, the included client and event libraries perform the majority of the heavy lifting for you.

require 'arborist/client'     # client library
require 'arborist/event_api'  # event api helpers

# Create a client object
client = Arborist::Client.new

# Create a new subscription object on the Manager.
# Without any arguments, this attaches to the root node, and "sees" all node events.
subid = client.subscribe

# Filter events with the criteria known from the subscription UUID.
client.event_api.subscribe( subid )

# Additionally, subscribe to heartbeat system events.
client.event_api.subscribe( 'sys.heartbeat' )

# Lets be nice and destroy the subscription object when 
# this script is finished.
at_exit do
    client.unsubscribe( subid )
end

# The #receive call blocks, so lets loop for each incoming message.
#
loop do
    message = client.event_api.receive

    # The #decode helper takes care of the MsgPack manipulations,
    # and returns the event as a parsed ruby hash.
    subid, event = Arborist::EventAPI.decode( message )

    puts "Got an event!: %p" % [ event ]
    puts
end