Node DSL

While Arborist has pluggable hooks for sourcing node data from various sources (databases, LDAP, etc), it only ships with file sources by default. This reference page outlines all possible options for this file backed source.

Please note: Any installed custom node types may extend or add DSL syntax options. See the RDoc for that specific node type -- this page only documents core Arborist.

• Hosts
• Services
• Resources
• Dependencies
• Flap Detection
• Example

Overview

Nodes files are loaded in by the Manager at startup. You can organize them however you please. Loading is 2-phase, so ordering by dependencies is not necessary. Files will be discovered irrespective of directory depth. Files must end in an .rb extension to be considered.

Of the three node types outlined on the Manager page, only the Host node type is referenced explicitly in the DSL. The service and resource nodes are normally intended to be a byproduct of describing the desired state of the host. The Host nodetype is assumed to be accessible over IP networks.

Hosts

Host nodes are created via the Arborist::Host class, with a unique identifier argument, and a ruby block with node keywords. All keywords are optional (though clearly not providing at least an address will make network monitoring more difficult than necessary.)

Host Keywords

keyword	description
address	One or more IP addresses (IPv4 or IPv6) to attach to this host. This can be called multiple times. If provided a hostname, it is automatically looked up via DNS.
config	Arbitrary key/value pair data to carry with the node. Monitors and Observers can use this data however they please. This can be called multiple times.
description	A human readable String describing the host's purpose.
depends_on	Other nodes that should be functional for this node to be considered up. See the Secondary Dependencies section below for syntax detail.
hostname	The canonical FQDN of the host node, if not determined via DNS from the address keyword.
parent	Attach this Host node under another node's identifier, forming the Manager tree hierarchy. If omitted, the Host node is attached to the root Arborist node.
resource	Create a new Arborist::Resource node as a child of this host. See Resource Options below for details.
service	Create a new Arborist::Service node as a child of this host. See Service Options below for details.
tags	One or more comma separated labels to attach to this node, for easy cross-selections with monitors and observers. This can be called multiple times.

Host Example

A full example that incorporates other topics in this category can be found later in this document. This is just an example of how to initialize a host node.

# Via the DSL
Arborist::Host 'unique-identifier' do
    address '10.1.23.4'
    # ...
end

# Ruby idiomatic
hostnode = Arborist::Node.create( :host, 'unique-identifier' )
hostnode.address( '10.1.23.4' )

Services

Services describe network accessible applications that run on their parent Arborist::Host node. There is one required one argument - the service name. This can be a well-known service (listed in /etc/services), or any label of your choosing.

A service node is created underneath its Arborist::Host in the tree with a fully-qualified identifier based on its parent identifier. For example, a service node labeled smtp under a host node with the identifier of mail-server would be automatically created with an identifier of 'mail-server-smtp.

Service Keywords

keyword	description
address	A single IP addresses (IPv4 or IPv6) to attach to this service. This is normally inherited automatically from the parent host node. This is to call out a specific IP address if the host has multiple. Note: The address provided must be present in the parent node.
app_protocol	An optional hint for monitors for how to communicate with this service, such as running an http service on a non-standard port.
config	Arbitrary key/value pair data to carry with the node. Monitors and Observers can use this data however they please. This can be called multiple times.
description	A human readable String describing the service.
depends_on	Other nodes that should be functional for this node to be considered up. See the Secondary Dependencies section below for syntax detail.
protocol	The IP protocol used by this service. Defaults to tcp.
port	The TCP/IP port the service communicates on. If a well-known service, this is determined automatically. Otherwise, it must be specified.
tags	One or more comma separated labels to attach to this node, for easy cross-selections with monitors and observers. This can be called multiple times.

Service Examples

# A simple example, using an /etc/services lookup for a well-known service.
service 'ssh'

# Any service not well known needs to manually specify a port.
# We also specify the optional application protocol here.
service 'rest-api', port: 8080, app_protocol: 'http'

# Default protocol is 'tcp', but you can manually set it.
service 'dhcp', port: 67, protocol: 'udp'

# A more elaborate example.
service 'internal' do
    description 'Proprietary weirdo service'
    protocol 'udp'
    port 9223
    app_protocol 'custom'
    config apikey: '8034ac45-0c3b-11e8-83d9-7cd30acd3ed8'
    depends_on 'frontend-proxy'
end

Resources

A resource node describes a finite property of a parent Arborist::Host node that is capable of being measured. Resources aren't directly responsible for the health of a host, but rather contribute towards it. CPU, disk capacity, available addressable memory, file descriptors, etc.

Resource nodes are created underneath their Arborist::Host in the tree with a fully-qualified identifier based on the parent identifier. There is one required argument, the resource label. A resource node labeled memory under a host node with the identifier of mail-server would be automatically created with an identifier of mail-server-memory.

Resource Keywords

keyword	description
address	A single IP addresses (IPv4 or IPv6) to attach to this resource. This is normally inherited automatically from the parent host node. This is to call out a specific IP address if the host has multiple. Note: The address provided must be present in the parent node.
category	A label for this resource, for easier selection from monitors and observers. If unset, this is the same as the identifier.
config	Arbitrary key/value pair data to carry with the node. Monitors and Observers can use this data however they please. This can be called multiple times.
description	A human readable String describing the resource.
depends_on	Other nodes that should be functional for this node to be considered up. See the Secondary Dependencies section below for syntax detail.
tags	One or more comma separated labels to attach to this node, for easy cross-selections with monitors and observers. This can be called multiple times.

Resource Examples

# A simple example.
resource 'load'

# An example that separates label from category.
resource '5-minute-load', category: 'load'

# A more elaborate example.
resource 'disk-capacity' do
    description 'Available hard drive space for mounts'
    category 'disk'
    config exclude: [ '^/var/tmp$' ]
    depends_on 'primary-storage-iscsi'
end

Secondary Dependencies

The depends_on keyword available to all node types creates one or more Arborist secondary dependencies. Primary dependencies are inferred directly from node ancestors - a service is automatically quieted and removed from search results if its parent node is down. A secondary dependency is a human declaration of a relationship between two nodes, that creates links across the tree.

When a node transitions to any state that removes it from search results (down, disabled, quieted), any nodes that indicate they are dependant on it are automatically transitioned to quieted with the reason.

Additionally, using the deps API call, you can ask a node what depends on me? This can make error reporting significantly more informative.

Dependency Examples

In the most simple case, you can just provide a list of fully qualified identifiers to the depends_on keyword. By default, all indicated nodes must be up for the dependency to evaluate successfully.

# All of these nodes must be operational for this node to be considered "okay".
depends_on 'primary-storage-iscsi', 'master-database'

There's a syntax helper if you want to indicate the same service/resource dependency on multiple host nodes:

# This expands to 'host1-disk', 'host2-disk', 'host3-disk'.
depends_on 'disk' on: [ 'host1', 'host2', 'host3' ]

Lets say a service relies on a load-balanced backend, and only needs one of them to be running successfully to be considered operational. This requires the any_of declarative.

# Only one of these dependencies needs to be up for this node to be considered "okay".
depends_on any_of( 'server-1-ldap', 'server-2-ldap' )

Technically, depends_on always wraps arguments in an all_of declaration. Knowing this, you can create complex dependency descriptions that are evaluated depth first.

# This node needs either 2 functional database backends,
# OR just one (of many) memcache nodes up to be considered "okay".
depends_on \
    any_of \
        any_of( 'memcache', on: ['server-1', 'server-2', 'server-3'] ),
        all_of( 'database' on: ['dbnode-1', 'dbnode-2'] )

Flap Detection

Arborist provides a path to squelch a node that is changing state too frequently. While disabled by default, this can be enabled globally in the configuration file by adjusting the status_history_size and flap_threshold values.

---
arborist:
  node:
    status_history_size: 15
    flap_threshold: 5

The above configuration says mark nodes as flapping if their status changes 5 times within 15 updates. Because frequency is an artifact of the Monitor, and notification of the Observer, what is considered too frequently can be individually adjusted per node:

keyword	description
flap_threshold	How many individual status changes in history should mark this node as flapping?
status_history_size	How many total status updates to consider when checking for a flapping node.

It should probably be noted that a node that is considered flapping has no direct behavioral changes. It is simply marked as such, so your Observer(s) can be crafted to make better informed decisions.

Full Example

Arborist::Host 'primary-database' do
    description 'PostgreSQL master server'
    address 'db-master.example.com'
    parent 'network-switch-12'

    config os: '11-1-release'
    tags :database,
         :postgresql,
         :freebsd

    service 'ssh'
    service 'postgresql', port: 5432 do
        depends_on 'iscsi', on: 'primary-storage'
    end

    resource 'load', description: 'machine load'
    resource 'disk' do
        description 'partition capacity'
        config include: [ '^/var/db' ]
    end
end