Client Commands

The Arborist client wraps the Tree API into a friendly command line interface and Ruby object. It will likely be the most common way Administrators will interact with a running Arborist manager.

Accessing the Client

From Ruby, simply instantiate a new Client object.

#!/usr/bin/env ruby

require 'arborist'
require 'arborist/client'

# Loading the configuration explicitly is only necessary
# if it has been altered from defaults.
Arborist.load_config( '/path/to/config.yml' )

client = Arborist::Client.new

client.status #=> {"server_version"=>"0.1.0", "state"=>"running", "uptime"=>5268.985041273, "nodecount"=>8}

From the command line, create a new interactive Client shell via the Arborist tool.

arborist client
status
=> {"server_version"=>"0.1.0", "state"=>"running", "uptime"=>5268.985041273, "nodecount"=>8}

You'll be dropped into a newly instantiated Client context, making it easy to experiment, or slice-and-dice data from the Manager.

Because it's really just a Pry environment, any Pry commands and all Ruby code is valid. See Pry documentation for more info (start with help.)

Commands

ack

An ack on a node transitions it into an acked state if it was down, or to disabled if in any other state. This way you can indicate an awareness of a service outage, or preemptively disable parts of the tree when performing scheduled maintenance.

Argument Required? Description
identifier yes The unique identifier of the node to ack.
message yes A human readable message I'm working on this!
sender yes The human responsible for performing the ack.
via no An optional source for the acknowledgement, if you care to track such things.
time no Acks default to the time the call is performed. You can override the ack time if needed by supplying a Ruby Time object.

A regular node in an up state:

fetch( from: 'hotsoup', depth: 0 ).first[ 'status' ]
=> "up"

Acknowledge the host to indicate some work will take place on it, and you don't want to hear alerts about it, its children, or its dependants:

ack identifier: "hotsoup", message: "Upgrading things!", sender: "Mahlon"
=> true

The new state:

fetch( from: 'hotsoup', depth: 0 ).first[ 'status' ]
=> "disabled"

clear_ack

Explicitly remove a previously set ack. Note that during transitions from acked to up, the acknowledgement is automatically removed, since the down state has been remedied.

In the case of a disabled node having its ack cleared, it will transition to unknown, so it can be re-checked.

Argument Required? Description
identifier yes The unique identifier of the node to remove the ack from.

Removing an acknowledgement:

clear_ack identifier: "hotsoup"
=> true

The new state:

fetch( from: 'hotsoup', depth: 0 ).first[ 'status' ]
=> "unknown"

deps

Display nodes that depend on the given node identifier - both primary (implicit) and secondary (explicit) dependencies. You can craft what will be affected if this node is down? reports.

Argument Required? Description
identifier yes The identifier of the node to search for dependants.

A simple example configuration for a service with a secondary dependency:

Arborist::Host( 'hotsoup' ) do
    description "Mahlon's workstation"
    address     '10.3.0.75'
    tags        :workstation

    service 'example', port: 10000, description: 'An example service' do
        depends_on 'tb05-05-memcache'
    end
end

It's simple enough to ask what it's dependencies are:

fetch( from: 'hotsoup-example' ).first[ 'dependencies' ]
=> {"behavior"=>"all", "identifiers"=>["tb05-05-memcache"], "subdeps"=>[]}

Using deps, you can reverse the query, and ask for dependants of a node:

deps identifier: 'tb05-05'
=> {"deps"=>["hotsoup-example"]}

dependencies_of

Fetching the dependencies of a particular node is such a common case that there's also a way to combine the dependency query and search from above: Given an identifier, this method returns full node information, and can optionally organize the result set by an arbitrary node attribute.

Argument Required? Description
properties no String, or an Array of Strings. Only return the listed properties, instead of the entire node object.
partition no String. Rather than return a hash of node data keyed by identifier, return a Hash of Arrays keyed by this attribute.

A simple example configuration to demonstrate both explicit and implicit dependencies:

Arborist::Host( 'storage01' ) do
    description "Block storage provider"
    address     '10.3.0.10'
    service 'iscsi', protocol: 'tcp', port: 3260
end

Arborist::Host( 'vm-host' ) do
    description "Virtual Machine Host"
    address     '10.3.0.75'
    service 'memcache', protocol: 'tcp', port: '11211'
    depends_on 'storage01-iscsi'
end

Arborist::Host( 'www01' ) do
    description "Virtual webserver"
    address     '10.3.0.80'
    parent      'vm-host'
end

The default result is the same as the search method. Nodes are returned as a Hash keyed by identifier.

dependencies_of 'storage01-iscsi'
=> {"vm-host"=>
  {"type"=>"host",
   "status"=>"unknown",
   "tags"=>[],
   "parent"=>"_",
   "description"=>"Virtual Machine Host",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>["storage01-iscsi"], "subdeps"=>[]},
   "status_changed"=>"1969-12-31T16:00:00-08:00",
   "status_last_changed"=>"1969-12-31T16:00:00-08:00",
   "last_contacted"=>"1969-12-31T16:00:00-08:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{},
   "hostname"=>"10.3.0.75",
   "addresses"=>["10.3.0.75"]},
 "vm-host-memcache"=>
  {"type"=>"service",
   "status"=>"unknown",
   "tags"=>[],
   "parent"=>"vm-host",
   "description"=>nil,
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"1969-12-31T16:00:00-08:00",
   "status_last_changed"=>"1969-12-31T16:00:00-08:00",
   "last_contacted"=>"1969-12-31T16:00:00-08:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{},
   "addresses"=>["10.3.0.75"],
   "port"=>"11211",
   "protocol"=>"tcp",
   "app_protocol"=>"memcache"},
 "www01"=>
  {"type"=>"host",
   "status"=>"unknown",
   "tags"=>[],
   "parent"=>"vm-host",
   "description"=>"Virtual webserver",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"1969-12-31T16:00:00-08:00",
   "status_last_changed"=>"1969-12-31T16:00:00-08:00",
   "last_contacted"=>"1969-12-31T16:00:00-08:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{},
   "hostname"=>"10.3.0.80",
   "addresses"=>["10.3.0.80"]}}

Pivoting the dependencies on an arbitrary value of a node, while returning a subset of the node attributes -- in this case, group nodes on their type, so you can separate hosts from services/resources:

dependencies_of 'storage01-iscsi', partition: 'type', properties: %w[type description]
=> {
    "host"=>
        [
            {"type"=>"host", "description"=>"Virtual Machine Host", "identifier"=>"vm-host"},
            {"type"=>"host", "description"=>"Virtual webserver", "identifier"=>"www01"}
        ],
    "service"=>
        [
            {"type"=>"service", "description"=>nil, "identifier"=>"vm-host-memcache"}
        ]
   }

fetch

Pull all stored node information from the Manager. Returns a flattened array of all serialized nodes. With no arguments, this starts at the root node and returns every node in the Manager.

Argument Required? Description
from no The node identifier to start fetching from. Defaults to the root node.
depth no Integer. Only returns children to this level. A depth of 0 returns only a single node.
tree no Boolean. Populate the children attribute of each node, returning a structured dataset.

Returning all serialized state:

fetch.length
=> 242

Grabbing detailed information for a specific node:

fetch from: 'tb01-05', depth: 0
=> [{"identifier"=>"tb01-05",
  "type"=>"host",
  "parent"=>"_",
  "description"=>"tb01-05",
  "tags"=>["vfx_farm", "farm"],
  "config"=>{},
  "status"=>"unknown",
  "properties"=>{},
  "ack"=>nil,
  "last_contacted"=>"1970-01-01T00:00:00+00:00",
  "status_changed"=>"1970-01-01T00:00:00+00:00",
  "errors"=>{},
  "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
  "quieted_reasons"=>{},
  "children"=>{},
  "addresses"=>["10.6.0.6"]}]

Starting at a specific node, return a structured dataset that includes children:

fetch from: 'tb01-05', tree: true
=> [{"identifier"=>"tb01-05",
  "type"=>"host",
  "parent"=>"_",
  "description"=>"tb01-05",
  "tags"=>["farm"],
  "config"=>{},
  "status"=>"unknown",
  "properties"=>{},
  "ack"=>nil,
  "last_contacted"=>"1970-01-01T00:00:00+00:00",
  "status_changed"=>"1970-01-01T00:00:00+00:00",
  "errors"=>{},
  "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
  "quieted_reasons"=>{},
  "children"=>
   {"tb01-05-memcache"=>
     {"identifier"=>"tb01-05-memcache",
      "type"=>"service",
      "parent"=>"tb01-05",
      "description"=>nil,
      "tags"=>[],
      "config"=>{},
      "status"=>"unknown",
      "properties"=>{},
      "ack"=>nil,
      "last_contacted"=>"1970-01-01T00:00:00+00:00",
      "status_changed"=>"1970-01-01T00:00:00+00:00",
      "errors"=>{},
      "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
      "quieted_reasons"=>{},
      "children"=>{},
      "addresses"=>["10.6.0.6"],
      "protocol"=>"tcp",
      "app_protocol"=>"memcache",
      "port"=>11211}},
  "addresses"=>["10.6.0.6"]}]

fetch_node

Retrieve a single node from the Manager. Returns the serialized node as a Hash, given an identifier.

Grabbing detailed information for a specific node:

fetch_node 'tb01-05'
=> {"identifier"=>"tb01-05",
  "type"=>"host",
  "parent"=>"_",
  "description"=>"tb01-05",
  "tags"=>["vfx_farm", "farm"],
  "config"=>{},
  "status"=>"unknown",
  "properties"=>{},
  "ack"=>nil,
  "last_contacted"=>"1970-01-01T00:00:00+00:00",
  "status_changed"=>"1970-01-01T00:00:00+00:00",
  "errors"=>{},
  "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
  "quieted_reasons"=>{},
  "children"=>{},
  "addresses"=>["10.6.0.6"]}

graft

Create a new node on a running Manager. Normally, you'd just add the node definition to the Manager's startup files. graft allows you to also make dynamic additions to the Manager's tree. Note: Changes to the running Manager won't survive daemon restarts, unless also reflected in startup configuration sources.

Argument Required? Description
identifier yes The new node identifier label.
type yes What kind of node? Common types are 'host', 'service', and 'resource'.
parent no Attach the node under this parent identifier. Defaults to the root node.
attributes no A Hash of operational attributes saved with the node.

Create a new host node with an IP address:

graft identifier: 'new-host', type: 'host', attributes: { addresses: ['10.4.6.20'] }
=> {"identifier"=>"new-host"}

Create a service under that node:

graft identifier: 'smtp', type: 'service', parent: 'new-host',  attributes: { description: "Email!", port: 25, protocol: 'tcp' }
=> {"identifier"=>"new-host-smtp"}

The node tree from the new host:

fetch from: 'new-host', tree: true
=> [{"identifier"=>"new-host",
  "type"=>"host",
  "parent"=>"_",
  "description"=>nil,
  "tags"=>[],
  "config"=>{},
  "status"=>"unknown",
  "properties"=>{},
  "ack"=>nil,
  "last_contacted"=>"1970-01-01T00:00:00+00:00",
  "status_changed"=>"1970-01-01T00:00:00+00:00",
  "errors"=>{},
  "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
  "quieted_reasons"=>{},
  "children"=>
   {"new-host-smtp"=>
     {"identifier"=>"new-host-smtp",
      "type"=>"service",
      "parent"=>"new-host",
      "description"=>"Email!",
      "tags"=>[],
      "config"=>{},
      "status"=>"unknown",
      "properties"=>{},
      "ack"=>nil,
      "last_contacted"=>"1970-01-01T00:00:00+00:00",
      "status_changed"=>"1970-01-01T00:00:00+00:00",
      "errors"=>{},
      "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
      "quieted_reasons"=>{},
      "children"=>{},
      "addresses"=>["10.4.6.20"],
      "protocol"=>"tcp",
      "app_protocol"=>"smtp",
      "port"=>25}},
  "addresses"=>["10.4.6.20"]}]

modify

Alter operational attributes for a given node. Note: Changes to the running Manager won't survive daemon restarts, unless also reflected in startup configuration sources.

Argument Required? Description
identifier yes The node identifier label.
attributes yes A Hash of operational attributes to modify.

Alter the tags and IP address of a host node:

modify identifier: 'hotsoup', attributes: { tags: 'workstation', addresses: '10.3.0.4' }
=> true

The updated attributes:

search criteria: { identifier: 'hotsoup' }, options: { properties: ['tags', 'addresses'] }
{"hotsoup"=>{"tags"=>["workstation"], "addresses"=>["10.3.0.4"]}}

Specified attributes are replaced when using modify. If you'd prefer to append, fetch the values first and merge the result. Unknown attributes are silently discarded. Support for which attributes are modifiable depend on the node type.

prune

Remove a node and all subordinates. Note: Changes to the running Manager won't survive daemon restarts, unless also reflected in startup configuration sources.

Argument Required? Description
identifier yes The node identifier label to be removed.

This host node has subordinates:

fetch( from: 'hotsoup' ).length
=> 5

Bye bye. Prune returns the removed node.

prune identifier: 'hotsoup'
=> {"identifier"=>"hotsoup",
 "type"=>"host",
 "parent"=>"_",
 "description"=>"Mahlon's workstation",
 "tags"=>["workstation"],
 "config"=>{},
 "status"=>"unknown",
 "properties"=>{},
 "ack"=>nil,
 "last_contacted"=>"1970-01-01T00:00:00+00:00",
 "status_changed"=>"1970-01-01T00:00:00+00:00",
 "errors"=>{},
 "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
 "quieted_reasons"=>{},
 "children"=>{},
 "addresses"=>["10.3.0.75"]}

Yep, it is gone:

fetch from: 'hotsoup'
RuntimeError: Arborist manager said: No such node hotsoup.

search

Return serialized node properties that match the provided criteria. User defined properties and operational attributes are merged into a single namespace.

Argument Required? Description
criteria no A Hash of attributes and properties to match on.
options no A Hash of behavior modifications for the search. Unknown options are ignored.

As a convenience, search treats a naked Hash argument as criteria, also, so the following are equivalent:


search type: 'host'
search criteria: { type: host }

Options:

Option Description
exclude_down Boolean. By default, all nodes are returned. If set, this option returns only nodes considered to be in a 'reachable' state (not in a status of down, disabled, or quieted.)
properties String, or an Array of Strings. Only return the listed properties, instead of the entire node object.
exclude A separate criteria Hash, that removes matching nodes from the result set.

By default, search returns a Hash of nodes, keyed on node identifier:

search
=> {"_"=>
  {"type"=>"root",
   "status"=>"up",
   "tags"=>[],
   "parent"=>nil,
   "description"=>"The root node.",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"1970-01-01T00:00:00+00:00",
   "last_contacted"=>"1970-01-01T00:00:00+00:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{}},
 "hotsoup"=>
  {"rtt"=>0.17,
   "_monitor_key"=>"ping",
   "type"=>"host",
   "status"=>"up",
   "tags"=>["workstation"],
   "parent"=>"_",
   "description"=>"Mahlon's workstation",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"2017-09-16T21:46:32+00:00",
   "last_contacted"=>"2017-09-16T22:49:21+00:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{},
   "addresses"=>["10.3.0.75"]},
 "hotsoup-disk"=>
  {"type"=>"resource",
   "status"=>"unknown",
   "tags"=>[],
   "parent"=>"hotsoup",
   "description"=>"Disk space",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"1970-01-01T00:00:00+00:00",
   "last_contacted"=>"1970-01-01T00:00:00+00:00",
   "ack"=>nil,
   "errors"=>{},
   "quieted_reasons"=>{},
   "config"=>{},
   "addresses"=>["10.3.0.75"]},
[..... omitted the rest for brevity]

You can omit non-reachable nodes with the exclude_down option:


search( status: 'down' ).length
=> 4
search( criteria: { status: 'down' }, options: { exclude_down: true }).length
=> 0

Example: Search for all acknowledged nodes.

search status: 'acked', type: 'host'
=> {"omegaman"=>
  {"_monitor_key"=>"ping",
   "type"=>"host",
   "status"=>"acked",
   "tags"=>["farm"],
   "parent"=>"_",
   "description"=>"omegaman",
   "dependencies"=>{"behavior"=>"all", "identifiers"=>[], "subdeps"=>[]},
   "status_changed"=>"2017-09-16T23:00:32+00:00",
   "last_contacted"=>"2017-09-16T23:00:47+00:00",
   "ack"=>{"message"=>"We know about this.", "sender"=>"Mahlon", "via"=>nil, "time"=>"2017-09-16T23:00:32+00:00"},
   "errors"=>{"ping"=>"is unreachable"},
   "quieted_reasons"=>{},
   "config"=>{},
   "addresses"=>["10.3.1.32"]}}

Example: Find all Host identifiers that have a higher than 0.6ms ICMP round trip time.

search( criteria: { type: 'host' }, options: { properties: 'rtt' } ).
    each_with_object( [] ) do |(node, props), acc|
        acc << node if props['rtt'] > 0.6
    end # => ["tb01-12", "tb02-08", "tb02-20", "tb04-09", "tb04-11", "tb04-18", "tb06-18", "tb06-19", "tb07-07"]

status

Return operational information about the current Arborist manager.

status
=> {"server_version"=>"0.1.0", "state"=>"running", "uptime"=>2256.735517215, "nodecount"=>242}

subscribe

This is a manual method to generate new subscriptions to a node. Subscriptions are triggered on node events, and sent to the Event API socket for consumption via ZMQ. See the Event API page for specifics on how to utilize this.

In practice, you'll more likely be using the Observer DSL to express subscription requirements.

Argument Required? Description
identifier no The node to receive events from. Default is the root node, which catches all events. This is most useful if your subscriber is only interested in a specific section of the Arborist tree.
criteria no A Hash of attributes and properties to match on.
event_type no A valid node event. Otherwise, matches on all events.
exclude no A separate criteria Hash, that removes matching nodes from the result set.

Subscribe to all events across Arborist (aka, FIRE HOSE mode):

subscribe
d9497710-5967-43d7-9bff-dd12970c0447

The UUIDs that are returned are the ZMQ subscription identifiers. Connect to the Event API with this identifier to receive matching events.

Subscribe to host nodes that transition to a down state, but only under the westside-gateway router:

subscribe criteria: { type: 'host' }, event_type: 'node.down', identifier: 'westside-gateway'
=> "0f8b63ca-80d0-44ee-8917-897d7b029245"

unsubscribe

Remove a prior subscription.

Argument Required? Description
subscription_id yes The UUID of the subscription.

A successful unsubscribe returns the attributes of the subscription.

unsubscribe '0f8b63ca-80d0-44ee-8917-897d7b029245'
=> {"event_type"=>"node.down", "criteria"=>{"type"=>"host"}}

An unknown subscription ID just returns nil.

unsubscribe '0f8b63ca-80d0-44ee-8917-897d7b029245'
=> nil

update

Update takes only one argument - a Hash keyed on identifiers. The values are user properties to replace. With many identifiers, you can perform batch updates.

User properties are part of the saved state file. If configured, these will be retained across Manager restarts.

Argument Required? Description
properties yes A hash of user properties.

Monitors use the properties hash to store arbitrary data for future display and use, such as the RTT example above for search.

Setting a property to nil deletes it.

A special property called error is used by monitors to affect the state of the node, alongside an internal _monitor_key property.

Store some extra information into a node:

update hotsoup: { things: [ :one, :two, :three ] }
=> true

Pull the data back out:

search criteria: { identifier: 'hotsoup' }, options: { properties: 'things' }
=> {"hotsoup"=>{"things"=>["one", "two", "three"]}}

Example: Manually clear a specific monitor's alert status (you shouldn't need do this in practice.)

downed_nodes = search( type: 'host', status: 'down' ).keys
data = downed_nodes.each_with_object( {} ) do |node, hash|
    hash[node] = { error: nil, _monitor_key: 'ping' }
end

update( data )