Spyglass on Streams

The Flow of Metrics via Streams

The collectd service collects node-level and service-level metrics from each node in the cluster. The collectd service hashes metrics to a stream and writes the metrics into topics in that stream.

Collectd creates two streams per cluster, Stream (0) and Stream (1). Each stream contains approximately 100+ topics. Topic names use the format <hostname>_<metricname>. For example: mfs81.qa.lab_cpu.percent.

Collectd maps the metric names to Stream (0) or Stream (1) using the djb2 hash algorithm, as shown:

static int hash(const char *str, int range)
{ int hash = 5381; int c; while ((c = *str++) != 0) hash = ((hash << 5) + hash) + c; /* hash * 33 + c */ return abs(hash%range); }

The algorithm hashes the metric name to an integer that is either 0 or 1.

Spyglass on Streams Performance

The performance of Streams for metrics depends on the size of the cluster, the number of OpenTSDB nodes available to consume metrics data, and the number of streams and partitions available to move the metrics data from the collectd service to OpenTSDB.

Default collectd and OpenTSDB settings work well for clusters with up to 100 nodes and three OpenTSDB nodes. If the number of nodes increases or you notice that performance is sluggish, you can improve performance by adding OpenTSDB nodes and modifying the number of streams.

Evaluating Streams Performance

You can use the stream cursor list and stream topic info maprcli commands to view the producer (collectd) and consumer (OpenTSDB) statistics. Check the statistics to see if there is an increase in lag time between producers and consumers. If you notice an increase in lag time, increase the number of consumers (OpenTSDB nodes) or modify the streams and partition settings, as explained in the following sections.

Determining How Many OpenTSDB Nodes to Install

Having multiple OpenTSDB nodes in the cluster distributes the workload. The number of partitions and OpenTSDB nodes determines the level of parallelism for consumption.

Each OpenTSDB node can consume one partition at a time. By default, metrics data is divided across five partitions in each topic and optimal parallelism is reached if there are five OpenTSDB nodes to consume the partitions. See Parallelism When Consuming Messages. Note that the term “consumer” in the topic equates to an OpenTSDB node in Spyglass on Streams.

If your cluster has 10 or more nodes, at least three OpenTSDB nodes should be available to consume metrics. Typically, for every 10x increase in nodes, you should add another OpenTSDB node. For example, if your cluster reaches a size of 100 nodes, have four OpenTSDB nodes available for consumption. Note that an increase in the number of OpenTSDB nodes may require an increase in the number of streams and/or partitions.

"fs.mapr.async.worker.threads=<value>"

Increasing the Number of Streams

The default setting for the number of streams is two. As a general guideline, for every 10x increase in the number of cluster nodes, add two additional streams. For example, if your cluster has 100 nodes, add two more streams, for a total of four.

To increase the number of streams, edit the streams-specific options in the collectd and OpenTSDB configuration files. The streams option in each file must have the same value. After you change streams parameters, reconfigure MapR Monitoring, as shown in Update the Monitoring Storage Nodes.

Changing the Automatic Stream Cursor Commits

You can adjust the frequency of automatic stream cursor commits for OpenTSDB. Modify the tsd.streams.autocommit.interval in opentsdb.conf The unit is thousands of seconds. The default value is '60000' which is 60 secs. For a system with heavy loads, consider changing the value to something like 5 minutes.