while(1) { fork; }

Collecting Metrics

Our adventure in metrics gathering is a bit long-winded for a story. I've divided it into sections so you can jump to the details you might want to see: Software, Glue, Hardware, Details, Headaches, Numbers, and Final Thoughts.

Please drop comments noting your performance numbers or to ask questions. We're happy to share what we can.

Franck wrote a blog post about our usage of this system here. I'm going to cover the deployment story here.

The Software

We started off with a single node, our new stats collection node, running collectd with the write_graphite plugin sending data to a carbon cache via loopback. We used the debian graphite-carbon package, and manually set up a graphite-web installation. The first data was localhost only metrics, and this kept us entertained for a few minutes.

Host Metrics

I was afraid to install Collectd on all hosts and just walk away. So to quickly get host metrics I found the ganglia plugin for Collectd, which allows it to receive metrics from hosts running gmond and output them to Graphite. Using this plugin wasn't the easiest thing in the world, but I'll detail that seperately.

We moved on to Collectd installations on all of our hosts after a week or so of playing with Graphite and other bits. By default Collectd is much more discrete in its measurements of the system than Ganglia. For some examples: Ganglia has cpu aggregate metrics, but Collectd has per-cpu metrics, Ganglia has ethernet metrics, but Collectd has per-interface and per-protocol metrics.

Our Collectd setup is using the Collectd 'network' [sic] plugin for transmission of metrics to the central stats host, which has a Collectd instance running to write them to graphite-carbon. This is a simplification of our setup and more details are below.

Application Metrics

At roughly the same time we started toying with Statsd, which handles aggregation of various data types. The first use of Statsd was web hit stats from Typepad. This included counts, byte sizes, and rendering time sliced by request method and response code.

One terminology change we made from the start was to call the 'ms' data type in statsd an 'aggregate'. It comes from being measurement of time in statsd, but the aggregation functions are useful for more data types. Someday a release of statsd will include this change as well.

We've moved on to collecting event counts, body sizes, execution times, and many other things that I can't even think of right now. Almost all of our products both internal and external have at least one metric being collected and analyzed right now, and the majority of these are handled through statsd for simplicity.

The Glue

We added simple proxies at two layers in our stack for multiple reasons. Franck has shared these via our devops-tools repo.

First, there is a proxy running on the stats host between collectd and carbon. The purpose of this proxy is to adjust the keys of metrics to a more organized tree for us. 

Second, there is a proxy running on every host. This proxy receives lines destined for statsd via loopback and does a few things. The base of the key is generated and automatically prepended to all metrics pushed through this proxy, which keeps that format in a common place. Counters and aggregates are multiplexed to both a host and an app level prefix, which allows us to analyze the metric per host or for the whole cluster easily and accurately (averaging averages is inaccurate). Gauges are not multiplexed due to their nature.

Examples of the keys are in the details section below.

The Hardware

Our stats machine is a basic server using no virtualization, 8 cores, 24GB ram I think. This is going to get upgraded, but not because of CPU or RAM saturation.

For storage, initially we had a 15k RPM SAS drive as our backing store for carbon, but this was quickly outgrown and replaced with an SSD. We ran out of space on the filesystem and started a RAID-0 of SSD for storage, which is still in use today.

The Details

Metric Tree Layout

Our graphite metric tree is layed out as a planned heirarchy to keep application and host metrics of related hosts and applications closer together in the tree, but still sorted to keep the number of nodes in one folder managable.

Some key examples (pun intended):

DEV.hosts.hachi.hachi-fork.interface.if_octets-eth0.rx
PROD.hosts.tp-web.oak-tp-web001.df.df-tmp.free
PROD.app.aggregate.tp-web.status.200.time.mean
PROD.hosts.tp-web.oak-tp-web001.

The first two are examples of collectd metric keys, and are formatted by the carbon proxy in the Glue section above.

The last two are collected by statsd and are formatted by both the proxies mentioned in the Glue section above.

Scaling

We ran out of IO bandwidth first, very early on. Switching to an SSD was the fix for that.

Then we ran out of disk space near 70k metrics. Creating a RAID-0 of multiple SSD was our fix.

Finally, we ran out of CPU time above 200k metrics. We now run multiple copies of carbon-cache, load balanced by carbon-relay, which are in turn load balanced by the carbon proxy.

We've tested up to 300k metrics and our IO bandwidth starts to top out. In this scenario the system continues to work, but starts consuming memory to handle it. This is noted as a headache below.

The Headaches

Collectd's Ganglia Plugin

First off, the debian version of collectd has a ganglia plugin included that isn’t linked against libganglia properly. After a short period of knocking around I decided to force the dynamic linker to preload libganglia, which let me get this running quickly. This was done by adding the following to /etc/default/collectd:

LD_PRELOAD=/usr/lib/libganglia-3.3.8.so.0.0.0
export LD_PRELOAD # kick it harder till it starts up :D

Then I discovered that ganglia’s use of multicast means that in an unprepared network I can’t subscribe to multicast groups across routers without some serious config.

Long story short, I enabled IGMP and multicast routing on all our network gear for a few weeks while the ganglia shim was in use. A side effect was that our Mac Mini desktop in the datacenter was storming the network because of some broken multicast bonjour thing. This has been fixed in newer versions of OSX apparently, but I solved that problem by turning off the Mac.

The Case of the Disappearing Metrics

This was already chronicled by Abe Hassan over here, and the Franck talks about Carbon's manhole, a debugging interface for carbon.

Memory Consumption

When you run out of IO bandwidth with carbon-cache, the daemon will start using an in-memory cache to help keep things going. I believe the intended scenario would have IO pressure holding writes back, which carbon would then try to sort so it can make fewer seeks happen by optimizing in user space.

Sadly, this wasn't working for us. The cache max size was probably set too high, which we adjusted down. This caused carbon to use its flow control feature to slow down the writes from collectd, which then started consuming memory instead.

At this point, but without deep testing. I belive the flow control option can become a dangerous thing in scenarios where you might prefer to drop metrics rather than exploding. We still have it turned on, but we haven't hit our limit again.

The Numbers

Our single stats machine is handling about 250k carbon metrics. The vast majority of which are updated every 10 seconds, which is our flush interval for both collectd and statsd.

The stats host is averaging about 15k IOPS/sec on our SSD RAID, with occasional spikes up to 25k and beyond.

Final Thoughts

The combined technology of Collectd, Statsd, and Graphite all come together in our environment to create a very nice toolset. We were able to tweak the way these systems interact by writing some simple data proxies. The fact that statsd and carbon both speak line protocol made this very simple.

Planning out the metric tree has helped to keep things organized, and queries smaller/simpler than they might otherwise be. We lessened the problem of adjusting the metric tree layout by hiding some of the path layout in common proxies, keeping it out of application code.

It took a little time to get our system to perform up to a level we were comfortable with, but it has been rock solid once we got there. Memory consumption on the stats host has been pretty much constant for weeks now.

Our proxies, apache log scraper, and a few other tools are already up in the saymedia/devops-tools.git repo. Go take a peek, we'll share more soon.

Next I'll cover tell a shorter story about the Riemann network monitoring system.

The tools that we choose

At my work, Say Media, we have a group shepherding tasks and plans in the 'devops' style. Franck and myself are this group, with input and assistance from others.

As a starting point we’re assuming that tools must already exist, and that we should only rebuild them from scratch if the barrier to entry is too high. Reuse, contribute, and participate in the Open Source landscape. That said, we will build tools to glue things together. My intention in this and the coming posts is to share our research, decisions, and rationale for tools we are looking at.

We started off this year by rolling out a replacement for system and application metric collection. We didn't perform any sort of experiment to test systems and see what worked the best. Instead we dove right in and started replacing Ganglia with a more discrete system comprised of Collectd, Carbon, Whisper, and Graphite (the latter three which are often considered a single item.) Statsd got added to collect counters, and much of this setup has been tweaked, tuned, and twisted to do what we expect. Finally we looked at riemann for analysis and triggering alerts.

Starting now I’ll be writing posts to document our  Collectd, Graphite, Statsd, and Riemann experience. There will certainly be more, but stay tuned and ask me to share more if you want.

Stop typing passwords: Force10 / ftos

On TV At work I sometimes play the role of a network engineer. I find the physical layer on down to IP to be quite interesting  and fun. That said, I absolutely hate typing my password. I have a core belief that anytime I have to type my password I'm taking time from the life of my wrists.

A fairly typical network gets a RADIUS server to make authentication management easier, but this is for username and password credentials. I want SSH key auth.

To accomplish this on Force10 devices running FTOS is simple enough if you read the docs, but I'm just going to lay it down here for shortest-path engineering.

As a config user we'll need to:

• enable ssh
• enable rsa auth for ssh
• create a user for yourself (having one supplied by rancid isn't enough to store keys on the device)

#conf t
(conf)#ip ssh server enable
(conf)#ip ssh rsa-authentication enable
(conf)#username hachi
(conf)#end

Then you will need to sign in as yourself (if you're not already) and:

• copy the keys onto the device
• load the keys into your account

#copy tftp://192.0.2.0/hachi.txt flash://hachi.txt
!!
400 bytes successfully copied
#ip ssh rsa-authentication my-authorized-keys flash://hachi.txt         
RSA keys added to user's list of authorized-keys.
Delete the file flash://hachi.txt : (yes/no) ? yes

These do not conflict with radius setup in my environment, so this is incredibly useful to me. Sign in with a key if I have one, sign in with a password if I don't. Yay.

Additional stuff

Your username line can include a password if you want/need one for local auth, and can include privilege levels (level 15 is the highest and automatically enables me at login)

If you want to clear your key set you use the 'no' form of the set command:

#no ip ssh rsa-authentication my-authorized-keys

Key based sudo auth

I am perpetually tired of typing in a password to authorize my actions. Though that's enough in my mind to warrant a search for a solution, there is also the world of auth system conflict.

Companies are perpetually in search of a "unified auth system." Or at least this is common enough that your email password may also be your sudo password.

Policy or paranoia often makes passwords be complex or otherwise awkward to type (or even remember)

Anyways, my complaints aside I'm out to find a solution... first idea I've come up with so far is to implement a PAM auth handler that uses your SSH_AUTH_SOCK to run a challenge-response auth check. Assuming the ssh auth sock system is secure I'd like to think I can trust this idea more than a password field.

GNU Parted Units

I bought some 2TB hard drives the other evening and threw them into my household server. While partitioning them up to be raided I kept running into an issue with parted where it would throw a warning:

Start? 2048KB
End? -1
Warning: The resulting partition is not properly aligned for best performance.

The fast answer here, include an 'i' in your units, otherwise parted assumes you're speaking decimal units. This will cause the alignment warning to go away.

Start? 2048KiB
End? -1

The interesting bit is hiding in libparted/unit.c, where the units are parsed. It seems that parted assumes you are speaking decimal units (kilobyte, megabyte, gigabyte, terabyte) unless you include the 'i' to indicate you mean base2 units (kibibyte, mibibyte, gibibyte, tibibyte).

This seems a little backwards since this is a program that deals directly with alignment, but whatever.

static PedUnit
parse_unit_suffix (const char* suffix, PedUnit suggested_unit)
{
        if (strlen (suffix) > 1 && tolower (suffix[1]) == 'i') {
                switch (tolower (suffix[0])) {
                        case 'k': return PED_UNIT_KIBIBYTE;
                        case 'm': return PED_UNIT_MEBIBYTE;
                        case 'g': return PED_UNIT_GIBIBYTE;
                        case 't': return PED_UNIT_TEBIBYTE;
                }
        } else if (strlen (suffix) > 0) {
                switch (tolower (suffix[0])) {
                        case 's': return PED_UNIT_SECTOR;
                        case 'b': return PED_UNIT_BYTE;
                        case 'k': return PED_UNIT_KILOBYTE;
                        case 'm': return PED_UNIT_MEGABYTE;
                        case 'g': return PED_UNIT_GIGABYTE;
                        case 't': return PED_UNIT_TERABYTE;
                        case 'c': return PED_UNIT_CYLINDER;
                        case '%': return PED_UNIT_PERCENT;
                }
        }