Samlare: expvar to Graphite
By anders pearson 24 Sep 2017
One of my favorite bits of the Go standard library is expvar. If you’re writing services in Go you are probably already familiar with it. If not, you should fix that.
Expvar just makes it dead simple to expose variables in your program via an easily scraped HTTP and JSON endpoint. By default, it exposes some basic info about the runtime memory usage (allocations, frees, GC stats, etc) but also allows you to easily expose anything else you like in a standardized way.
This makes it easy to pull that data into a system like Prometheus (via the expvarCollector) or watch it in realtime with expvarmon.
I still use Graphite for a lot of my metrics collection and monitoring though and noticed a lack of simple tools for geting expvar metrics into Graphite. Peter Bourgon has a Get to Graphite library which is pretty nice but it requires that you add the code to your applications, which isn’t always ideal.
I just wanted a simple service that would poll a number of expvar endpoints and submit the results to Graphite.
So I made samlare to do just that.
You just run:
$ samlare -config=/path/to/config.tomlWith a straightforward TOML config file that would look something like:
CarbonHost = "graphite.example.com"
CarbonPort = 2003
CheckInterval = 60000
Timeout = 3000
[[endpoint]]
URL = "http://localhost:14001/debug/vars"
Prefix = "apps.app1"
[[endpoint]]
URL = "http://localhost:14002/debug/vars"
Prefix = "apps.app2"
FailureMetric = "apps.app2.failure"
That just tells samlare where your carbon server (the part of Graphite that accepts metrics) lives, how often to poll your endpoints (in ms), and how long to wait on them before timing out (in ms, again). Then you specify as many endpoints as you want. Each is just the URL to hit and what prefix to give the scraped metrics in Graphite. If you specify a FailureMetric, samlare will submit a 1 for that metric if polling that endpoint fails or times out.
There are more options as well for renaming metrics, ignoring metrics, etc, that are described in the README, but that’s the gist of it.
Anyone else who is using Graphite and expvar has probably cobbled something similar together for their purposes, but this has been working quite well for me, so I thought I’d share.
As a bonus, I also have a package, django-expvar that makes it easy to expose an expvar compatible endpoint in a Django app (which samlare will happily poll).