published on
tags: cassandra prometheus docker devops

Monitoring Dockerised Cassandra with Prometheus

As part of the move away from a proprietary in memory database madb, last.fm is moving to Apache Cassandra. Cassandra is far more easier to deploy to a public cloud (in our case GCE), scale up/down and query. madb, being completely in memory does have pretty good performance characteristics, but scaling requires machines with more ram, sharding by user and in has some weird and interesting bugs.

We deploy a dockerised version of Cassandra, but the standard image doesn’t support monitoring via Prometheus. It’s easy to add however - all you need is a JMX java agent, and one for Prometheus already exists. Initially our Dockerfile looked something like this:

FROM cassandra

ADD "http://central.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.10/jmx_prometheus_javaagent-0.10.jar" /usr/local/lib/jmx_prometheus_javaagent.jar
ADD prometheus.yaml /usr/local/etc/prometheus.yaml

RUN chmod a+r /usr/local/lib/jmx_prometheus_javaagent.jar
ENV JVM_OPTS "$JVM_OPTS -javaagent:/usr/local/lib/jmx_prometheus_javaagent.jar=61621:/prometheus/cassandra.yml "

However trying to run this resulted in a nice stack trace rather than a running Cassandra instance. The root of this stack trace was..


java.net.BindException: Address already in use

I first wrote our Dockerfile to use ENV JVM_OPTS, however it seems the problem is that if the jmx_prometheus_javaagent jar is defined in JVM_OPTS for the whole container it gets started before Cassandra. So the solution was a small rewrite, adding the JVM_OPTS line to cassandra-env.sh instead. Our Dockerfile thus became something like this:

FROM cassandra

ADD "http://central.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.10/jmx_prometheus_javaagent-0.10.jar" /usr/local/lib/jmx_prometheus_javaagent.jar
ADD prometheus.yaml /usr/local/etc/prometheus.yaml

RUN chmod a+r /usr/local/lib/jmx_prometheus_javaagent.jar
RUN echo 'JVM_OPTS="$JVM_OPTS -javaagent:/usr/local/lib/jmx_prometheus_javaagent.jar=61621:/usr/local/etc/prometheus.yaml"' >> /etc/cassandra/cassandra-env.sh