Posts
-
Netdata, Telegraf, and Prometheus Exporters
For a while now I’ve been using Netdata as my default option when collecting metrics, but I’ve heard a bit about other options and got curious. I couldn’t find a direct comparison so I decided to do a quick one myself. My primary candidates are Netdata, Prometheus Exporters, and Telegraf. My approach is essentially to install each on a separate EC2 instance on Amazon Web Services (AWS), scrape the metrics using Prometheus, and do some exploration in Grafana.
-
Timeseries and Timeseries Again (pt. 2)
One new development I have been pretty excited about is the addition of Loki to our monitoring stack. Developed by the same individuals who develop for Grafana, Loki is a simple log aggregation system designed to be horizontally scalable, cost-effective, and easy to operate. It styles itself as “like Prometheus, but for logs”, and that’s really all I’ve wanted for the past year - that you can switch back and forth between metrics and logs within the same interface is just a bonus. Loki relies on promtail to tag and send logs from hosts you care about. Grafana can then use the Loki server as a datasource to present logs to the user, who can narrow down a subset of data based on labels (I decided on hostname, ip address, and filename as a starting set of labels) and perform simple greps on the relevant collection of log events. Loki is still in an alpha stage and as such, there is not a ton of documentation out there have successfully deployed Loki and Promtail instances that are good enough as a minimum viable product. Here is an updated diagram of the monitoring architecture we’re working with:
-
Timeseries and Timeseries again
This will be a high-level overview of the metrics stack that I use and maintain at work. Due to the complexity of the tools I won’t go into too much detail and will omit some pieces of the architecture that aren’t relevant to the topic.
-
An Introduction
I mainly intend to use this space to braindump challenges I face and things I learn in my field. It is mainly for my own benefit, to reinforce things in my own mind, but maybe it will be of some use to someone else.