Monitoring our Cluster

It took quite a lot of work to setup our Raspberry Pi Cluster properly, would be a shame if we overlooked any signs of malfunctioning right ?
During this article I' ll show how we setup an application stack to monitor our brand new environment with help of well known open source tools : Prometheus, Alert manager, Grafana.
A brief introduction to Prometheus, our metric collector of choice. It's task is collecting information from whichever target available, into a centralized monitoring platform. Targets vary from machines to processes, jobs or even services such as AWS Cloudwatch,
To accomplish these tasks, Prometheus makes use of Exporters, basically agents that extracts requested data from Targets, which is later pulled by our Prometheus server.
Extracted data can be then served from its simple WebUI for us to consult.
We will use docker-compose to bring our Prometheus to life.
here' s the content of our compose :

version: '3'
volumes:
  prometheus-data:
    driver: local
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - /etc/prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--web.enable-lifecycle"

the second argument passed passed to "--web.enable-lifecycle" is necessary to being able to reload the configuration on the fly, starting from Prometheus 2.0. We are not ready to start this container yet. Notice a new configuration is needed before starting our container, we place it in the specified path /etc/prometheus/prometheus.yml with the following content

- job name: prometheus
  scrape_interval: 5s
  static configs:
   - targets: ['localhost:9090']

Now we bring it up.

docker-compose up -d

We have our metrics accessible at http://localhost:9090 . Wouldn't be amazing if there were a way to make sense of such information at first glance ? That's where Grafana comes in handy, since we can use Prometheus metrics as data source, to create beautiful, and intuitive dashboards we can use to monitor our infrastructure at large, visually. Grafana 's capabilities extend beyond the scope of this article, I invite you to read the official documentation to further dwell into it's workings. First off, we need to install Grafana in our machine : Log in to Grafana : http://localhost:3000 Now is time to add Prometheus as data sources Add Grafana to Prometheus configuration file :

- job_name: grafana
  static_configs:
   - targets: ['localhost:3000']

https://github.com/prometheus/node_exporter

Monitoring our Cluster

Comments

Subscribe to our Newsletter