Docker Container Monitoring

Introduction

Docker: Docker is an open-source project that automates the deployment of applications inside software containers. 

Container: Using containers, everything required to make a piece of software run is packaged into isolated containers. Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work are needed. This makes for efficient, lightweight, self-contained systems and guarantees that software will always run the same, regardless of where it’s deployed.

Docker-Compose: Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application's services. Then, using a single command, you create and start all the services from your configuration.

cAdvisor (Container Advisor) : It provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects aggregates, processes, and exports information about running containers.

https://hub.docker.com/r/google/cadvisor/ 

https://github.com/google/cadvisor

Prometheus: Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.  It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets.

https://prometheus.io/

Prometheus Monitoring stack : 

Prometheus-core : Time series database to store metrics

Node-Exporter : Exporter of node metrics

PromDash : Prometheus dashboard

AlertManager : 

Docker containers for prometheus stack: https://hub.docker.com/u/prom/

Grafana: Grafana is the ‘face’ of Prometheus. While Prometheus exposes some of its internals like settings and the stats it gathers via basic web front-ends, it delegates the heavy lifting of proper graphical displays and dashboards to Grafana.

Alertmanager: Alertmanager manages the routing of alerts which Prometheus raises to various different channels like email, pagers, slack - and so on. So while Prometheus collects stats and raises alerts it is completely agnostic of where these alerts should be displayed. This is where the alertmanager picks up.

Requirements:

In order to follow along, you will need only two things

docker

docker-compose

Follow the links for installation instructions to install docker and docker-compose. 

Getting Started:

Launching Prometheus:
We will use  docker-compose.yml  for installing prometheus

# docker-compose.yml
version: '2'
services:
prometheus:
    image: prom/prometheus:0.18.0
    volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
        - '-config.file=/etc/prometheus/prometheus.yml'
    ports:
        - '9090:9090'

and a prometheus configuration file prometheus.yml:

# prometheus.yml
global:
    scrape_interval: 5s
    external_labels:
        monitor: 'my-monitor'
scrape_configs:
    - job_name: 'prometheus'
      target_groups:
          - targets: ['localhost:9090']

As you can see, inside docker-compose.yml we map the prometheus config file into the container as a volume and add a -config.file parameter to the command pointing to this file.

To launch prometheus, run the command

docker-compose up

Visit http://localhost:9090/status to confirm the server is running and the configuration is the one we provided.

Targets

Further down below the ‘Configuration’ on the status page you will find a section ‘Targets’ which lists a ‘prometheus’ endpoint. This corresponds to the scrape_configs setting by the same job_name and is a source of metrics provided by Prometheus. In other words, the Prometheus server comes with a metrics endpoint - or exporter, as we called it above - which reports stats for the Prometheus server itself.

The raw metrics can be inspected by visiting http://localhost:9090/metrics.

Adding a node-exporter target

While it’s certainly a good idea to monitor the monitoring service itself, this is just going to be an additional aspect of the set-up. The main point is to monitor other things by adding targets to the scrape_configs section in prometheus.yml . As described above, these targets need to export metric in the prometheus format.

One such exporter is node-exporter, another piece of the puzzle provided as part of Prometheus. What it does is collect system metrics like cpu/memory/storage usage and then it exports it for Prometheus to scrape. The beauty of this is that it can be run as a docker container while also reporting stats for the host system. It is therefore very easy to instrument any system that can run docker containers.

We will add a configuration setting to our existing docker-compose.yml to bring up node-exporteralongside prometheus . However, this is mainly for convenience in this example as in a normal setup where one prometheus instance is monitoring many other machines these other exporters would likely be launched by other means.

Here’s what our new docker-compose.yml looks like:

# docker-compose.yml
version: '2'
services:
    prometheus:
        image: prom/prometheus:0.18.0
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
        command:
            - '-config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'
    node-exporter:
        image: prom/node-exporter:0.12.0rc1
        ports:
            - '9100:9100'

We simply added a node-exporter section. Configuring it as a target only requires a small extension to prometheus.yml :

# prometheus.yml
global:
    scrape_interval: 5s
    external_labels:
        monitor: 'my-monitor'
scrape_configs:
    - job_name: 'prometheus'
      target_groups:
          - targets: ['localhost:9090']
    - job_name: 'node-exporter'
      target_groups:
          - targets: ['node-exporter:9100']

Grafana :

grafana:
        image: grafana/grafana:3.0.0-beta7
        environment:
            - GF_SECURITY_ADMIN_PASSWORD=pass
        depends_on:
            - prometheus
        ports:
            - "3000:3000"



The complete final version version of all config files can be found in this https://github.com/vikramshinde12/dockprom.

After restarting the service with
docker-compose up

you can access Grafana at http://localhost:3000/login

Complete Monitoring Stack installation

Components included

•  cAdvisor
• NodeExporter
• Prometheus
• AlertManager
• Grafana
• Slack

  

Install

Detailed video for the docker monitoring stack installation

Clone dockprom repository on your Docker host, cd into dockprom directory and run compose up:

• $ git clone https://github.com/stefanprodan/dockprom
• $ cd dockprom
• $ docker-compose up -d

Containers:

• Prometheus (metrics database) http://:9090
• AlertManager (alerts management) http://:9093
• Grafana (visualize metrics) http://:3000
• NodeExporter (host metrics collector)
• cAdvisor (containers metrics collector)

While Grafana supports authentication, the Prometheus and AlertManager services have no such feature. You can remove the ports mapping from the docker-compose file and use NGINX as a reverse proxy providing basic authentication for Prometheus and AlertManager.

Setup Grafana

Navigate to http://:3000 and login with user admin password changeme. You can change the password from Grafana UI or by modifying the user.config file.

From the Grafana menu, choose Data Sources and click on Add Data Source. Use the following values to add the Prometheus container as data source:

• Name: Prometheus
• Type: Prometheus
• Url: http://prometheus:9090
• Access: proxy

Now you can import the dashboard temples from the grafana directory. From the Grafana menu, choose Dashboards and click on Import.

Following dashboards can be imported

Docker Host Dashboard

The Docker Host Dashboard shows key metrics for monitoring the resource usage of your server:

• Server uptime, CPU idle percent, number of CPU cores, available memory, swap and storage
• System load average graph, running and blocked by IO processes graph, interrupts graph
• CPU usage graph by mode (guest, idle, iowait, irq, nice, softirq, steal, system, user)
• Memory usage graph by distribution (used, free, buffers, cached)
• IO usage graph (read Bps, read Bps and IO time)
• Network usage graph by device (inbound Bps, Outbound Bps)
• Swap usage and activity graphs

Docker Containers Dashboard

The Docker Containers Dashboard shows key metrics for monitoring running containers:

• Total containers CPU load, memory and storage usage
• Running containers graph, system load graph, IO usage graph
• Container CPU usage graph
• Container memory usage graph
• Container cached memory usage graph
• Container network inbound usage graph
• Container network outbound usage graph

Note that this dashboard doesn’t show the containers that are part of the monitoring stack.

     Slack Configuration

Setup alerting

The AlertManager service is responsible for handling alerts sent by Prometheus server. AlertManager can send notifications via email, Pushover, Slack, HipChat or any other system that exposes a webhook interface. A complete list of integrations can be found here.

You can view and silence notifications by accessing http://:9093.

The notification receivers can be configured in alertmanager/config.yml file.

To receive alerts via Slack you need to make a custom integration by choose incoming web hooks in your Slack team app page. You can find more details on setting up Slack integration here.

Copy the Slack Webhook URL into the api_url field and specify a Slack channel.

route:
    receiver: 'slack'

receivers:
    - name: 'slack'
      slack_configs:
          - send_resolved: true
            text: "{{ .CommonAnnotations.description }}"
            username: 'Prometheus'
            channel: '#'
            api_url: 'https://hooks.slack.com/services/'

   Grafana Alert Configuration:

On the Notification Channels page hit the New Channel button to go the the page where you can configure and setup a new Notification Channel

You specify name and type, and type specific options. You can also test the notification to make sure it’s working and setup correctly.