Introduction

I will be making a note series of what I have learned today regarding tracking statistics using a certain set of tools which form the core of the Monitoring Stack these days in a lot of companies.

I am going to discuss very briefly here about the tools as I have already read a lot about them and just making a revision notes for myself. For more in-depth details of the tools go here (Source - Digital Ocean)

Before I summarize the tools let us understand why are we monitoring our servers. For small scale application this does not matter much but when an application scales tracking becomes necessary. Reason being -

• to find out the timely changes in the system.
• to monitor all the resources at the same time.
• to make decision regarding resources.
• change in the code would help us compare the the previous monitoring stats.
• debugging.

Let us now discuss the tools we will be using to monitor the system. For a monitoring system we need three things -

1. A GUI, a dashboard, a web panel which help us visualize the data.
2. A database which stores the monitor data.
3. A receiver to input the data to the database.
4. A collector which keep tracks of all the resources and send the changes to be stored.

Graphite is one such tool. It consist of three components -

• Graphite Web (1)
• Whisper (2)
• Carbon (3)

Little more detail on this here -

Graphite Web

Graphite is an excellent tool for organizing and rendering visual representations of data gathered from your system. (Source- )

A Django project that provides a GUI for graphite. This is what you see when you visit a Graphite install in your browser. It also has an admin area that lets you set up users that are allowed to create and save graphs. You can deploy this like any other Django app via wsgi. (Definition Source )

Github - https://github.com/graphite-project/graphite-web

Whisper

Whisper is a time-series database file format. It doesn't actually run as a server or anything, it's just a standard format for storing time-series data. These databases are a fixed size, and once the data gets old it falls off the other end. Whisper is a replacement for RRDTool, which I will talk about later.

Carbon

Carbon is a daemon that runs constantly in the background. Its main job is to take in statistics from the network and write them to disk.

It consists of multiple daemon each of one can be used to forward data to whisper. These daemon can be -

• carbon-cache.py
• carbon-relay.py
• carbon-aggregator.py

For most basic configurations, a single carbon-cache.py instance is sufficient to handle the data reception.

Multiple instances can be run at once as your setup grows. These can be balanced by a carbon-relay.py or carbon-aggregator.py daemon in front.

This diagram would help you visualize the Graphite Architecture.

Graphite Architecture Diagram

Stats Gathering and Delivery

Now after we are done understanding the Graphite Architecture let us understand the how will graphite will store system, cpu, networking stats. It is done by a tool known cas CollectD

CollectD

Collectd can gather statistics about many different components of a server environment. It allows you to easily track common metrics like memory usage, CPU load, network traffic etc. This allows you to easily correlate events with the state of your systems.

Let us now move on to the installation part.