Full monitoring system: Graphite, collectd and StatsD – Part 2

Introduction

In the first part of this series we installed and configured Graphite on an Ubuntu 16.04 server.
Remember that Graphite is software for visualizing data, so it’s almost useless if there isn’t a way to collect data and pass it on. This is were collectd become a fundamental part of the full monitoring system: it is a daemon which gathers metrics and makes the information available to the system admin.

In this tutorial we will explore the steps for installing and configuring collectd to pass data to Graphite.

Collectd Installation

collectd is available in repositories, so you can install it with apt:

# apt install collectd collectd-utils

That’s it! Now, let’s configure.

Configuring collectd

Edit collectd configuration file:

# $EDITOR /etc/collectd/collectd.conf

In the first lines there is the hostname section. Here, the default file looks like this:

#Hostname "localhost" 
FQDNLookup true

This works fine if there is a real domain name configured. It’s the DNS which gets the proper domain.

Since, in this tutorial, graphite and collectd will run on the same machine, uncomment the first line and set a hostname.

On line 42, there is a parameter called Interval which is the time that collectd waits before querying data. Uncomment that line, and change the value from 10 to the shortest interval value chosen in Graphite configuration. In the previous tutorial, it was 20 seconds, so the line will look like this:

Interval 20

The matching of these two values is fundamental. If they do not match some data could be lost!

Now it’s time to choose the services that collectd will gather information about. To do this, collectd uses plugins. Starting on line 88, there is a very long list. Most of the lines are commented, but you’ll want to uncomment apache and write_graphite. After this, the following plugins will be enabled

LoadPlugin apache
LoadPlugin battery
LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin entropy
LoadPlugin interface
LoadPlugin irq
LoadPlugin load
LoadPlugin memory
LoadPlugin processes
LoadPlugin rrdtool
LoadPlugin swap
LoadPlugin users
LoadPlugin write_graphite

Further down in the file, there are plugins configuration blocks. Uncomment the Apache block and add an Instance sub-block; now the block will look like this:

<Plugin apache> 
#       <Instance "foo"> 
#               URL "http://localhost/server-status?auto" 
#               User "www-user" 
#               Password "secret" 
#               VerifyPeer false 
#               VerifyHost false 
#               CACert "/etc/ssl/ca.crt" 
#               Server "apache" 
#       </Instance> 
# 
#       <Instance "bar"> 
#               URL "http://some.domain.tld/status?auto" 
#               Host "some.domain.tld" 
#               Server "lighttpd" 
#       </Instance> 
 
        <Instance "Graphite"> 
                URL "http://mydomain/server-status?auto" 
                Server "apache" 
        </Instance> 
</Plugin>

Next, configure the df block, which gives information about space occupied on the disk. For example:

<Plugin df> 
        Device "/dev/sda3" 
#       Device "192.168.0.2:/mnt/nfs" 
        MountPoint "/" 
        # FSType "btrfs" 
 
        # ignore rootfs; else, the root file-system would appear twice, causing 
        # one of the updates to fail and spam the log 
        # FSType rootfs 
        # ignore the usual virtual / temporary file-systems 
        # FSType sysfs 
        # FSType proc 
        # FSType devtmpfs 
        # FSType devpts 
        # FSType tmpfs 
        # FSType fusectl 
        # FSType cgroup 
        # IgnoreSelected true 
 
#       ReportByDevice false 
#       ReportInodes false 
 
#       ValuesAbsolute true 
#       ValuesPercentage false 
</Plugin>

Finally, go to the write_graphite section and uncomment it:

<Plugin write_graphite> 
        <Node "graphite"> 
                Host "localhost" 
                Port "2003" 
                Protocol "tcp" 
                LogSendErrors true 
                Prefix "collectd" 
                Postfix "collectd" 
                StoreRates true 
                AlwaysAppendDS false 
                EscapeCharacter "_" 
        </Node> 
</Plugin>

Note: remember that “2003” is the port number on which Carbon will listen.

Save and exit.

Configure Apache

Configure the ‘Apache virtual hosts’ file enabled for Graphite:

# $EDITOR /etc/apache2/sites-available/apache2-graphite.conf

Below the content block, add the following lines:

Alias /content/ /usr/share/graphite-web/static/
    <Location "/content/">
        SetHandler None
    </Location>

    <Location "/server-stats">
        SetHandler server-status
        Require all granted
    </Location>

    ErrorLog ${APACHE_LOG_DIR}/graphite-web_error.log

This means that Apache will serve statistics on the /server-stats page.

Save, exit and reload Apache:

# systemctl reload apache2

Check that everything is well configured by accessing http://localhost/server-stats with a web browser.

Configure storage schemas

To configure the storage schemas correctly, edit the following configuration file:

# $EDITOR/etc/carbon/storage-schemas.conf

In that file, above the default block, insert a new block. For example:

[collectd]
pattern = ^collectd.*
retentions = 20s:1d, 5m:7d, 10m:1y

Save, exit and restart the services:

# systemctl restart carbon-cache
# systemctl restart collectd

Conclusion

Now collectd should be up and running, recording all the server activities and flushing these records to Graphite.
It’s possible to configure collectd with additional plugins, or modify the configuration files to accomplish new/other tasks.

In the last part of this series of tutorials, we will talk about installing and configuring StatsD, a program for caching data collected by collectd. Stay tuned!