How To Install And Use Sysdig Troubleshooting Tool

Introduction

One of the most imporant task for an system administrator is the troubleshooting and diagnostic of problems. Today you can say goodbye to top, lsof, strace, iostat, ps, etc., and you can use  sysdig: The most powerful Linux system troubleshooting. 

Sysdig is open source, system-level exploration and a troubleshooting tool that captures system state and activity from a running Linux instance using a linux kernel facility called tracepoint , then save, filter and analyze. Sysdig combines the benefits of many utilities such as strace, tcpdump, and lsof into one single application.

Sysdig is also packed with a set of scripts called Chisels that make it easier to extract useful information and do troubleshooting.

In this article we’ll explore the installation and basic usage of sysdig to perform system monitoring and troubleshooting on Linux.

For more information, you can see the website of Sysdig.

Features Of Sysdig

  • Fast and stable
  • Easy to Use
  • Debugging pretty much anything
  • Crazy Powerful
  • and more

Installing Sysdig On Ubuntu

To install

sysdig

via apt we will need to setup the apt repository maintained by Draios the company behind

sysdig

. We can do this by running the following

curl

commands.

The following commands should be run by root user.


curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -  
curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list

After this command you need to update your server. Use this command to do that.


apt-get update

The

sysdig

 tools requires the kernel headers package before installing it. Flow the command to install the kernel headers package.

Use this command to check the kernel version.

uname -r

apt-get install linux-headers-<kernel version>

Now you can install sysdig on ubuntu using this command.


apt-get install sysdig

Installing Sysdig On CentOS

Before stating with the installation itself, you will need to setup the yum repository that will use this key to verify the authenticity of the package you’re about to download.

Use the

rpm

tool with the

--import

flag To manually add the Draios key to your RPM keyring.


sudo rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public

Then, download the Draios repository and configure yum to use it:


sudo curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/stable/rpm/draios.repo

After that try to make update to your server using this command

yum update

Then you need to enable extra packages for enterprise Linux (EPEL) The EPL repository is needed in order to download the Dynamic Kernel Module Support (DKMS) package, which is used by sysdig tool. Flow the command to enable EPEL repository:


sudo yum -y install epel-release

Then you need to install kernel headers in order to setup 

<strong>sysdig-probe</strong> module used by <strong>sysdig</strong>.

sudo yum -y install kernel-devel-$(uname -r)

Now you can install sysdig.


sudo yum -y install sysdig

Using Sysdig

The simplest and easiest method to use sysdig is by invoking it without any argument.By default, sysdig prints the information for each captured event on a single line.


sysdig

you will see output like this example.

25722 22:48:41.788762892 3 sshd (1733) > rt_sigprocmask
25723 22:48:41.788763312 3 sshd (1733) < rt_sigprocmask
25724 22:48:41.788763603 3 sshd (1733) > rt_sigprocmask
25725 22:48:41.788763801 3 sshd (1733) < rt_sigprocmask
25726 22:48:41.788764486 3 sshd (1733) > read fd=11(<f>/dev/ptmx) size=16384
25727 22:48:41.788765210 2 sysdig (3745) > switch next=318 pgft_maj=0 pgft_min=910 vm_size=92452 vm_rss=5532 vm_swap=0
25728 22:48:41.788766062 3 sshd (1733) < read res=118 data=8622 22:48:41.759980138 2 sysdig (3745) > switch next=318 pgft_maj=0 pgft_min...
25729 22:48:41.788778756 2 <NA> (318) > switch next=3745(sysdig) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
25731 22:48:41.788785389 3 sshd (1733) > select

To write output of the sysdif in a file you can use the

-w

flag  and specify the file name.


sysdig -w &lt;output file&gt;

Example:


sysdig -w trace.dump

You can read the output file using this command


sysdig -r &lt;output file&gt;

Example:


sysdig -r trace.dump

Sysdig

command has filters that allow you to filter the output to specific information. You can find a list of available filters by running this command.


sysdig -l

sysdig -l
 ----------------------
    Field Class: fd

    fd.num            the unique number identifying the file descriptor.
    fd.type           type of FD. Can be 'file', 'ipv4', 'ipv6', 'unix', 'pipe', 'e
                      vent', 'signalfd', 'eventpoll', 'inotify' or 'signalfd'.
    fd.typechar       type of FD as a single character. Can be 'f' for file, 4 for
                      IPv4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pi
                      pe, 'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i'
                       for inotify, 'o' for uknown.
    fd.name           FD full name. If the fd is a file, this field contains the full path.
                       If the FD is a socket, this field contain the connection tuple.
&lt;truncated output&gt;

You can use the “proc.name” filter to capture all of the

sysdig

events for a specific process.


sysdig -r tracefile.dump proc.name=name-of-process

Example:


sysdig -r tracefile.dump proc.name=sshd
    530 23:45:02.804469114 0 sshd (917) &lt; select res=1
    531 23:45:02.804476093 0 sshd (917) &gt; rt_sigprocmask
    532 23:45:02.804478942 0 sshd (917) &lt; rt_sigprocmask
    533 23:45:02.804479542 0 sshd (917) &gt; rt_sigprocmask
    534 23:45:02.804479767 0 sshd (917) &lt; rt_sigprocmask
    535 23:45:02.804487255 0 sshd (917) &gt; read fd=3(&lt;4t&gt;10.0.0.12:55993-&gt;162.0.0.80:22) size=16384

You can see here many examples how to use sysdig.

That all!

Hope this article help you.