Project Aims

Honeypots have existed as a concept in computer security for many years and are currently enjoying a resurgence. Many current tools and solutions for honeypots are “low interaction” honeypots, i.e. mimic a real service to see what an attacker does. However, attackers can often detect that they are interacting with a honeypot. Real systems that are instrumented to detect an attacker (high interaction honeypots) are harder to detect as honeypots.

Honeypots deployed on the public internet often just catch attackers who are “spraying and praying” for easy shells, and so anyone they catch are unlikely to be a real threat to an organisation. Honeypots deployed internally on a network are more interesting as anyone they catch already has the capability to send packets to internal hosts and as such are likely to be a real threat.

Organisations may find it useful to easily clone existing machines and turn them into high interaction honeypots, so that the internal network can be prepared with convincing hosts for an attacker to stumble upon.

The purpose of this intern research project was to investigate sysdig and falco tools, and how we can leverage them in order to quickly set up, monitor and investigate high interaction honeypots.

Sysdig and Falco

Sysdig is an open source tool, which can capture and save system state and activity from a running Linux machine. It is then easy to perform an offline analysis of the process, network and I/O activities that took place. Falco, an open source add on to Sysdig, is a behavioural activity monitor designed to detect anomalous activity in applications. Falco can detect and alert on any behaviour that involves making Linux system calls. Thanks to sysdig’s core decoding and state tracking functionality, falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process.

Turning Servers into Honeypots

Every server that we would like to act as honeypot should have filebeat (or another shipper for log data) and the cifs-utils package in order to mount a remote samba share, where we save the sysdig capture files.

Manual Installation

Sysdig and falco can be deployed in different Linux distributions. Below you can see the installation process, which is easy and quite straight forward, for different Linux distributions.

1.  Installation for RHEL, CentOS and Fedora

• • Trust the Draios GPG key and configure the yum repository
  • Install EPEL repository (this is required only if DKMS is not available in the distribution. You can verify if DKMS is available with “yum list dkms“
  • Install kernel headers
  • Install falco

The following commands perform the above steps (run below commands with root privileges).

rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
curl -s -o /etc/yum.repos.d/draios.repo https://download.draios.com/stable/rpm/draios.repo
rpm -i https://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
yum -y install kernel-devel-$(uname -r)
yum -y install falco psmisc

2. Installation for Ubuntu, Debian

Follow the same concept as for RHEL family operating systems by executing the following commands:

curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -
curl -s -o /etc/apt/sources.list.d/draios.list https://download.draios.com/stable/deb/draios.list
apt-get update
apt-get -y install linux-headers-$(uname -r)
apt-get -y install falco

Automated Installation

We wanted to be able to rapidly deploy sysdig and falco in any Linux distribution. Hence we developed a simple chef cookbook, described below and available on GitHub.

When the cookbook finishes its tasks, sysdig starts running in file rotation mode for continuous capture. Also falco starts running, sending all the alerts to syslog. We ensured that filebeat was running and shipping syslogs to an ELK stack we set up for centralised analysis. In addition we checked that we mounted the directory where the sysdig captured file existed to the samba share. The last step was to configure the root account with weak credentials, like passw0rd and test123 in our case.

To create a honeypot, a suitable victim machine should be cloned and have the cookbook run on it. We can run this cookbook on Red Hat, Fedora, CentOS, Debian or Ubuntu operating systems. To run the cookbook, run the following commands with root privileges.

wget https://packages.chef.io/files/stable/chefdk/1.0.3/ubuntu/16.04/chefdk_1.0.3-1_amd64.deb
dpkg -i chefdk_1.0.3-1_amd64.deb

The above link was used in order to install chefdk in a Debian operating system. For other architecture packages see this link.

Now we should install git and clone the cookbook repository.

apt-get install git -y
mkdir ~/cookbooks
cd ~/cookbooks
git clone https://github.com/mwrlabs/honeypot_recipes sysdig-falco
cd sysdig-falco

Before running the cookbook, we should ensure that the we are running the latest kernel version. In this way we will avoid different versions between the kernel and kernel-devel packages.

Then we can run the cookbook by executing the following command:

chef-client --local-mode --runlist 'recipe[sysdig-falco]'

When the cookbook completes it’s tasks, we can see that a file, called mysysdig, has been created under the /etc/init.d/ directory. This init script will start sysdig with the following arguments:

sysdig -C 5000 -W 3 -w /usr/local/src/image.$(date +"%Y%m%d-%H%M%S").gz > /dev/null 2>&1 &

Sysdig will start in continuous capture mode with file rotation(option -W). Also we combine the previous option with the -C option, to configure the size of the files. Finally with -w option we save the capture files in /usr/loca/src/ directory. We can change these options, just by editing the mysysdig.erb file, which is in the cookbook under the /sysdig-falco-cookbook/templates/default/ directory.

Samba Server

Apart from the honeypot servers, we need a samba server in order to send our captured files to in real time. Below are the steps we followed to set up the samba server.

apt-get install samba -y
smbpasswd -a ubuntu

Where ubuntu is our username. Create a directory, where the sysdig captured files will be saved.

mkdir -p /home/ubuntu/image

Then edit the smb.conf file, which is under the /etc/samba/ directory, and in the end of this file append the following:

[image]
path = /home/ubuntu/image
valid users = ubuntu
read only = no

Finally start the samba service.

/etc/init.d/samba restart

Now we are ready to start the init script, which will start the sysdig program. However, before this we need to mount the remote file share under the /usr/local/src/ directory.

apt-get install cif-utils -y
mount.cifs -o user=ubuntu //IP_of_samba_server/image /usr/local/src/image/

A password prompt appears and we should put the password that we chose(from the smbpasswd -a ubuntu command we ran).

Now we ensure that falco is running, and start sysdig through the init script.

/etc/init.d/falco status
/etc/init.d/mysysdig start

Monitoring the attackers

We experimented with monitoring for when an attacker was interacting with our honeypots. We used the following architecture for our honeypots:

We spawned:

• Several Linux machines that acted as the honeypots
• One Ubuntu machine that acted as a samba server
• One Ubuntu machine on which we installed the ELK (Elasticsearch Logstash Kibana) stack.

The choice of the ELK stack was made as it is suited well to consuming and analysing data from syslog, which sysdig/falco output to.

We installed filebeat on the two honeypots in order to send both syslog and auth.log to the ELK stack. The link we followed to install ELK stack and filebeat can be found here.

To detect attackers we monitored authentication logs (auth.log) in Kibana. As well as the authentication logs we were monitoring falco alerts. For example the following alerts from falco indicates that someone interacted with one of our honeypots.

As a crude measure, we could also observe the size of the sysdig captured files. A sudden increase would indicate that someone is interacting with our honeypots as little activity should be taking place on these machines.

Case Studies

The purpose of this intern project was to investigate rapidly deploying Sysdig / Falco for honeypotting hosts. To see whether the monitoring done by sysdig and falco was effective for detecting attackers we placed honeypots online, running on cloud based infrastructure.

Case Study 1

We exposed the honeypots to the internet and quickly we had our first results. An attacker managed to guess the weak root credentials we had set up as we can see in the following screenshot.

This incidence happened at 17:25:00 as we can see from the above picture. At 17:26:36, an attacker with a different IP managed to guess the root credentials as we can see from the following picture.

We believe that the same attacker was using two different IP addresses. Initially the attacker brute forced the honeypot from the 221.194.44.216 IP address, possibly using a script or a previously compromised host and once the credentials were found, may have been alerted by the script and came back from the 103.42.31.138 IP address. The following images show hits from the two IP addresses.

From the first image above, we can observe that the IP address 221.194.44.216 has 2522 generated alerts(above right corner). These alerts are associated with failed attempts at guessing the root credentials as we can see from the following image.

The attacker found the root credentials and revisited from the 103.42.31.138 IP address. Searching for different alerts that are associated with this specific IP address we found only two alerts as we can see in the following picture.

The falco alerts showed that someone broke into our honeypot as we can see in the following picture.

We quickly downloaded the sysdig captured file and ran the following chisel (chisels are little scripts that analyse the sysdig event stream to perform useful actions).

sysdig -r image.20160721-1547320 -c list_login_shells

We used another chisel to see what commands the attacker ran, based on the loginshellid, by running the following command through sysdig.

sysdig -r image.20160721-1547320 -c spy_users proc.loginshellid=1743

From the alerts of falco we saw that the attacker created a file in the /etc/cron.hourly directory with the name zbbpxdqalfe.sh. Helpfully through sysdig it was easy to see the contents of the file, as sysdig recorded every single I/O operation.

Running the following command:

sysdig -r image.20160721-1547320 -A -c echo_fds fd.filename=zbbpxdqalfe.sh

we got the following output.

Searching for the command ”mv /usr/bin/wget /usr/bin/good ” that the attacker ran, we found an analysis about an ELF malware binary trying to DDoS different targets. The analysis spoke about a polymorphic malware, which creates different executable files, with random names, under the ”/bin” or ”/usr/bin” directories, something that we can see from the following falco alerts.

Case Study 2

We exposed the honeypots with some common usernames that botnets usually try to brute force. More specifically, we added three accounts with usernames test, user, user2, with the same password as the username. A couple of hours later, someone logged in with the username “user“. After fourteen minutes the attacker came back and logged in again, as we can see from auth.log through kibana.

We checked falco alerts, but nothing seemed to be relevant with the specific incident. We downloaded the sysdig capture file from the samba server and tried to see what happened. First we used the “spy user” chisel to see all the interactive commands that had been executed.

sysdig -r image.20160729-104943.gz0 -c spy_users

We observed some of the commands that had been executed by the attackers. In order to see more clearly what the attacker tried to do, we ran the same chisel again, but this time we used proc.loginshellid filter field to restrict the output of the spy user chisel. As we can see from the previous image, the loginshellid of the attacker is 2308, so we ran:

sysdig -r image.20160729-104943.gz0 -c spy_users proc.loginshellid=2308

and observed the following output.

As we can see the attacker tried to run some binaries for different processor architectures. The first binary they tried to run is called kaiten. Searching for kaiten binary in google, we can see that this malware binary is written in C, and has been openly in many exploit database sites for a long time. The main purpose is to DDoS different targets, via utilizing our unix boxes and IoT devices. More information can be found in this link.

Analyzing the binaries, we found that none of these had symbols stripped, making it easier to reverse engineer them. After some additional research, a script on github was discovered to set up this bot. None of the hardcoded encryption keys/channel keys have been changed, so it is fairly obvious that the attacker just downloaded and ran this script. Reversing the Jackmy$arch binaries, we discovered that these binaries hide their process listing in ps aux by replacing it’s first argument to the process (the proc name) with /usr/bin/sshd.

Sample Name	Bot Class	C2 IP
Jackmy$arch	QBot	212.24.102.44:666
Kaiten	STD-IRC Bot	212.24.102.44
rbot	STD-IRC Bot

Conclusion

Whether honeypots are useful to an organisation is based on if the effort to reward ratio is right. Too much effort should not be spent honeypotting a network as it could likely better be spent on other areas of defence.

Source:https://labs.mwrinfosecurity.com/blog/high-interaction-honeypots-with-sysdig-and-falco#