Forensic Analysis Tools
Version: Draft 1 (10/13/05)
You think something is wrong with a machine that you manage or you want
to see if something suspicious is running on your machine.
How would you, with current tools, go about analyzing the system? What
are the limits of these tools?
First thing is that there are two kinds of forensic data on a machine:
volatile and non-volatile. Volatile data is that which would dissapear
if you were to shutdown the machine. For example, information about
running processes, open files, and open connections is volatile.
Files and logs, on the other hand, are non-volatile.
We will first look at capturing live data. Here are a bunch of tools
that people use today on the two common platforms: Windows and
Linux/Unix.
netstat (both Windows/Linux)
% netstat -a
gives information on active connections.
% netstat -an
is a variant that shows raw IP addresses, as opposed to IP domain
names.
Try it out
What are you looking for?
You are looking for ports that are in ESTABLISHED mode or LISTENING
mode that you cannot associate with a service that
the machine normally provides.
Below is an example for one client Linux machine.
tcp
0 0
0.0.0.0:32769
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:32770
0.0.0.0:*
LISTEN
tcp
0 0
127.0.0.1:32771
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:804
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:873
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:111
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:6000
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:22
0.0.0.0:*
LISTEN
tcp
0 0
127.0.0.1:631
0.0.0.0:*
LISTEN
tcp
0 0
0.0.0.0:7100
0.0.0.0:*
LISTEN
tcp
0 0
127.0.0.1:631
127.0.0.1:40346
ESTABLISHED
tcp
0 0
127.0.0.1:40346
127.0.0.1:631
ESTABLISHED
tcp
0 0
141.213.8.165:40363
141.213.8.202:993 ESTABLISHED
tcp
0 0
141.213.8.165:800
141.213.4.18:2049 ESTABLISHED
tcp
0 0
141.213.8.165:797
141.213.4.18:2049 ESTABLISHED
tcp
0 0
141.213.8.165:796
141.213.4.18:2049 ESTABLISHED
tcp
0 0
141.213.8.165:40364
141.213.4.18:22
ESTABLISHED
udp
0 0
0.0.0.0:32768
0.0.0.0:*
udp
0 0
0.0.0.0:32769
0.0.0.0:*
udp
0 0
0.0.0.0:514
0.0.0.0:*
This machine normally runs only ssh server and is also running a VNC
server. We know that ssh server runs on port 22. VNC server
is on port 6000. So, the lines in bold appear OK. But what about the
rest?
We could look up the list ports that are used for various services at http://www.portsdb.org.
That tells us that 111 is associated with Sun RPC, 873 is associated
with rsync, 631 with Internet Printing Protocol, etc. Ports below 1024
are generally associated with systems services. The above may be
flagged suspicious if you did not think you were running these services
on your machine.
Ports such as 32769-32771 are a little more suspicious. In the ports
database, they are associated with filenet, but could also be
associated with Sun RPC.
What we would probably like to do at this point is find out is the
binary that is associated with the above.
lsof and fport
On Linux: you can use the "lsof" command. Run it as root, otherwise you
will not get a complete listing. (On Windows, you should download a
program "fport" from http://www.foundstone.com
(in Resources->Free Tools. Do a google search if you have
trouble finding it, but make sure you download it from foundstone
site). That gives similar results as "lsof".
% lsof -i | grep LISTEN
returns the following:
X
52 root 1u
IPv4
75 TCP *:x11 (LISTEN)
portmap 4675
rpc 4u IPv4
3861 TCP *:sunrpc (LISTEN)
rpc.statd 4695 rpcuser 6u
IPv4 3892
TCP *:32769 (LISTEN)
cupsd 4803
root 0u IPv4
4244 TCP
localhost.eecs.umich.edu:ipp (LISTEN)
xinetd 4857
root 6u IPv4
4333 TCP *:rsync (LISTEN)
xinetd 4857
root 8u IPv4
4334 TCP
localhost.eecs.umich.edu:32771 (LISTEN)
amd
4867 root 5u
IPv4 4300
TCP *:804 (LISTEN)
xfs
4960 xfs 3u
IPv4 4408
TCP *:xfs (LISTEN)
fam 15469
aprakash 0u IPv4
4334 TCP
localhost.eecs.umich.edu:32771 (LISTEN)
fam 15469
aprakash 1u IPv4
4334 TCP
localhost.eecs.umich.edu:32771 (LISTEN)
fam 15469
aprakash 2u IPv4
4334 TCP
localhost.eecs.umich.edu:32771 (LISTEN)
sshd 20295
root 3u IPv4
10487450 TCP *:ssh (LISTEN)
Now, we find out that it is a program called "fam" that is listening on
32771. I don't know what "fam" is, so I can investigate that further.
% locate fam (on Linux)
tells me that it is a program in
/usr/bin/fam
Further lookup of it tells me that it is to look for modification of
files. At this point, I would probably check with a sysadmin to make
sure they were running this service. I don't see it in /etc/init.d --
which is where normally startup programs would reside. I could look
further into this by taking a dump of the process. But first, I could
use "lsof" command to find out more.
% lsof | grep fam
fam 15469 aprakash
cwd DIR
3,1
4096 2 /
fam 15469 aprakash
rtd DIR
3,1
4096 2 /
fam 15469 aprakash
txt REG
3,1 142252 1137946
/usr/bin/fam
fam 15469 aprakash
mem REG
3,1 107724 746361
/lib/ld-2.3.2.so
fam 15469 aprakash
mem REG
3,1 1578228 129850
/lib/tls/libc-2.3.2.so
fam 15469 aprakash
mem REG
3,1 213244 129852
/lib/tls/libm-2.3.2.so
fam 15469 aprakash
mem REG
3,1 30604 746374
/lib/libgcc_s-3.3.2-20031023.so.1
fam 15469 aprakash
mem REG
3,1 739636 1217760
/usr/lib/libstdc++.so.5.0.5
fam 15469 aprakash
mem REG
3,1 51152 746393
/lib/libnss_files-2.3.2.so
fam 15469
aprakash 0u
IPv4
4334
TCP localhost.eecs.umich.edu:32771 (LISTEN)
fam 15469
aprakash 1u
IPv4
4334
TCP localhost.eecs.umich.edu:32771 (LISTEN)
fam 15469
aprakash 2u
IPv4
4334
TCP localhost.eecs.umich.edu:32771 (LISTEN)
fam 15469
aprakash 3u unix
0xd14f0780
11150093 /tmp/.fam_socket
fam 15469
aprakash 4u unix
0xe2cb9b00
11150099 /tmp/.famsMcU6O
fam 15469
aprakash 5r
FIFO
0,5
11150103 pipe
fam 15469
aprakash 6w
FIFO
0,5
11150103 pipe
fam 15469
aprakash 7r
DIR
3,1 8192
210913 /etc
fam 15469
aprakash 8r
DIR
3,1 4096
568300 /usr/share/mime-info
fam 15469
aprakash 9r
DIR
3,5 4096 1622025
/z/aprakash/.gnome/mime-info
fam 15469 aprakash
10r DIR
3,1 4096
405853 /etc/X11/desktop-menus
fam 15469 aprakash
11r DIR
3,1 12288 811202
/usr/share/applications
fam 15469 aprakash
12u unix
0xc67b7b00
11150237 /tmp/.famPndKQo
fam 15469 aprakash
13u unix
0xf108c780
11150247 /tmp/.famywFoBY
fam 15469 aprakash
14r DIR
3,5 4096 4751410
/z/aprakash/.gnome2/vfolders/applications
fam 15469 aprakash
15r DIR
3,1 4096
859873 /etc/X11/applnk
fam 15469 aprakash
16r DIR
3,1 4096
294629 /usr/share/applnk
fam 15469 aprakash
17r DIR
3,1 4096
278264 /etc/X11/applnk/Graphics
fam 15469 aprakash
18r DIR
3,1 4096
211798 /usr/share/control-center-2.0/capplets
fam 15469 aprakash
19r DIR
3,1 4096
294630 /usr/share/applnk/Multimedia
fam 15469 aprakash
20r DIR
3,1 4096
227490 /usr/share/desktop-menu-files
fam 15469 aprakash
21r DIR
3,5 4096 4751408
/z/aprakash/.gnome2/vfolders
fam 15469 aprakash
22r DIR
3,5 4096 2408697
/z/aprakash/.Trash
fam 15469 aprakash
23r DIR
3,5 4096 3702791
/z/aprakash/.gnome2/nautilus-scripts
fam 15469 aprakash
24r DIR
3,5 4096 1786103
/z/aprakash/.Trash/untitled folder 1
fam 15469 aprakash
25r DIR
3,5 4096 4636689
/z/aprakash/.Trash/untitled folder
fam 15469 aprakash
26r DIR
3,5 4096
163841 /z/aprakash/Desktop
fam 15469 aprakash
28r DIR
3,5 4096
163899 /z/aprakash/Desktop/Trash
The above tells me all the files that fam has open as well as libraries
that are loaded. At this point, it starts to appear that is related
to the Gnome desktop environment that I am running. That may be normal
or it may not. It may require more investigation later.
Meanwhile, I have decided to block the ports 32771 on which fam is
listening -- till I know what it is up to.
I could find a little more about fam by using the following commands:
% ls -l
-rwxr-xr-x 1 root root 142252 Aug 12 2003 /usr/bin/fam
At least, it does not appear to have been updated recently -- so may be
not an attack. (But timestamps can be manipulated if the attacker had
root access).
ldd (Linux)
I can check what libraries it relies on. If I conclude that fam is
misbehaving, I should go back and make sure that all the libraries that
it relies on have not been tampered with or don't have known exploits.
[ 4 ] aprakash -: ldd /usr/bin/fam
libstdc++.so.5 =>
/usr/lib/libstdc++.so.5 (0x007c0000)
libm.so.6 =>
/lib/tls/libm.so.6 (0x004f4000)
libgcc_s.so.1 =>
/lib/libgcc_s.so.1 (0x007b6000)
libc.so.6 =>
/lib/tls/libc.so.6 (0x003b9000)
/lib/ld-linux.so.2 =>
/lib/ld-linux.so.2 (0x003a1000)
I can check its md5sum and compare it with a "standard" version of fam
(on another machine that I trust or from a place where fam is
available).
% md5sum /usr/bin/fam
7d10063389d83f6c3e344345647e58a1 /usr/bin/fam
Ideally, I should have saved md5sums of critical files on the system so
that I could diff them against the current snapshot. A program
called "tripwire" does that.
I can go further and look at network activity of fam, but more on that
later.
Windows tools
-------------------
On Windows, you would go through a similar process:
% netstat -a OR netstat -an
to determine list of active ports.
% fport
(free from http://www.foundstone.com)
to find the applications associated with the ports.
Once you found the applications, if anything raises a question, you
could investigate further by using a program called "pslist"
(similar to ps, from http://www.sysinternals.com). It may tell you how
long a suspicious process has been running, as well as
other processes that were launched around the same time.
% pslist
Listing similar to ps on Unix.
Another useful command from www.sysinternals.com is the following:
% psexesvc
It reports information about a service on Windows. Often, attack code
on Windows will be wrapped as a service (daemon on Unix) so
that it launches every time the machine starts.
To find files that open for a process on Windows, use the command
(again from www.sysinternals.com):
% psfile
If the command reports that the program has pipes open to other
machine, that could raise a flag.
Dumping the state of a running program
It can be useful to dump the state of a running program to get an idea
of loaded libraries, open files, etc. On Windows, a program
that can be used for this is "userdump". Search for "Microsoft
OEM Support Tools" at the Microsoft web site and download a
package "Oem3sr2.zip".
You can use userdump to dump a process as follows:
% userdump <processID> <dumpfile>
(You can get the processID from fport or pslist).
For example, to dump the process 1032,
% userdump 1032 C:\temp\1032.dmp
Ideally, you want to dump the stuff to a safe machine, since your
machine may not be stable if it is under attack.
Once you have a dump, you can use several utilities to analyze the dump.
% dmpchk C:\temp\1032.dmp
Will print out the pathname of the executable for the process, as well
as DLLs used.
You can find dmpchk at Microsoft Web site as part of Debugging
Tools for Windows (free). (or search for this, if the link is out
of date).
You can also use a program "strings" that will look for ASCII strings
in the dump. That can also give clues about the program.
This program is available at http://www.sysinternals.com/.
What you may want to look for are presence of strings that have paths
to other executables, such as ftp, irc, etc. That may mean that the
program is using other programs to communicate with the outside.
It is also important to examine non-volatile data (logs, registry,
files), etc. but that is whole another topic.
Analyzing Network Activity
For intrusion detection and forensic analysis, besides examining open
ports, processes, etc., it can be useful to monitor and analyze network
activity. Here are the key tools that one can use for this.
Root/administrative access is likely to be required to use these tools.
Capturing all the Content
1. tcpdump (http://www.tcpdump.org .
A Windows version is available at
http://winpcap.org/windump ): dumps TCP activity to the console.
2. ethereal (http://www.ethereal.com):
captures packet-level activity and can give you contents of the
session. Available for all platforms.
3. etherape (http://etherape.sourceforge.net):
presents ethereal content data graphically.
4. Ngrep (http://www.packetfactory.net/projects/ngrep/):
you can serach for strings in captured packets.
5. Flowgrep (http://www.monkey.org/~jose/software/flowgrep):
searching for patterns across a stream of packets.
Capturing a specific session
Ethereal (by its "follow TCP stream" feature) or tcpdump can be used,
with appropriate arguments, to capture activity on specific ports. But
other tools that are specifically
designed for that are:
1. Argus (http://www.qosient.com/argus):
can be run in "batch" mode (on captured packets) or "live" mode
(without packets going to disk).
2. Tcptrace (http://www.tcptrace.org):
works in batch mode. It can take dumps from tcpdump and extract
sessions.
3. Tcpflow (http://www.circlemud.org/~jelson/software/tcpflow/):
command-line tool to extract flows, unlike a GUI-based tool such as
Ethereal.
Alert Generation Tools
Snort is the de-facto standard at the present time. Alteranatives
are Bro and Shoki.
1. Snort (http://www.snort.org)
2. Bro (http://www.bro-ids.org)
3. Sguil (http://www.sguil.net):
Provides GUI around Snort and network monitoring tools.
Statistical Analysis Tools
This area is relatively weak at this point in terms of open source
tools, but the following are worth a try:
1. Tcpdstat (http://staff.washington.edu/dittrich/talks/core02/tools/tools.html):
may be the best option
2. Tcpstat (http://www.frenchfries.net/paul/tcpstat/)
3. Ntop (http://www.ntop.org)
Vulnerability Assessment Tools
Ask Laura to present her experience on this?
1. Nessus (http://www.nessus.org):
This seems to be be the most popular. It scans systems for
vulnerabilities. Here is an introduction to it.
Other, simpler tools, include "nmap" (to find out open ports),
"superscan" (similar to nmap from Foundstone), and SAINT.