ngrep Instructions for:

“ngrep_hour.sh” and “ngrep_daily.sh” script for use with Shadow IDS hourly raw TCPDump files

 

 

Author:

Jordan Ritter

jpr5@darkridge.com

 

Version:

1.40.1

 

Description:

ngrep strives to provide most of GNU grep's common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes TCP, UDP and ICMP across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands bpf filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.

Manpage:

http://www.packetfactory.net/projects/ngrep/

 

Background:

 

This is a highly functional utility for use with packet dumps.  ngrep has the functionality not only to capture packets in real time and search for regular expressions in real time, but it also has the ability to read previously recorded pcap files.  This means it can read in binary tcpdump files which works well with raw Shadow logs.  Parsing through previously recorded Shadow logs is the main intent of this utility and is what these instructions are geared towards.

 

As the description above states; ngrep provides most of GNU grep's common features.  The strengths here lie in allowing regular expressions to be applied to network packets.  This allows a phenomenal amount of flexibility in matching patterns you might be looking for within a tcpdump file.  Unlike tcpdump BPF filters, you do not need to know the exact byte offset to search from.  ngrep will look at the entire packet, including the payload and attempt to match the regular expression provided.  In addition to this functionality ngrep also allows the use of BPF filters which further enhances optimizes searching and process efficiency.  BPF filters can be saved in files and applied from there which allows for some very advanced filtering (Shadow configurations are based upon BPF filters saved in files).

 

The Shadow analysis station works by copying raw tcpdump files from the remote sensors and then running them through a bunch of BPF filters and PERL scripts to create hourly HTML files.  As these raw files have already been copied to the analysis station, this central repository of logs makes a good location for employing ngrep.

 

What we will be doing is:

 

1.        provide some variables to the ngrep_hour script so it can choose the log file you wish to perform analysis on

2.        take a gzip'ed tcpdump raw log and make this available to ngrep for processing

3.        provide some variables to the script which will be interpreted by ngrep

4.        taking the output from ngrep and writing it to a file for further analysis

 

So why would you want to use ngrep?

 

Say you had a scan for that you want to find out more about.  You want to see if any servers had responded to the request, returning the directory contents of the servers C drive.  A positive return of these contents would surely identify a server in need of follow-up action.  You could do this very quickly and efficiently with ngrep through the ngrep_hour.sh or ngrep_daily.sh  scripts.  Here is how…

 

provide the following regular expression when prompted and you would get back all of the hits with the strings c+dir+c and Directory of c. 

 

[C|c][+][D|d][I|i][R|r][+][C|c]|Directory of [C|c]

 

Not only would you get this, but you would get every variation of the exploit using upper and lower case ASCII characters.

 

This is just an example of some of the things you can do with regular expressions.

 

When filling in the variable prompted for by the script, you do not need any quotes as identified by the ngrep manpage.  These have been hard-coded into the script.  Simply supply your expression.

 

To perform the most basic of content string searches you need enter the string you wish to match.  An example would be:

 

200 OK

 

This would match every packet with the string '200 OK' in it which would imply that you were looking for HTTP responses from servers where the src port would be 80 and the protocol would be TCP.  These are all returned pages from the web server to the client.

 

This being said, it would be better to follow the regular expression with a BPF filter that is quicker at parsing through large files like those from the Shadow sensors.  The filter portion of the ngrep command would look like this:

 

tcp src port 80

 

Now if we put it all together, this is what the command actually being executed by the script will look something like:

 

/bin/gunzip -c /tcpdump_results/$SENSOR/$DATE/tcp.2002*$HOUR.gz | /usr/local/bin/ngrep -qt -I - "$MYSEARCH" $FILTER >> /home/shadow/result_$FILE.txt

 

or

 

/bin/gunzip -c /tcpdump_results/$SENSOR/$DATE/tcp.2002*$HOUR.gz | /usr/local/bin/ngrep -qt -I - "200 OK" tcp src port 80 >> /home/shadow/result_$FILE.txt

 

To play with this a little create a test dump by running the following command (note – tcpdump version 3.2.6 or greater):

 

Run: tcpdump –Xvn –w /home/shadow/scripts/mytest

Run: ping –p 2f62696e2f7368202020 –c 15 172.16.0.1

Run: ping –p 2f7573722f62696e2f7368202020 –c 15 172.16.0.1

Run: ping –c 15 172.16.0.1

 

The tcpdump capture should now be terminated.  Each analyst should try out the script and see what the output looks like as well as put it to use during analysis.

 

Run: tcpdump -Xvn -r /home/shadow/scripts/mytest > /tmp/mytest.txt

 

Then have a look at the file:

 

Run: more /tmp/mytest.txt

 

Notice the patterns in this file in the ASCII portion of the payload.  Now try and run ngrep against this raw tcpdump file and provide some regular expressions to try and grab variations of the traffic.  An example would be:

 

Run: ngrep -q -t -I /home/shadow/scripts/mytest '/bin/sh' ip and icmp > /tmp/mytest2.txt

Run: more /tmp/mytest2.txt

Run: ngrep -t -A 1 -I /home/shadow/scripts/mytest '*bin/sh' > /tmp/mytest3.txt

Run: more /tmp/mytest3.txt

Run: ngrep -qt -I /home/shadow/scripts/mytest '.bin.sh' > /tmp/mytest4.txt

Run: more /tmp/mytest4.txt

 

Play with it a bit more or try it on some other tcpdump captures and see what you can come up with.  This should greatly reduce analysis time when looking for specific patterns within packets in the future.

 

The ngrep_hour.sh and ngrep_daily.sh scripts will guide you through with prompts.  When you have good grasp of what ngrep can do and how to specify filters, give these scripts a try.

 

Another useful feature of ngrep is the option to have ngrep dump x number of packets after each match of the regular expression is found.  This is similar in functionality to the 5.0 version of DragonIDS (though not as refined as to keep track of sessions).  If your interested in seeing what the next few packets on a network look like after a string your matching on, modify the ngrep arguments and provide a –A n where n is the number of packets following a match that should be displayed as well.

 

http://www.whitehats.ca/main/members/Malik/malik_shadow/malik_ngrep_hour.html

http://www.whitehats.ca/main/members/Malik/malik_shadow/malik_ngrep_daily.html

 

Some good resources for learning about regular expressions may be found at the following URL's:

 

http://www.phpbuilder.com/columns/dario19990616.php3

http://www.ists.dartmouth.edu/IRIA/knowledge_base/ngrep-intro.v0.8.htm

http://gnosis.cx/publish/programming/regular_expressions.html

http://www.packetfactory.net/projects/ngrep/

 

ngrep static UNIX ELF Binary

http://prdownloads.sf.net/ngrep/ngrep-1.40.1-linux-elf-static.gz

 

ngrep RPM

http://prdownloads.sf.net/ngrep/ngrep-1.40.1-1.i386.rpm

 

ngrep Source Code

http://prdownloads.sf.net/ngrep/ngrep-1.40.1.tar.gz

 

ngrep Win32 Binary

http://prdownloads.sf.net/ngrep/ngrep-1.40-win32-full.zip

 

ngrep Win32 MS Visual C++ Source Code

http://prdownloads.sf.net/ngrep/ngrep-1.40-win32-source.zip

 

 

17 June 2002

Jamie French

J.French@whitehats.ca


Non-Active Sitemap

Copyright © 2000-2014 Whitehats.ca
Contact Information 519.221.9132 : Web Contact webmaster@whitehats.ca