ngrep Instructions for:
“ngrep_hour.sh” and “ngrep_daily.sh” script for use with Shadow IDS hourly raw TCPDump files
ngrep strives to provide most of GNU grep's common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes TCP, UDP and ICMP across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands bpf filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.
This is a highly functional utility for use with packet dumps. ngrep has the functionality not only to capture packets in real time and search for regular expressions in real time, but it also has the ability to read previously recorded pcap files. This means it can read in binary tcpdump files which works well with raw Shadow logs. Parsing through previously recorded Shadow logs is the main intent of this utility and is what these instructions are geared towards.
As the description above states; ngrep provides most of GNU grep's common features. The strengths here lie in allowing regular expressions to be applied to network packets. This allows a phenomenal amount of flexibility in matching patterns you might be looking for within a tcpdump file. Unlike tcpdump BPF filters, you do not need to know the exact byte offset to search from. ngrep will look at the entire packet, including the payload and attempt to match the regular expression provided. In addition to this functionality ngrep also allows the use of BPF filters which further enhances optimizes searching and process efficiency. BPF filters can be saved in files and applied from there which allows for some very advanced filtering (Shadow configurations are based upon BPF filters saved in files).
The Shadow analysis station works by copying raw tcpdump files from the remote sensors and then running them through a bunch of BPF filters and PERL scripts to create hourly HTML files. As these raw files have already been copied to the analysis station, this central repository of logs makes a good location for employing ngrep.
What we will be doing is:
1. provide some variables to the ngrep_hour script so it can choose the log file you wish to perform analysis on
2. take a gzip'ed tcpdump raw log and make this available to ngrep for processing
3. provide some variables to the script which will be interpreted by ngrep
4. taking the output from ngrep and writing it to a file for further analysis
So why would you want to use ngrep?
Say you had a scan for that you want to find out more about. You want to see if any servers had responded to the request, returning the directory contents of the servers C drive. A positive return of these contents would surely identify a server in need of follow-up action. You could do this very quickly and efficiently with ngrep through the ngrep_hour.sh or ngrep_daily.sh scripts. Here is how…
provide the following regular expression when prompted and you would get back all of the hits with the strings c+dir+c and Directory of c.
[C|c][+][D|d][I|i][R|r][+][C|c]|Directory of [C|c]
Not only would you get this, but you would get every variation of the exploit using upper and lower case ASCII characters.
This is just an example of some of the things you can do with regular expressions.
When filling in the variable prompted for by the script, you do not need any quotes as identified by the ngrep manpage. These have been hard-coded into the script. Simply supply your expression.
To perform the most basic of content string searches you need enter the string you wish to match. An example would be:
This would match every packet with the string '200 OK' in it which would imply that you were looking for HTTP responses from servers where the src port would be 80 and the protocol would be TCP. These are all returned pages from the web server to the client.
This being said, it would be better to follow the regular expression with a BPF filter that is quicker at parsing through large files like those from the Shadow sensors. The filter portion of the ngrep command would look like this:
tcp src port 80
Now if we put it all together, this is what the command actually being executed by the script will look something like:
/bin/gunzip -c /tcpdump_results/$SENSOR/$DATE/tcp.2002*$HOUR.gz | /usr/local/bin/ngrep -qt -I - "$MYSEARCH" $FILTER >> /home/shadow/result_$FILE.txt
/bin/gunzip -c /tcpdump_results/$SENSOR/$DATE/tcp.2002*$HOUR.gz | /usr/local/bin/ngrep -qt -I - "200 OK" tcp src port 80 >> /home/shadow/result_$FILE.txt
To play with this a little create a test dump by running the following command (note – tcpdump version 3.2.6 or greater):
Run: tcpdump –Xvn –w /home/shadow/scripts/mytest
Run: ping –p 2f62696e2f7368202020 –c 15 172.16.0.1
Run: ping –p 2f7573722f62696e2f7368202020 –c 15 172.16.0.1
Run: ping –c 15 172.16.0.1
The tcpdump capture should now be terminated. Each analyst should try out the script and see what the output looks like as well as put it to use during analysis.
Run: tcpdump -Xvn -r /home/shadow/scripts/mytest > /tmp/mytest.txt
Then have a look at the file:
Run: more /tmp/mytest.txt
Notice the patterns in this file in the ASCII portion of the payload. Now try and run ngrep against this raw tcpdump file and provide some regular expressions to try and grab variations of the traffic. An example would be:
Run: ngrep -q -t -I /home/shadow/scripts/mytest '/bin/sh' ip and icmp > /tmp/mytest2.txt
Run: more /tmp/mytest2.txt
Run: ngrep -t -A 1 -I /home/shadow/scripts/mytest '*bin/sh' > /tmp/mytest3.txt
Run: more /tmp/mytest3.txt
Run: ngrep -qt -I /home/shadow/scripts/mytest '.bin.sh' > /tmp/mytest4.txt
Run: more /tmp/mytest4.txt
Play with it a bit more or try it on some other tcpdump captures and see what you can come up with. This should greatly reduce analysis time when looking for specific patterns within packets in the future.
The ngrep_hour.sh and ngrep_daily.sh scripts will guide you through with prompts. When you have good grasp of what ngrep can do and how to specify filters, give these scripts a try.
Another useful feature of ngrep is the option to have ngrep dump x number of packets after each match of the regular expression is found. This is similar in functionality to the 5.0 version of DragonIDS (though not as refined as to keep track of sessions). If your interested in seeing what the next few packets on a network look like after a string your matching on, modify the ngrep arguments and provide a –A n where n is the number of packets following a match that should be displayed as well.
Some good resources for learning about regular expressions may be found at the following URL's:
ngrep static UNIX ELF Binary
ngrep Source Code
ngrep Win32 Binary
ngrep Win32 MS Visual C++ Source Code
17 June 2002
© 2000-2014 Whitehats.ca