G R A N T S T A V E L Y G R A N T S T A V E L Y G R A N T S T A V E L Y G R A N T S T A V E L Y

I've found this script very useful for network security monitoring, incident handling, and analyst training.

The problem(s)

Frequently, security analysts kick off tcpdump full packet captures on unix servers with tapped interfaces at trust zone perimeters which they leave running in the background. Often this is done in a hurry in order to catch something before it disappears, and just as often, the capture is left running for a few days, weeks, or even months at a time.

Unfortunately, this leaves dozens of tcpdump capture files strewn about home directories, all of them poorly named and all too often left running and forgotten. Which ones are junk? Which can be archived or deleted? What a mess!

The Horror

And this is even more problematic for those of you practicing network security monitoring with daemonlogger or a similar rolling full packet capture daemon.

How can an analyst dig through hundreds of large pcap files for just the traffic they want? Whether it is 50Gb of accumulated network traffic that left a T1 connection over the past week, or that left an OC3 connection in the past 15 hours, it is too much to pull a file at a time, even with a guess where to start. These nsm daemons can easily generate a Gb of data every minute! Neither tcpdump -r nor daemonlogger -r accept wild cards. Hunting for an event one file at a time? No thanks.

Who wants to mess with ps, grep, and kill, much less the oft repeated -nn -s 1516 -i interface during incident handling?

A solution

I created a perl script to manage starting, monitoring, and stopping all packet captures - live or from active daemonlogger pcap files. I retrained all my analysts to use it. I updated my sensor monitoring scripts to start using it too.

I think this would be useful at ISP's, .edu's, and enterprise organizations so I refactored what my team and I have been using for the past year to make it significantly better. Time to release it!

capture

> capture
Usage: capture [-h?lsmv] [r|R] [-a analyst -d 'quoted description' -e 'quoted expression'] [-f filter]

Starting captures

Script the repetitive stuff, force the informative and useful stuff, and prevent mistakes.

> sudo capture -a grant -d 'traffic to grantstavely.com' -e 'host 205.134.166.178'
Password:
Started: grant_traffic.to.grantstavely.com_Tue.Feb17.2009-20.20.22.UTC_1234902022_.pcap

Monitoring captures

> sudo capture -l
grant Tue Feb 17 15:20:22 2009 traffic to grantstavely com
will Tue Feb 17 15:24:05 2009 irc traffic
philip Tue Feb 17 15:24:41 2009 strange malware on 8081

Too many, I'm checking on my own captures only!

> sudo capture -lf grant
grant Tue Feb 17 15:20:22 2009 traffic to grantstavely com

Need more detail?

> sudo capture -lv
Analyst: grant
Size (bytes): 720.00 KB
Started: Tue Feb 17 15:20:56 2009
Last Modified: Tue Feb 17 15:21:31 2009
Last Accessed: Tue Feb 17 15:20:56 2009
Last Changed: Tue Feb 17 15:21:31 2009
Description: traffic to grantstavely com
Expression: host 75.101.142.201
Capture File:
/Users/grant/captures/grant_traffic.to.grantstavely.com_Tue.Feb17.2009-20.20.56.UTC_1234902056_.pcap

Stopping Captures

> sudo capture -svf grant
Analyst: grant
Size (bytes): 720.00 KB
Started: Tue Feb 17 15:20:56 2009
Last Modified: Tue Feb 17 15:21:31 2009
Last Accessed: Tue Feb 17 15:20:56 2009
Last Changed: Tue Feb 17 15:21:31 2009
Description: traffic to grantstavely com
Expression: host 75.101.142.201
Capture File:
/Users/grant/captures/grant_traffic.to.grantstavely.com_Tue.Feb17.2009-20.20.56.UTC_1234902056_.pcap

Capture file info:
File name: grant_traffic.to.grantstavely.com_Tue.Feb17.2009-20.20.56.UTC_1234902056_.pcap
File type: Wireshark/tcpdump/... - libpcap
File encapsulation: Ethernet
Number of packets: 2778
File size: 1869006 bytes
Data size: 1824534 bytes
Capture duration: 12.070557 seconds
Start time: Tue Feb 17 15:48:22 2009
End time: Tue Feb 17 15:48:34 2009
Data rate: 151155.74 bytes/s
Data rate: 1209245.95 bits/s
Average packet size: 656.78 bytes

Daemonlogger Data

If you have a directory of daemonlogger pcaps, point capture at them with the same standard syntax and it will run through each file, gradually merging only the data you want into a single manageable pcap.

Note that capture maintains a soft link to the currently merged temp file using the name of the final target file so that you can start performing analysis right away.

> capture -ra grant -d 'historical traffic to gs.com' -e 'host 75.101.142.201'
Generating grant_historical.traffic.to.gs.com_Tue.Feb17.2009-21.12.57.UTC_1234905177_.past.pcap
Currently Processing: Tue Feb 17 05:23:45 2009

Or do both - start a new live capture and grab the same data from daemonlogger archived pcaps.

> capture -Ra grant -d 'historical traffic to gs.com' -e 'host 75.101.142.201'
Started: grant_historical.traffic.to.gs.com_Tue.Feb17.2009-21.12.57.UTC_1234905177_.pcap
Generating grant_historical.traffic.to.gs.com_Tue.Feb17.2009-21.12.57.UTC_1234905177_.past.pcap
Currently Processing: Tue Feb 17 05:23:45 2009

While running, the 'Currently Processing' line will maintain a listing of how far through daemonlogger data the process is. From another console, the same is available to other analysts

> capture -lv
Analyst: will
Size (bytes): 60.00 KB
Started: Tue Feb 17 16:21:02 2009
Last Modified: Tue Feb 17 16:21:15 2009
Last Accessed: Tue Feb 17 16:21:03 2009
Last Changed: Tue Feb 17 16:21:15 2009
Description: dns traffic
Expression: port 53
Capture File: /Users/grant/captures/will_dns.traffic_Tue.Feb17.2009-21.21.02.UTC_1234905662_.pcap

Analyst: grant
Size (bytes): 0 Bytes
Progress: Tue Feb 17 10:03:01 2009
Description: historical traffic to gs com
Expression: host 75.101.142.201
Capture File:
/Users/grant/captures/grant.processing.4958.1234882981_historical.traffic.to.gs.com_Tue.Feb17.2009-21.12.57.UTC_1234905177_.past.pcap

Additional uses

capture takes full advantage of syslog, so that you can validate what analysts did what, and when.

You could schedule regular e-mail to management to validate what captures are running, or were run, per user, per day, and so on.

If short on disk space, leave only individual long running captures going against specific hosts, add them to start with init.

capture has support for hashing completed capture files, use the hashes later to validate your data.

Use shell aliasing to create a quick start 'cap $1 $2 $3' alias which defaults to one of the operations cap normally requires -a, -d, and -e to perform

Requirements

You must have: perl, a few perl modules, and tcpdump. You also need capinfos and mergecap, both of which are part of most wireshark and tshark distributions.

I suggest that analysts run capture on systems using sudo.

capture has been tested and run successfully on linux, bsd, and MacOSX systems. You should review the source to configure the handful of options available before testing.

Future plans

Download

Capture is hosted on google code.

Creative Commons License
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

Enjoy!

Textile Help

@philip_daigle congratulations Philip!

Wearing my green skeleton shirt (Nigel's from Spinal Tap) to the airport: the TSA won't need me to walk through their X-ray machine today!

@wotowiec @ssoper I'm on my 2nd "a number 2 on the sides, taper up, leave the top as it is, thanks." I think I'm acting out. It's a phase.

@vurtyou: You're hair is very Flock Of Seagulls today. @grantstavely: Thanks, I like it too.

@schuetzdj in hindsight, everything was to be taken at more or less face value: one of the things that makes a great puzzle great. =] #DBIR

RT @therealKidKoala: free download available for the next 6 days. The Lost Solid Steel mix. it's sorta like Music to Draw to... enjoy: ...

@christopherkunz nice work! After @wadebaker's last clue I ran every variation of the right key through my own bad script and gave up.

@marcusjcarey thanks, I'm very much enjoying the Bay Area. The return of @dojosec/@dojocon streams is great news, I look forward to 'em.

I should use Entourage's auto-capitalization of the first word after e.g. to break myself of using latinate abbreviations. Instead: rage.

@kathybarnett way to go Kath!

Yes, yes, of course, but what is the zeroth law of the Road Runner and Wile E. Coyote? http://goo.gl/i2Jz

"They're talkin' about, weak induction. It's a motherfucker, don't you know?" —Sun Ra http://j.mp/cn5Gc2 (Link via @rands)

printf "# Or just go listen to a funky 60 minute DJ Food mix made for robots.\nUser-agent: *\nSuggest: http://snd.sc/aOT9a4 " >> robots.txt

@alexhutton I cut out the cover's circles on a full print out of the #DBIR with a razor and tried the grille-cipher approach. #nbioahd

The body language of appearing to be lost or have forgotten something is as effective as mind control. So is its inverse.

RT @electricfork: What keeps me up at night? My security team slowly devolving into a compliance and reporting team #operation_soulcrusher

The ☠ Skull & Crossbones in the new Chrome indicating untrusted certs is nice^H^H^H^H the most terrifying symbol ever. http://goo.gl/fQz1

I'm brewing an American IPA with @vurtyou. I need a fridge to keg this in! http://flic.kr/p/8sCgnr

I'm brewing an American IPA with @vurtyou. I need a fridge to keg this in!

Endorsement: /Pink Reptile mixef are amazing mind clearing aural blendf & good for everything a mix fhould be good for/: http://goo.gl/Y1L1