Noise

This post was syndicated from: SANS Internet Storm Center, InfoCON: green and was written by: SANS Internet Storm Center, InfoCON: green. Original post: at SANS Internet Storm Center, InfoCON: green

Theres been a lot of discussion about the recent stories on parsing firewall logs – Marks story at http://isc.sans.org/diary.html?storyid=8293 , Daniels story at http://isc.sans.org/diary.html?storyid=8347 , and Kyles at http://isc.sans.org/diary.html?storyid=8362 have covered a number of methods and tools for plumbing the depths of your firewall logs.

In these stories, its been stressed that theres gold in them there logs! Reviewing your logs is legally required under several regulatory frameworks, and just plain makes sense reviewing inbound and outbound traffic is an excellent way to find stuff being sent or received that shouldnt be happening, finding malware or finding violations of corporate policies.

But, you say, thats all great, but many firewall logs are over 500MB per day, and if you’re not a command line guru with grep, uniq, sort, awk or perl, what do you do? Or what if the firewall log output is just so much scrambled eggs to you? How are you supposed to plow through all that text and data for the few pearls that you can expect to find that might indicate a problem? For me, the answer is easy, use tools that summarize Netflow data. Netflow is a facility that is available on many network devices that examines all the traffic through the device interfaces, and summarizes it by source and destination IP address, as well as source and destination port and how much data was sent or received.. It then sends this summarized data to a server application called a Netflow Collector. Netflow is generally associated with Cisco gear, but there is an RFC equivalent in sFlow (RFC 3176) that is implemented by many other vendors, or a Juniper specific version in jFlow.

Continuing on, the Netflow collector then stashes this data into a database, and then gives you a nice web front-end to the data, allowing you to slice and dice the addresses and associated values in prepackaged reports, or do ad-hoc queries. So if you want to see why internet bandwidth was maxed out last Tuesday over lunch, who the culprit was and what they were doing, its a piece of easy!

It sounds complicated, but in practice its generally about 4-5 lines of config on the device (router, switch or firewall – check your documentation for specifics), and a GUI setup on the server. There are lots of Netflow Collector apps out there, I wont start the religious war of stating that one is better than another I use any one of 7 or 8 different ones, depending on which client Im working with that day.

Lets take a look at a typical lets review the firewall activity session that you might have as part of your daily routine. This data is from a client site where I set Netflow up last week, I was going through an orientation session with the client IT Team (which is also the Incident Handling team at this organization), as well as using the tool in response to widespread user complaints about internet performance issues.

Lets start at the TCP applications (aka sort data by TCP destination port)screen in this example were just looking at the data from the last hour, for the inside interface of the firewall.

On the face of it, all looks well, all the usual suspects are there, but lets dig a bit deeper lets take a closer look at SMTP.

The SMTP traffic looks pretty much as we expected lots and lots of mail being sent from the mail server ( 10.0.0.73 ). But hey whats that station 10.0.0.233? – should there be another SMTPsender? After some digging, it turns out we had a workstation using a personal POP/SMTP email client from work this was a clear violation of the Acceptable Use Policy at this organization.

Lets go back to the main screen, and dig into the TCP_App section, which is the bit bucket that this particular Netflow application puts things into when it doesnt recognize what the target tcp port is.

Jackpot! What we have here is a number of stations, all running peer-to-peer applications (each line is a different target ip address). This was no surprise two days after the Oscars, but this is another clear violation of this Organizations Acceptable Use Policy, and one of the best ways to introduce malware into the Organization as well. Not only that, it takes LOTS of bandwidth and LOTS of address translation resources (aka memory)at the firewall sessions like this can easily affect Internet performance for the entire corporation. Depending on the country, this might be a great way to get sued under copyright infringement as well !

Now lets look at the data a bit differently lets look at session totals over the last hour by IPaddress, sorted by volume.

Take a look at that first line thats a station on the inside, using an anonymizer proxy out on the internet (tcp/8080). OUCH thats someone who is not only violating policy, theyre knowingly trying to cloak their actions. Theyre also the heaviest user in the last hour. Again, were 2 days after the Oscars, so its no mystery what that 200mb session is all about. But on any other week, there would be a real chance of finding some call the cops type illegal activity going on with proxy sessions like this.
Needless to say, after this short exploration, we’re working on a egress filter for this firewall. The we trust our usersposition not only ignores the fact that even if you trust your users, trusting your users’ malware should be part of your business model, but as you can see from this, you can’t trust (all of)your users either.
You can see from this that using a good Netflow Collector application will give you a great window into the traffic transiting your firewall or router, pretty much as granular as you want to be. We collected all this data in about 10 minutes, running a tutorial for the IT group at the same time. I still use grep, awk and the rest more than I use Netflow, but a good Netflow app can give you nice management style reports, historical queries into your router or firewall data and really granular analysis with almost no time investment. If you’re not a CLIperson, Netflow can go a long way towards getting you really deep into your firewall activity.

=============== Rob VandenBrink, Metafore ==============