From frank at knobbe.us Thu Oct 16 13:00:12 2008 From: frank at knobbe.us (Frank Knobbe) Date: Thu, 16 Oct 2008 12:00:12 -0500 Subject: [Discussion] Congrats Message-ID: <1224176412.32692.9.camel@localhost> Hey guys, congrats on the funding. I have plans and schematics (sort of) for a distributed IP filter device that is capable of blocking billions of IP addresses without much performance impact (near wire-speed). It's a two-tiered setup with a filter engine/device and a controller, which, in Snortsam-fashion, can be linked to allow blocking in a cloud. The device (in bridge-mode) would be able to filter up to a couple billion hostile IP addresses. My thought was that this would be the perfect border defense for large entities like the US Government (protect all departments, all entry points). Would you be interested in discussing the plans and possibly work with me on creating such a beast? Cheers, Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/8cc81c89/attachment.bin From aludwig at packetspy.com Thu Oct 16 13:18:42 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Thu, 16 Oct 2008 13:18:42 -0400 Subject: [Discussion] Congrats In-Reply-To: <1224176412.32692.9.camel@localhost> References: <1224176412.32692.9.camel@localhost> Message-ID: <48F77772.1010203@packetspy.com> I am assuming fpga's? (oh and howdy folks!) Andre Ludwig Frank Knobbe wrote: > Hey guys, > > congrats on the funding. > > I have plans and schematics (sort of) for a distributed IP filter device > that is capable of blocking billions of IP addresses without much > performance impact (near wire-speed). It's a two-tiered setup with a > filter engine/device and a controller, which, in Snortsam-fashion, can > be linked to allow blocking in a cloud. The device (in bridge-mode) > would be able to filter up to a couple billion hostile IP addresses. > > My thought was that this would be the perfect border defense for large > entities like the US Government (protect all departments, all entry > points). Would you be interested in discussing the plans and possibly > work with me on creating such a beast? > > Cheers, > Frank > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From frank at knobbe.us Thu Oct 16 13:21:54 2008 From: frank at knobbe.us (Frank Knobbe) Date: Thu, 16 Oct 2008 12:21:54 -0500 Subject: [Discussion] Congrats In-Reply-To: <48F77772.1010203@packetspy.com> References: <1224176412.32692.9.camel@localhost> <48F77772.1010203@packetspy.com> Message-ID: <1224177714.32692.22.camel@localhost> On Thu, 2008-10-16 at 13:18 -0400, Andre Ludwig wrote: > I am assuming fpga's? Nope. I don't think that's really necessary, although it would help of course :) -Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/557b5300/attachment.bin From aludwig at packetspy.com Thu Oct 16 13:32:13 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Thu, 16 Oct 2008 13:32:13 -0400 Subject: [Discussion] Congrats In-Reply-To: <1224177714.32692.22.camel@localhost> References: <1224176412.32692.9.camel@localhost> <48F77772.1010203@packetspy.com> <1224177714.32692.22.camel@localhost> Message-ID: <48F77A9D.8050505@packetspy.com> Just seems like a ton of data to have to access at wirespeed to process and parse (esp with 10ge around the corner). Not to mention the massive infrastructure that would have to be built out on the backend to feed/qualify/verify proper data to use. (but everyone i have seen attached to this list should understand that) Andre Ludwig Frank Knobbe wrote: > On Thu, 2008-10-16 at 13:18 -0400, Andre Ludwig wrote: > >> I am assuming fpga's? >> > > Nope. I don't think that's really necessary, although it would help of > course :) > > -Frank > > From frank at knobbe.us Thu Oct 16 13:34:46 2008 From: frank at knobbe.us (Frank Knobbe) Date: Thu, 16 Oct 2008 12:34:46 -0500 Subject: [Discussion] Congrats In-Reply-To: <48F77A9D.8050505@packetspy.com> References: <1224176412.32692.9.camel@localhost> <48F77772.1010203@packetspy.com> <1224177714.32692.22.camel@localhost> <48F77A9D.8050505@packetspy.com> Message-ID: <1224178486.32692.25.camel@localhost> On Thu, 2008-10-16 at 13:32 -0400, Andre Ludwig wrote: > Just seems like a ton of data to have to access at wirespeed to process > and parse (esp with 10ge around the corner). Depends :) It's not like a traditional firewall. It's taking a different approach. Simpler but faster. Forget about hashtables :) -Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/b7a5c148/attachment.bin From rick at support-intelligence.com Thu Oct 16 13:49:06 2008 From: rick at support-intelligence.com (Rick Wesson) Date: Thu, 16 Oct 2008 17:49:06 +0000 Subject: [Discussion] how can i help Message-ID: <48F77E92.4090607@support-intelligence.com> I liked the idea, what can i do to lend a hand. -rick From jives at security.berkeley.edu Thu Oct 16 14:26:43 2008 From: jives at security.berkeley.edu (John Ives) Date: Thu, 16 Oct 2008 11:26:43 -0700 Subject: [Discussion] Where to start In-Reply-To: <48F77E92.4090607@support-intelligence.com> References: <48F77E92.4090607@support-intelligence.com> Message-ID: <48F78763.5030008@security.berkeley.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Not knowing much about this project as it was written up for DHS, I would think that early on there should be a discussion about what works and what doesn't work with the current selection of IDS/IPS products. At about the same time I would think we should start brainstorming what our ideal IDS solutions would look like/contain. As for the fpga, discussion, I would think an ideal solution is one that can work on commodity hardware, but can be easily adapted to fpga's when needing to deal with high bandwidth areas where commodity starts to fail. John - -- - ------------------------------------------------------------------------- John Ives Phone (510) 642-7773 System & Network Security Cell (510) 229-8676 University of California, Berkeley - ------------------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEcBAEBAgAGBQJI94djAAoJEJkidK6qbywsVZcH/0S4c80zID67hGXfiPaGMEAR /byo1rQXzoVRUzQbWagfmOuVMc7cu9f4g6HeOZQjcfZCMs+Cv9jyqp3/d62qa4eD aG3aBXL+hNlJQEOa7uIgQTLegGZkaUvTlLeYsMV6ZG6BzaD4h8cBtd2jICUTmAB6 ZeBplyQlumyhjoNLnIBdD2FUWXUAAjtWa2HQhVBp8m4v8Ejq+DjrHlt1PzD+AeXk hWt7sjPsfyyKBettV97jps+eIkfeoMUq8vUyZEXt9LdYodNaIYoJ4FX2Z7AVWhyz Jymyhpk1TnjQsbQqCe2T27I3sOIozvLte1eAT4mqXXHRjEs3PyocDwSwiRxkj7M= =EFLf -----END PGP SIGNATURE----- From frank at knobbe.us Thu Oct 16 14:52:18 2008 From: frank at knobbe.us (Frank Knobbe) Date: Thu, 16 Oct 2008 13:52:18 -0500 Subject: [Discussion] Where to start In-Reply-To: <48F78763.5030008@security.berkeley.edu> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> Message-ID: <1224183138.3837.5.camel@localhost> On Thu, 2008-10-16 at 11:26 -0700, John Ives wrote: > As for the fpga, discussion, I would think an ideal solution is one that > can work on commodity hardware, but can be easily adapted to fpga's when > needing to deal with high bandwidth areas where commodity starts to fail. It is. No FPGA's in my idea. I never written a formal proposal outlining the specs. It's all in my head at the moment (and has been there for a while...*sigh*). I'll put something together when I get some time, though I thought perhaps a discussion with a few folks in person might be a better way to see if this seed could grow or would rot. Besides, Matt still owes my a beer anyway... (*hint*hint* ;) Cheers, Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/b6536479/attachment.bin From aludwig at packetspy.com Thu Oct 16 15:40:10 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Thu, 16 Oct 2008 15:40:10 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F78763.5030008@security.berkeley.edu> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> Message-ID: <48F7989A.1030306@packetspy.com> So lets take a look at why IDS in its current form sucks. 1. Signature matching has its use but can not be an end all for detection of attacks. (easy to bypass, layer 7 hurts, encryption hurts matching, etc) a. Signatures infer that you know what the attack looks like, this doesnt help much b. Signatures can change over time, they require lots of administrative overhead and testing before, during, and even after use 2. Performance and wire speed is always going to be an issue (as it is a safe assumption that it will continue to increase over time), this becomes a problem with inline (IPS). 3. Inability of vendors/producers of IDS technology to keep ahead of the curve (be it signature awareness, or be it protocol parsers, etc) 4. Anomaly detection based technologies can be effective but on their own they tend to be more of a burden then a help. And the single largest issue IMHO, is the lack of infrastructure to properly correlate and analyze data to produce actionable intelligence. This intelligence can then be piped to various types of "technical controls" to be used as part of the overall security architecture. Granted this isn't necessarily a weakness in IDS as much as it is a weakness in the approach the industry has taken. Please feel free to add/expand on these thoughts or introduce other thoughts to this discussion. (this thread is of course in response to John ives) Andre Ludwig From david.glosser at gmail.com Thu Oct 16 19:56:54 2008 From: david.glosser at gmail.com (David Glosser) Date: Thu, 16 Oct 2008 19:56:54 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F7989A.1030306@packetspy.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: Matt, this begs for a wiki or something once you gather enough information. -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). -ways to get alerts for fast-flux and other things where current sigs are prone to FPs On Thu, Oct 16, 2008 at 3:40 PM, Andre Ludwig wrote: > So lets take a look at why IDS in its current form sucks. > > 1. Signature matching has its use but can not be an end all for > detection of attacks. (easy to bypass, layer 7 hurts, encryption hurts > matching, etc) > a. Signatures infer that you know what the attack looks like, > this doesnt help much > b. Signatures can change over time, they require lots of > administrative overhead and testing before, during, and even after use > 2. Performance and wire speed is always going to be an issue (as it is > a safe assumption that it will continue to increase over time), this > becomes a problem with inline (IPS). > 3. Inability of vendors/producers of IDS technology to keep ahead of > the curve (be it signature awareness, or be it protocol parsers, etc) > 4. Anomaly detection based technologies can be effective but on their > own they tend to be more of a burden then a help. > > > > And the single largest issue IMHO, is the lack of infrastructure to > properly correlate and analyze data to produce actionable > intelligence. This intelligence can then be piped to various types of > "technical controls" to be used as part of the overall security > architecture. Granted this isn't necessarily a weakness in IDS as much > as it is a weakness in the approach the industry has taken. > > Please feel free to add/expand on these thoughts or introduce other > thoughts to this discussion. (this thread is of course in response to > John ives) > > Andre Ludwig > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From jonkman at jonkmans.com Thu Oct 16 20:12:05 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Thu, 16 Oct 2008 20:12:05 -0400 Subject: [Discussion] Welcome! Message-ID: <48F7D855.9020500@jonkmans.com> Welcome everyone to the open info sec foundation discussion list. I think we've seen somewhere near record one day mailing list signup rates. I take that to mean this project is something more than just a few of us are eager to see. So, to kick start discussions, we have a few goals to make initially here. First, the most important thing: 1. A mascot Gotta have a mascot. The best suggestion we've seen to date I think is an evil looking beaver. The analogy being that the new engine is here to dam up the flood of information and make it useful. In the form of hydroelectric power I guess. Will start a new thread for feature issues. Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jives at security.berkeley.edu Thu Oct 16 20:42:30 2008 From: jives at security.berkeley.edu (John Ives) Date: Thu, 16 Oct 2008 17:42:30 -0700 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F7989A.1030306@packetspy.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: <48F7DF76.5020405@security.berkeley.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 While I can't really argue with the basics of anything you say here I am going to put in my two cents anyway. Andre Ludwig wrote: > 1. Signature matching has its use but can not be an end all for > detection of attacks. (easy to bypass, layer 7 hurts, encryption hurts > matching, etc) > a. Signatures infer that you know what the attack looks like, > this doesnt help much > b. Signatures can change over time, they require lots of > administrative overhead and testing before, during, and even after use Your right, however I will point out that 1a and 1b can be handled, to a degree, by having signatures that don't look at specific attacks but look at the vulnerability being attacked. If the problem is that product A has a buffer overflow condition that and can only accepted 64 bytes of data before its buffer is filled, look for data at byte 65 to detect the attack. Unfortunately, from the network, it can be hard to tell if what was attacked was actually running that vulnerability without a comprehensive vulnerability assessment program. > 2. Performance and wire speed is always going to be an issue (as it is > a safe assumption that it will continue to increase over time), this > becomes a problem with inline (IPS). This is true but in my ideal world the IDS infrastructure would be clustered. We are actually using a product now that distributes traffic by netblocks to lower the actual bandwidth each box sees to a manageable level, effectively making a pseudo-cluster. > 3. Inability of vendors/producers of IDS technology to keep ahead of > the curve (be it signature awareness, or be it protocol parsers, etc) Your probably preaching to the choir on this one, but, to be fair I think security is one area where I doubt the good guys are ever going win (at least not as long as we are so adamantly focused on defense). The best we can hope for, baring some fundamental shift in the landscape, is a tie. I say this not to be defeatist, but because security is a zero sum game. The defenders only have to make one mistake to loose, while the attackers only need to get it right once to win. With that in mind, I would say the best hope we have is to start playing more offense, but at this point I don't know best to accomplish this without breaking laws, etc. But, I digress, because hearing my somewhat depressing view of the state of security today is not why everyone joined this list. Also, it doesn't do anything to move along a conversation about IDS. :) > 4. Anomaly detection based technologies can be effective but on their > own they tend to be more of a burden then a help. I've found that many times anomaly detection is best when used with other detection mechanisms. > And the single largest issue IMHO, is the lack of infrastructure to > properly correlate and analyze data to produce actionable > intelligence. This intelligence can then be piped to various types of > "technical controls" to be used as part of the overall security > architecture. Granted this isn't necessarily a weakness in IDS as much > as it is a weakness in the approach the industry has taken. Damn Straight! I wonder how many of us have had to write our own scripts/programs to help us parse and use the data we get from all of the security tools we run. John - -- - ------------------------------------------------------------------------- John Ives Phone (510) 642-7773 System & Network Security Cell (510) 229-8676 University of California, Berkeley - ------------------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEcBAEBAgAGBQJI9992AAoJEJkidK6qbywslbAH/0iFF3HuAx4q1QC/NBHl3MOP E9UmTNxXhcA4rVyP7KRftCIWBzdpErmL0jpEaFgDReY4NoTmm2XjepvdWdePCZwx tzrou71cWotXRz7B/GWYQavJDePlYj0d6Rd3NmKSbeWnLUIPOZFvWlppCMHV1M1C vf0HP5PptCRBqtPeoLHO+wljxUYmwpA2IgGr4jmCL8hMRnOgdmhMh/dIWEnf+xQM Nc8/I0F02viiFqv7uFQ8bdlCnO0HjbV7/aY5AKb/oCRvI00AUTQkBbPbG2h87te5 BCd0RhOSQ91nOcSHsWtnZ5wpOxMH/6VcRdQUiEXn1QjUQ7AaaLYApo4U9xy0oAk= =8Jfd -----END PGP SIGNATURE----- From jonkman at jonkmans.com Thu Oct 16 21:00:32 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Thu, 16 Oct 2008 21:00:32 -0400 Subject: [Discussion] Features Message-ID: <48F7E3B0.20002@jonkmans.com> Here's the big thread. And don't be afraid to start sub-threads for specifics here. The features we want to go after here are the primary reason we sought this funding and are taking this challenge on. Existing stuff works, but there's SO much more we could be doing by looking past traditional ips strengths. The challenge is that those things aren't conducive to making a commercial product with millions invested in development. No one can take this risk now, so we're going this route to make it happen. We have information about bad guys, bad places, and bad patterns. Lots of it, terabytes of it. We've got gigs of data about bad stuff in the sandnet at emerging threats alone. But most of that we can't effectively act upon. We can't give huge lists of bad IPs to most tools, we can't feed behavior patterns to existing tools, we can't share scan data globally, etc. So here we are. I have things I wish I could do, you have things you wish you could do, over the next couple of months we aim to get to the core set of the most important things that most of us want to be able to do. Then we'll go after it. So here's my wish list: 1. Native multithreading. Not each preprocessor or post processor can go to a thread, but each stream can take a thread. Think apache. More servers = more requests served. THe complications of sharing state between them and the like is a challenge, but solvable. 2. IP Reputation Sharing I want to feed these gigs of data I have and other projects have into my security devices and let it use that data to make smarter decisions. IP reputation isn't a new concept, but applying it in realtime will be a challenge. But this also opens us up to the possibility of sharing reputation data between ourselves. Imagine clouds of peer organizations sharing ip reputation between their security devices. Each benefits from teh data gained and contributes back what they encounter. All organizations become more safe. Then imagine organizations that collect this data for a living. We have an avenue for this data to be more commercially viable. 3. Native ipv6 Of course. No brainer there. 4. Native Hardware acceleration support There are a number of hardware acceleration technologies that could be more effectively built into the engine from the start, versus the back-asswards reverse engineering we have to do now to effectively accelerate. 5. Scoring Spam-assassin style point scoring. This would go a long way to eliminating false positives. The absolutely sure 100% guaranteed true positive rules of course would still hit. But the ones that are wrong as often as right could be given a score, say a half a point. If something else happens from that host within a certain timeframe that pushes that over a threshold then all of these alerts come back and can be acted upon with more confidence they're real. Complicated, but worthwhile. OK, those are my initial wish list items. Who has more? What else should we do? Any problems with the above? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From rullric at utah.gov Thu Oct 16 21:12:42 2008 From: rullric at utah.gov (Richard Ullrich) Date: Thu, 16 Oct 2008 19:12:42 -0600 Subject: [Discussion] [Mascot] Message-ID: <48F79231.8924.00FD.0@utah.gov> Outta be a Wolf... big, hungry, sneaky, naughty, and fast... ready to eat the bad guys/packets up... Maybe even a Lycanthrope! Ok, Ok... sorry - got carried away... Richard Ullrich Intrusion Detection Analyst (aka 'Xyberwolf') From david.glosser at gmail.com Thu Oct 16 21:15:38 2008 From: david.glosser at gmail.com (David Glosser) Date: Thu, 16 Oct 2008 21:15:38 -0400 Subject: [Discussion] Welcome! In-Reply-To: <48F7D855.9020500@jonkmans.com> References: <48F7D855.9020500@jonkmans.com> Message-ID: Somehow I'd like this guy in it ( tardigrades): http://www.marsanomalyresearch.com/evidence-reports/2004/078/mars-organism-survivor.htm or http://www.msnbc.msn.com/id/26613502/ They exist in places we would consider as exceptionally hostile, they are resistant to hard environments, radiation, 194 F (90 C)and -321 F (-196 C) temperatures, and can seem to repair their own damaged DNA. They are only 0.1 to 1.0 mm in length, about the size of an IP packet :) On Thu, Oct 16, 2008 at 8:12 PM, Matt Jonkman wrote: > Welcome everyone to the open info sec foundation discussion list. I > think we've seen somewhere near record one day mailing list signup > rates. I take that to mean this project is something more than just a > few of us are eager to see. > > So, to kick start discussions, we have a few goals to make initially > here. First, the most important thing: > > 1. A mascot > Gotta have a mascot. The best suggestion we've seen to date I think is > an evil looking beaver. The analogy being that the new engine is here to > dam up the flood of information and make it useful. In the form of > hydroelectric power I guess. > > Will start a new thread for feature issues. > > Matt > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/f6897622/attachment.html From jonkman at jonkmans.com Thu Oct 16 21:23:11 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Thu, 16 Oct 2008 21:23:11 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F7DF76.5020405@security.berkeley.edu> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F7DF76.5020405@security.berkeley.edu> Message-ID: <48F7E8FF.2090306@jonkmans.com> John Ives wrote: >> 1. Signature matching has its use but can not be an end all for >> detection of attacks. (easy to bypass, layer 7 hurts, encryption hurts > Your right, however I will point out that 1a and 1b can be handled, to a > degree, by having signatures that don't look at specific attacks but > look at the vulnerability being attacked. If the problem is that > product A has a buffer overflow condition that and can only accepted 64 > bytes of data before its buffer is filled, look for data at byte 65 to > detect the attack. Unfortunately, from the network, it can be hard to > tell if what was attacked was actually running that vulnerability > without a comprehensive vulnerability assessment program. For vulnerabilities that's true. But if you run the ET rules I think you'll find as much or more value in our detection of post infection activity. These aren't vulnerabilities, for the most part it's hostile traffic trying really hard to look like legit traffic. Here's where scoring and such come into play. We can write signatures that will only push a bad actor over the alerting threshold if they do all of the things then we alert. I think it's a misconception that IDS is for vulnerabilities only. Known or unknown. I think it's value lies as much or more in policy enforcement, malware/infection detection, etc. But PLEASE let's not start the "IDS isn't meant for policy enforcement" war. It's not meant for it, but it's damn good at it, so it's going to be used for it. :) >> 3. Inability of vendors/producers of IDS technology to keep ahead of >> the curve (be it signature awareness, or be it protocol parsers, etc) > > Your probably preaching to the choir on this one, but, to be fair I > think security is one area where I doubt the good guys are ever going > win (at least not as long as we are so adamantly focused on defense). > The best we can hope for, baring some fundamental shift in the > landscape, is a tie. I say this not to be defeatist, but because > security is a zero sum game. The defenders only have to make one > mistake to loose, while the attackers only need to get it right once to > win. With that in mind, I would say the best hope we have is to start > playing more offense, but at this point I don't know best to accomplish > this without breaking laws, etc. But, I digress, because hearing my > somewhat depressing view of the state of security today is not why > everyone joined this list. Also, it doesn't do anything to move along a > conversation about IDS. :) > IP Reputation I think will give us an advantage here. They have to build botnets now to get spam delivered before they are blacklisted, then they toss the bot and infect more. That had a fundamental shift on how spammers operate. I think we can do the same for a lot of badness. If every site can know about and block a known bad IP, and do so in near realtime, then reconnaissance becomes something you need a botnet for, and becomes much more difficult. And ideally you'd only get one shot to make an attack and you're blacklisted. On shot against ANY participating network, ont one shot PER network. >> 4. Anomaly detection based technologies can be effective but on their >> own they tend to be more of a burden then a help. > > I've found that many times anomaly detection is best when used with > other detection mechanisms. > Scoring and thresholds to alert!! >> And the single largest issue IMHO, is the lack of infrastructure to >> properly correlate and analyze data to produce actionable >> intelligence. This intelligence can then be piped to various types of >> "technical controls" to be used as part of the overall security >> architecture. Granted this isn't necessarily a weakness in IDS as much >> as it is a weakness in the approach the industry has taken. > > Damn Straight! I wonder how many of us have had to write our own > scripts/programs to help us parse and use the data we get from all of > the security tools we run. Agreed! What could we do at the engine level to make this more useful? We hope to have ip reputation and hit scoring. Those could go a long ways in prioritizing. What else? matt _______________________________________________ Discussion mailing list Discussion at openinfosecfoundation.org http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jeremy at sudosecure.net Thu Oct 16 22:33:13 2008 From: jeremy at sudosecure.net (Jeremy) Date: Thu, 16 Oct 2008 21:33:13 -0500 Subject: [Discussion] Features In-Reply-To: <48F7E3B0.20002@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> Message-ID: <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> Comments are inline but first I would like to add an item: 6. Network intelligence through passive identification. What I mean by this is like Snorts RNA, NMAP, and p0s fingerprinting to dynamically update inventory data for correlation engine usage. I think we all know how great it would be to have this data which could dynamically or auto magically configure/reconfigure signatures and correlation engines to align with applications and operating systems to truly substantiate the risk of attacks and/or threats. Having the availability of correlated network intelligence could make tremendous headway in the world of anomaly detection i.e. a windows xp home computer correlated with network flow data and port data could easily be alerted on for any outgoing port 25 connections, as they are most likely the newest member of a spam bot network. Obviously this is just a trivial example but it still is not a trivial task to pull off even with the most advanced SIM. This would also allow for operational performance enhancements by providing context regarding the network they operate in. The dynamic configuration and reconfiguration of network devices to ignore threats that are not applicable would definitely enhance performance by reducing overhead caused by signatures and other security measures. On Thu, Oct 16, 2008 at 8:00 PM, Matt Jonkman wrote: > Here's the big thread. And don't be afraid to start sub-threads for > specifics here. > > The features we want to go after here are the primary reason we sought > this funding and are taking this challenge on. Existing stuff works, but > there's SO much more we could be doing by looking past traditional ips > strengths. The challenge is that those things aren't conducive to making > a commercial product with millions invested in development. No one can > take this risk now, so we're going this route to make it happen. > > We have information about bad guys, bad places, and bad patterns. Lots > of it, terabytes of it. We've got gigs of data about bad stuff in the > sandnet at emerging threats alone. But most of that we can't effectively > act upon. We can't give huge lists of bad IPs to most tools, we can't > feed behavior patterns to existing tools, we can't share scan data > globally, etc. > > So here we are. I have things I wish I could do, you have things you > wish you could do, over the next couple of months we aim to get to the > core set of the most important things that most of us want to be able to > do. Then we'll go after it. > > So here's my wish list: > > 1. Native multithreading. > Not each preprocessor or post processor can go to a thread, but each > stream can take a thread. Think apache. More servers = more requests > served. THe complications of sharing state between them and the like is > a challenge, but solvable. I think you are dead on with this one, you have an advantage in that no code has been written so mutithreading doesn't have to be worked into the current code base. I think in many cases it is harder to convert single threaded applications into multithreaded apps than it would be to just rewrite applications with multithreading. > > > 2. IP Reputation Sharing > I want to feed these gigs of data I have and other projects have into my > security devices and let it use that data to make smarter decisions. IP > reputation isn't a new concept, but applying it in realtime will be a > challenge. But this also opens us up to the possibility of sharing > reputation data between ourselves. > > Imagine clouds of peer organizations sharing ip reputation between their > security devices. Each benefits from teh data gained and contributes > back what they encounter. All organizations become more safe. > > Then imagine organizations that collect this data for a living. We have > an avenue for this data to be more commercially viable. > I love this idea as well, but would not limit it to just IP data. The more data that is shared and collaborated on in a scoring system like you mention further down in your post the better in my opinion. I would like to see a correlation engine (ie you scoring scheme) factor in shared DNS data, Domain Reputations, ASN Reputations, ISP Reputations, Port Statistics, Protocol Statistics, Threat Indexes/Activities and statisitcs, vulnerability probabilities and activities, Network/Application/OS awareness, and GEO type statistics. I know this is kind of a big dream to have widely dispersed geographical networks sharing statistical data in real time that could be correlated to ensure even the smallest networks obtained the intelligence level and threat visibilities/awareness levels of the largest networks in the cloud... Obviously this is no trivial task! What I do see is if this idea/dream became reality, you would see AI implemented into IDS', Firewalls, Routers, Servers, Applications, and anything else that could listen for these correlated data statistics and adjust their configurations based off live data. Imagine an IDS/IPS that could auto magically load and unload specific rules to meet the threats seen on other networks in anticipation of attacks to come that may not have reached them yet or routers that could create null routes for troubled/bad IP subnets based off of data intelligence seen by this super cloud. Even web servers that could create mod rewrite rules or acls to prevent exploit bots from delivering their drive by badness, which I am sure have all grown to love in the last six months... I see limitless possibilities, but reality may play a factor here ;) > > 3. Native ipv6 > Of course. No brainer there. > > > 4. Native Hardware acceleration support > There are a number of hardware acceleration technologies that could be > more effectively built into the engine from the start, versus the > back-asswards reverse engineering we have to do now to effectively > accelerate. > Really cool idea and would like to see this go forward, but again not a trivial mission. With hardware acceleration you must factor in cost. Why do we spend hours hacking the kernel, libpcap, and applications we use? Easy cause most of us are under budget constraints and can not afford to implement many of these hardware accelerators... I know we are going through the 10 Gb issues, and well buying a Bivio and Network General sniffer to do this is very painful for us and will break the bank.... that being said I welcome competitors in this area as I believe competition could cause some of these hardware acceleration technologies to fall in price. > > 5. Scoring > Spam-assassin style point scoring. This would go a long way to > eliminating false positives. The absolutely sure 100% guaranteed true > positive rules of course would still hit. But the ones that are wrong as > often as right could be given a score, say a half a point. If something > else happens from that host within a certain timeframe that pushes that > over a threshold then all of these alerts come back and can be acted > upon with more confidence they're real. Complicated, but worthwhile. > > > > OK, those are my initial wish list items. Who has more? What else should > we do? Any problems with the above? > > Matt > > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From david.glosser at gmail.com Thu Oct 16 22:50:53 2008 From: david.glosser at gmail.com (David Glosser) Date: Thu, 16 Oct 2008 22:50:53 -0400 Subject: [Discussion] Features In-Reply-To: <48F7E3B0.20002@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> Message-ID: what is the "audience" for this? Will it be only for corporations? Will some of the "rules" and logic somehow be available to the end-user (ie desktop app, browser plug-in, etc?) scoring sounds great, also IP/domain reputation. On Thu, Oct 16, 2008 at 9:00 PM, Matt Jonkman wrote: > Here's the big thread. And don't be afraid to start sub-threads for > specifics here. > > The features we want to go after here are the primary reason we sought > this funding and are taking this challenge on. Existing stuff works, but > there's SO much more we could be doing by looking past traditional ips > strengths. The challenge is that those things aren't conducive to making > a commercial product with millions invested in development. No one can > take this risk now, so we're going this route to make it happen. > > We have information about bad guys, bad places, and bad patterns. Lots > of it, terabytes of it. We've got gigs of data about bad stuff in the > sandnet at emerging threats alone. But most of that we can't effectively > act upon. We can't give huge lists of bad IPs to most tools, we can't > feed behavior patterns to existing tools, we can't share scan data > globally, etc. > > So here we are. I have things I wish I could do, you have things you > wish you could do, over the next couple of months we aim to get to the > core set of the most important things that most of us want to be able to > do. Then we'll go after it. > > So here's my wish list: > > 1. Native multithreading. > Not each preprocessor or post processor can go to a thread, but each > stream can take a thread. Think apache. More servers = more requests > served. THe complications of sharing state between them and the like is > a challenge, but solvable. > > > 2. IP Reputation Sharing > I want to feed these gigs of data I have and other projects have into my > security devices and let it use that data to make smarter decisions. IP > reputation isn't a new concept, but applying it in realtime will be a > challenge. But this also opens us up to the possibility of sharing > reputation data between ourselves. > > Imagine clouds of peer organizations sharing ip reputation between their > security devices. Each benefits from teh data gained and contributes > back what they encounter. All organizations become more safe. > > Then imagine organizations that collect this data for a living. We have > an avenue for this data to be more commercially viable. > > > 3. Native ipv6 > Of course. No brainer there. > > > 4. Native Hardware acceleration support > There are a number of hardware acceleration technologies that could be > more effectively built into the engine from the start, versus the > back-asswards reverse engineering we have to do now to effectively > accelerate. > > > 5. Scoring > Spam-assassin style point scoring. This would go a long way to > eliminating false positives. The absolutely sure 100% guaranteed true > positive rules of course would still hit. But the ones that are wrong as > often as right could be given a score, say a half a point. If something > else happens from that host within a certain timeframe that pushes that > over a threshold then all of these alerts come back and can be acted > upon with more confidence they're real. Complicated, but worthwhile. > > > > OK, those are my initial wish list items. Who has more? What else should > we do? Any problems with the above? > > Matt > > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/9573b65c/attachment.html From aludwig at packetspy.com Thu Oct 16 23:00:57 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Thu, 16 Oct 2008 23:00:57 -0400 Subject: [Discussion] Features In-Reply-To: <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> Message-ID: <48F7FFE9.3080508@packetspy.com> Since this is an "ideas" thread and not "comment on ideas" thread ill keep it short and blunt. I left a few things out of the "why ids sucks" thread on purpose, one of which i will address here. 7. Ability to look across packets in a stream, be it 3 packets before or 4 packets ahead. The ability to follow conditions of state that are triggered by packets before or after a "match" would be extremely valuable. (rpc comes to mind, as well as some application layer attacks) Ideally the ability to perform functions against packets (be it arithmetic, comparisons, or functions on "running variables" for a "attack detection set"). The concept here of course is to follow a tcp stream looking for precursory actions that are needed to finesse an endpoint into an exploitable situation. Be it forcing the endpoint to allocate a buffer of X size with XYZ rpc call, that is later exploited by calls to 3 other RPC calls with varying parameters. The critical key is to engineer a system that affords you the flexibility to effectively "emulate" and follow increasingly complex attack scenarios to the point of detecting actual attacks and not simply triggering a FP. My single largest frustration with snort (beyond the fact that i absolutely hate its limited ability to detect "proper attacks") is that you are effectively looking at one packet at a time. You do not have the ability to create per tcp stream variables that would allow you to compare against packets furthur down the stream. 8. Some sort of meta language needs to be created that easily and effectively can communicate any applications "state". Even if this means creating application specific "translation" modules that allow end users to effectively outline functions used by an application (and what they do). This of course would come in handy by allowing end users the ability to effectively "peer" inside the "mapped logic" of an application. This in turn (if done right) could aid in an analysts ability to alert and investigate malicious behavior of an application vs simply producing mind numbingly general "signatures' and hoping for the best. (xss alerts come to mind, as well as other web app attacks) Jeremy wrote: > Comments are inline but first I would like to add an item: > > 6. Network intelligence through passive identification. What I mean > by this is like Snorts RNA, NMAP, and p0s fingerprinting to > dynamically update inventory data for correlation engine usage. I > think we all know how great it would be to have this data which could > dynamically or auto magically configure/reconfigure signatures and > correlation engines to align with applications and operating systems > to truly substantiate the risk of attacks and/or threats. Having the > availability of correlated network intelligence could make tremendous > headway in the world of anomaly detection i.e. a windows xp home > computer correlated with network flow data and port data could easily > be alerted on for any outgoing port 25 connections, as they are most > likely the newest member of a spam bot network. Obviously this is > just a trivial example but it still is not a trivial task to pull off > even with the most advanced SIM. This would also allow for > operational performance enhancements by providing context regarding > the network they operate in. The dynamic configuration and > reconfiguration of network devices to ignore threats that are not > applicable would definitely enhance performance by reducing overhead > caused by signatures and other security measures. > > On Thu, Oct 16, 2008 at 8:00 PM, Matt Jonkman wrote: > >> Here's the big thread. And don't be afraid to start sub-threads for >> specifics here. >> >> The features we want to go after here are the primary reason we sought >> this funding and are taking this challenge on. Existing stuff works, but >> there's SO much more we could be doing by looking past traditional ips >> strengths. The challenge is that those things aren't conducive to making >> a commercial product with millions invested in development. No one can >> take this risk now, so we're going this route to make it happen. >> >> We have information about bad guys, bad places, and bad patterns. Lots >> of it, terabytes of it. We've got gigs of data about bad stuff in the >> sandnet at emerging threats alone. But most of that we can't effectively >> act upon. We can't give huge lists of bad IPs to most tools, we can't >> feed behavior patterns to existing tools, we can't share scan data >> globally, etc. >> >> So here we are. I have things I wish I could do, you have things you >> wish you could do, over the next couple of months we aim to get to the >> core set of the most important things that most of us want to be able to >> do. Then we'll go after it. >> >> So here's my wish list: >> >> 1. Native multithreading. >> Not each preprocessor or post processor can go to a thread, but each >> stream can take a thread. Think apache. More servers = more requests >> served. THe complications of sharing state between them and the like is >> a challenge, but solvable. >> > > I think you are dead on with this one, you have an advantage in that > no code has been written so mutithreading doesn't have to be worked > into the current code base. I think in many cases it is harder to > convert single threaded applications into multithreaded apps than it > would be to just rewrite applications with multithreading. > > >> 2. IP Reputation Sharing >> I want to feed these gigs of data I have and other projects have into my >> security devices and let it use that data to make smarter decisions. IP >> reputation isn't a new concept, but applying it in realtime will be a >> challenge. But this also opens us up to the possibility of sharing >> reputation data between ourselves. >> >> Imagine clouds of peer organizations sharing ip reputation between their >> security devices. Each benefits from teh data gained and contributes >> back what they encounter. All organizations become more safe. >> >> Then imagine organizations that collect this data for a living. We have >> an avenue for this data to be more commercially viable. >> >> > I love this idea as well, but would not limit it to just IP data. The > more data that is shared and collaborated on in a scoring system like > you mention further down in your post the better in my opinion. I > would like to see a correlation engine (ie you scoring scheme) factor > in shared DNS data, Domain Reputations, ASN Reputations, ISP > Reputations, Port Statistics, Protocol Statistics, Threat > Indexes/Activities and statisitcs, vulnerability probabilities and > activities, Network/Application/OS awareness, and GEO type statistics. > I know this is kind of a big dream to have widely dispersed > geographical networks sharing statistical data in real time that could > be correlated to ensure even the smallest networks obtained the > intelligence level and threat visibilities/awareness levels of the > largest networks in the cloud... Obviously this is no trivial task! > What I do see is if this idea/dream became reality, you would see AI > implemented into IDS', Firewalls, Routers, Servers, Applications, and > anything else that could listen for these correlated data statistics > and adjust their configurations based off live data. Imagine an > IDS/IPS that could auto magically load and unload specific rules to > meet the threats seen on other networks in anticipation of attacks to > come that may not have reached them yet or routers that could create > null routes for troubled/bad IP subnets based off of data intelligence > seen by this super cloud. Even web servers that could create mod > rewrite rules or acls to prevent exploit bots from delivering their > drive by badness, which I am sure have all grown to love in the last > six months... I see limitless possibilities, but reality may play a > factor here ;) > >> 3. Native ipv6 >> Of course. No brainer there. >> >> >> 4. Native Hardware acceleration support >> There are a number of hardware acceleration technologies that could be >> more effectively built into the engine from the start, versus the >> back-asswards reverse engineering we have to do now to effectively >> accelerate. >> >> > Really cool idea and would like to see this go forward, but again not > a trivial mission. With hardware acceleration you must factor in > cost. Why do we spend hours hacking the kernel, libpcap, and > applications we use? Easy cause most of us are under budget > constraints and can not afford to implement many of these hardware > accelerators... I know we are going through the 10 Gb issues, and > well buying a Bivio and Network General sniffer to do this is very > painful for us and will break the bank.... that being said I welcome > competitors in this area as I believe competition could cause some of > these hardware acceleration technologies to fall in price. > >> 5. Scoring >> Spam-assassin style point scoring. This would go a long way to >> eliminating false positives. The absolutely sure 100% guaranteed true >> positive rules of course would still hit. But the ones that are wrong as >> often as right could be given a score, say a half a point. If something >> else happens from that host within a certain timeframe that pushes that >> over a threshold then all of these alerts come back and can be acted >> upon with more confidence they're real. Complicated, but worthwhile. >> >> >> >> OK, those are my initial wish list items. Who has more? What else should >> we do? Any problems with the above? >> >> Matt >> >> >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From FDoane at lancope.com Thu Oct 16 23:05:08 2008 From: FDoane at lancope.com (Frank Doane) Date: Thu, 16 Oct 2008 23:05:08 -0400 Subject: [Discussion] proposed name for project Message-ID: jids3asim4nextgenecos -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081016/fac4b802/attachment.html From jives at security.berkeley.edu Fri Oct 17 01:23:36 2008 From: jives at security.berkeley.edu (John Ives) Date: Thu, 16 Oct 2008 22:23:36 -0700 Subject: [Discussion] Features In-Reply-To: <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> Message-ID: <48F82158.3020903@security.berkeley.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeremy wrote: > Comments are inline but first I would like to add an item: > > 6. Network intelligence through passive identification. What I mean > by this is like Snorts RNA, NMAP, and p0s fingerprinting to > dynamically update inventory data for correlation engine usage. I > think we all know how great it would be to have this data which could > dynamically or auto magically configure/reconfigure signatures and > correlation engines to align with applications and operating systems > to truly substantiate the risk of attacks and/or threats. Having the > availability of correlated network intelligence could make tremendous > headway in the world of anomaly detection i.e. a windows xp home > computer correlated with network flow data and port data could easily > be alerted on for any outgoing port 25 connections, as they are most > likely the newest member of a spam bot network. Obviously this is > just a trivial example but it still is not a trivial task to pull off > even with the most advanced SIM. This would also allow for > operational performance enhancements by providing context regarding > the network they operate in. The dynamic configuration and > reconfiguration of network devices to ignore threats that are not > applicable would definitely enhance performance by reducing overhead > caused by signatures and other security measures. If your really interested in finding out what is running on a desktop system passively I would suggest looking at the updates it attempts to download. With MS patches alone you can tell things like specific OS, version of Office and sometime even CPU. The Apple updater is rather detailed as well if you care to parse it enough. This is even before you look at the various applications like Acrobat, or AV products that update across the web. For things like Mozilla, which does its update over TLS, it goes to 'you are now up to date' page after every new version has been installed. It gets harder with the various UNIX's but even there, there are things you can tell (what distro is it getting updates for, does it download ports and if so what version, etc). John -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) iQEcBAEBAgAGBQJI+CFYAAoJEJkidK6qbyws/bQIAL/qe5nUerryYHVB/vr+evEB ynPqtqYXm12kn+cRpVoN2pe+I9+8/xEMsjpkHC7qgmP/5P3n46X555bMRmxyRrze 2FRYdGtVCcjlsRWAgmz46LwqAgA/97aefAm1b0P7NgkGCPM69jQ6H0TArmnB/xUQ HBejEFw/y9Yv+phojFnKePaFMUtati0wTu9ANRBvMISpm5C2N6jOUemxYm5cVlsd 0qWFc2Fhb7FHUmwjVuO5XaVL90YEdSaegl86i827qdiBMWxktc77ty3HeYPHA1Mg /6mBaTivCRVlJ5wFPX8pEztCtecBhhGAfQoKZM3Oo28G7zhvoSlMtYaHawJjqGo= =42IY -----END PGP SIGNATURE----- From jonkman at jonkmans.com Fri Oct 17 01:27:37 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 01:27:37 -0400 Subject: [Discussion] proposed name for project In-Reply-To: References: Message-ID: <48F82249.50602@jonkmans.com> Haha!! Not sure that'd look good on a t-shirt. :) matt Frank Doane wrote: > jids3asim4nextgenecos > > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jives at security.berkeley.edu Fri Oct 17 02:14:06 2008 From: jives at security.berkeley.edu (John Ives) Date: Thu, 16 Oct 2008 23:14:06 -0700 Subject: [Discussion] Features In-Reply-To: <48F7FFE9.3080508@packetspy.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F7FFE9.3080508@packetspy.com> Message-ID: <48F82D2E.1060402@security.berkeley.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 To throw in my own loose thoughts... 9. The ability to pull files out of the stream in real-time. e.g. If a user attempts to download a file named codec.exe pull a copy of that file from the tcp stream and send it to a AV/sandbox. If used with a sandbox it would mean that, in essence, each client on the network would become a sort of honeyclient, identifying malware during normal activity. (of course this is of particular interest to me since I am slowly building scripts to do something similar - though not in real time - using our existing IDS infrastructure and some of my own rules). 10. (an extension of Matt's #2 [reputation] and Andre's 8 [language to view app state] and my own #9) a language/protocol for it to interact with other security devices. Simple uses would be 'I saw X attempt to do Y to Z so add a firewall rule to stop it X' or 'File malware.exe in www.hacker.net/badstuff/ was identified as malware in the sandbox/AV system so drop any packets requesting it.' Obviously this is closely related to Matt's IP reputation scoring, but this is based upon immediate threat on a targeted site. This should be reciprocal also with something like OSSEC or mod_security being able to dynamically inject immediate information to the IDS/IPS since they can see the interaction at the end-point after decryption. John -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) iQEcBAEBAgAGBQJI+C0tAAoJEJkidK6qbyws5cIH/0/hJjwpeZUa3c88dGOWcTfa O9Zv8vpOkvq0RJGYFHT+9QKxGlrqk2dgngQiC9oz55MhllWePL94Ir067dj3baoJ TbgAUBMwNK7fbQhC0evvBzOiq7z6Lsei+EO3pBhUHpt7NKyokGEgTbnieR4/3JF6 pFOEFuv0dZURtXyShhZHzkaQO2oxP1p1kbvHqNaWSn2ljkMhpbyhP7pnZnTNucfJ tsoxibcOlud6+LEfTyeQ4Ajz7upXHarEb8U4hdTx095Y6ZmLHKsio5CMpnu0tmxW 3L0RKae+NpjMt9abaBvVqfiH24EJe6aBpIWPXlls0hl0rip5PG3tOYfJwqIJ7M0= =Aold -----END PGP SIGNATURE----- From joep.gommers at gmail.com Fri Oct 17 02:47:03 2008 From: joep.gommers at gmail.com (Joep Gommers) Date: Fri, 17 Oct 2008 08:47:03 +0200 Subject: [Discussion] [Fwd: Re: Features] Message-ID: <48F834E7.6080109@gmail.com> Hi List, John Ives wrote: > To throw in my own loose thoughts... > > 9. The ability to pull files out of the stream in real-time. e.g. If a > user attempts to download a file named codec.exe pull a copy of that > file from the tcp stream and send it to a AV/sandbox. If used with a > sandbox it would mean that, in essence, each client on the network would > become a sort of honeyclient, identifying malware during normal > activity. (of course this is of particular interest to me since I am > slowly building scripts to do something similar - though not in real > time - using our existing IDS infrastructure and some of my own rules). > Although I'll reply to this thread soon in more detail, I have to quickly underline that this would be a feature of great value. Ow, and nice to meet you all - my first post! Not just because of malcode collecting, but also that once sandboxed, this (slightly delayed) intelligence is extremely valueable in terms of intrusion detection on a more high-level event correlation/policy/compliance level. I would imagine this feature using signatures of sort, defining start-, endpoint and contextual pointers that could 'extract' raw data from streams - inside protocol on different layers (be it tcp/ftp/smtp) - and perform action on. Scan with AV engine, send to sendbox et cetera. Brainstorming, wonder if it would be possible then to save checksums of these extracted datasets, and mark as 'safe' after being scanned. A configuration option could block 'unscanned' or deemed 'unsafe' checksums. -- Best regards, Joep Gommers iSIGHT Partners +1 (571) 451-2007 +1 (703) 879-1681 Skype: jgommers -- Best regards, Joep Gommers +1 (571) 451-2007 +1 (703) 879-1681 Skype: jgommers From jamie.riden at gmail.com Fri Oct 17 04:13:17 2008 From: jamie.riden at gmail.com (Jamie Riden) Date: Fri, 17 Oct 2008 09:13:17 +0100 Subject: [Discussion] Features In-Reply-To: <48F82D2E.1060402@security.berkeley.edu> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F7FFE9.3080508@packetspy.com> <48F82D2E.1060402@security.berkeley.edu> Message-ID: <17b0fcab0810170113m465b089ehb55e2922ff1a5507@mail.gmail.com> 2008/10/17 John Ives : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > To throw in my own loose thoughts... > > 9. The ability to pull files out of the stream in real-time. e.g. If a > user attempts to download a file named codec.exe pull a copy of that > file from the tcp stream and send it to a AV/sandbox. If used with a > sandbox it would mean that, in essence, each client on the network would > become a sort of honeyclient, identifying malware during normal > activity. (of course this is of particular interest to me since I am > slowly building scripts to do something similar - though not in real > time - using our existing IDS infrastructure and some of my own rules). This would be very nice to have in rule syntax - I'm thinking of something similar to a triggered tcpxtract. "if we've seen alert X, then start capturing all executables that are downloaded from a particular host." cheers, Jamie ( tcpxtract allows you to define templates to recover things such as exe files from a tcpdump file - http://tcpxtract.sourceforge.net/ ) -- Jamie Riden / jamesr at europe.com / jamie at honeynet.org.uk UK Honeynet Project: http://www.ukhoneynet.org/ From jamie.riden at gmail.com Fri Oct 17 04:26:49 2008 From: jamie.riden at gmail.com (Jamie Riden) Date: Fri, 17 Oct 2008 09:26:49 +0100 Subject: [Discussion] Features In-Reply-To: <48F7E3B0.20002@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> Message-ID: <17b0fcab0810170126q5ea6e572o8c29c7c6290ef036@mail.gmail.com> 2008/10/17 Matt Jonkman : > > 2. IP Reputation Sharing > I want to feed these gigs of data I have and other projects have into my > security devices and let it use that data to make smarter decisions. IP > reputation isn't a new concept, but applying it in realtime will be a > challenge. But this also opens us up to the possibility of sharing > reputation data between ourselves. Got to be a little bit careful here - I've seen my DNS servers appear in dshield attack logs before, for entirely normal traffic. Another time someone promoted a random domain controller to be a stratum 1 NTP server and wondered why he was getting 'attacks' on port 123/udp. My point is, it could be quite easy to pollute the data set with false positives. > 5. Scoring > Spam-assassin style point scoring. This would go a long way to > eliminating false positives. The absolutely sure 100% guaranteed true > positive rules of course would still hit. But the ones that are wrong as > often as right could be given a score, say a half a point. If something > else happens from that host within a certain timeframe that pushes that > over a threshold then all of these alerts come back and can be acted > upon with more confidence they're real. Complicated, but worthwhile. I guess my mental model for snort rules includes a degree of belief that the alert is indicative of a real problem, plus an idea of severity, if the alert is a real problem and not a false positive. If I see a lot of traffic from 169.254/16, that's probably a genuine alert because some end user has a misconfigured system, but I don't really care too much about it. If I see a sasser FTP transfer outound, probably a genuine alert and I *really* care about that. And then there are the NOP sled alerts which are usually false positives, but if they are correct, they are important. So I'd prefer it if we could use two numbers to score alerts with - confidence and severity, so to speak. This approach might end up being quite similar to the IP reputation system in a lot of ways. cheers, Jamie -- Jamie Riden / jamesr at europe.com / jamie at honeynet.org.uk UK Honeynet Project: http://www.ukhoneynet.org/ From lists at inliniac.net Fri Oct 17 04:38:58 2008 From: lists at inliniac.net (Victor Julien) Date: Fri, 17 Oct 2008 10:38:58 +0200 Subject: [Discussion] Features In-Reply-To: <48F7FFE9.3080508@packetspy.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F7FFE9.3080508@packetspy.com> Message-ID: <48F84F22.7000305@inliniac.net> Hi everyone, Andre Ludwig wrote: > Since this is an "ideas" thread and not "comment on ideas" thread ill > keep it short and blunt. > > I left a few things out of the "why ids sucks" thread on purpose, one of > which i will address here. > > 7. Ability to look across packets in a stream, be it 3 packets before or > 4 packets ahead. The ability to follow conditions of state that are > triggered by packets before or after a "match" would be extremely > valuable. (rpc comes to mind, as well as some application layer > attacks) Ideally the ability to perform functions against packets (be > it arithmetic, comparisons, or functions on "running variables" for a > "attack detection set"). The concept here of course is to follow a tcp > stream looking for precursory actions that are needed to finesse an > endpoint into an exploitable situation. Be it forcing the endpoint to > allocate a buffer of X size with XYZ rpc call, that is later exploited > by calls to 3 other RPC calls with varying parameters. The critical key > is to engineer a system that affords you the flexibility to effectively > "emulate" and follow increasingly complex attack scenarios to the point > of detecting actual attacks and not simply triggering a FP. My single > largest frustration with snort (beyond the fact that i absolutely hate > its limited ability to detect "proper attacks") is that you are > effectively looking at one packet at a time. You do not have the > ability to create per tcp stream variables that would allow you to > compare against packets furthur down the stream. > > One thing we've been thinking about is something like Snort's flowbits, but then way extended. Having a way to capture data from the packets/stream, setting and calculating with counters, etc all per packet, flow/stream, host, network and globally. And then provide the rule language to compare each and everyone of those with each other, do translations etc. ModSecurity has features like this and they are very useful. For example in ModSecurity you can match on the password on HTTP basic auth by capturing the string containing the username and password from the HTTP header, splitting it, decoding it's base64. All from the rules language. Technically very challenging I'm sure, but lets keep the discussion on what we want in an ideal world and worry about how to do implementations that perform well in the real world later :P Regards, Victor > 8. Some sort of meta language needs to be created that easily and > effectively can communicate any applications "state". Even if this > means creating application specific "translation" modules that allow end > users to effectively outline functions used by an application (and what > they do). This of course would come in handy by allowing end users the > ability to effectively "peer" inside the "mapped logic" of an > application. This in turn (if done right) could aid in an analysts > ability to alert and investigate malicious behavior of an application vs > simply producing mind numbingly general "signatures' and hoping for the > best. (xss alerts come to mind, as well as other web app attacks) > > > Jeremy wrote: > >> Comments are inline but first I would like to add an item: >> >> 6. Network intelligence through passive identification. What I mean >> by this is like Snorts RNA, NMAP, and p0s fingerprinting to >> dynamically update inventory data for correlation engine usage. I >> think we all know how great it would be to have this data which could >> dynamically or auto magically configure/reconfigure signatures and >> correlation engines to align with applications and operating systems >> to truly substantiate the risk of attacks and/or threats. Having the >> availability of correlated network intelligence could make tremendous >> headway in the world of anomaly detection i.e. a windows xp home >> computer correlated with network flow data and port data could easily >> be alerted on for any outgoing port 25 connections, as they are most >> likely the newest member of a spam bot network. Obviously this is >> just a trivial example but it still is not a trivial task to pull off >> even with the most advanced SIM. This would also allow for >> operational performance enhancements by providing context regarding >> the network they operate in. The dynamic configuration and >> reconfiguration of network devices to ignore threats that are not >> applicable would definitely enhance performance by reducing overhead >> caused by signatures and other security measures. >> >> On Thu, Oct 16, 2008 at 8:00 PM, Matt Jonkman wrote: >> >> >>> Here's the big thread. And don't be afraid to start sub-threads for >>> specifics here. >>> >>> The features we want to go after here are the primary reason we sought >>> this funding and are taking this challenge on. Existing stuff works, but >>> there's SO much more we could be doing by looking past traditional ips >>> strengths. The challenge is that those things aren't conducive to making >>> a commercial product with millions invested in development. No one can >>> take this risk now, so we're going this route to make it happen. >>> >>> We have information about bad guys, bad places, and bad patterns. Lots >>> of it, terabytes of it. We've got gigs of data about bad stuff in the >>> sandnet at emerging threats alone. But most of that we can't effectively >>> act upon. We can't give huge lists of bad IPs to most tools, we can't >>> feed behavior patterns to existing tools, we can't share scan data >>> globally, etc. >>> >>> So here we are. I have things I wish I could do, you have things you >>> wish you could do, over the next couple of months we aim to get to the >>> core set of the most important things that most of us want to be able to >>> do. Then we'll go after it. >>> >>> So here's my wish list: >>> >>> 1. Native multithreading. >>> Not each preprocessor or post processor can go to a thread, but each >>> stream can take a thread. Think apache. More servers = more requests >>> served. THe complications of sharing state between them and the like is >>> a challenge, but solvable. >>> >>> >> I think you are dead on with this one, you have an advantage in that >> no code has been written so mutithreading doesn't have to be worked >> into the current code base. I think in many cases it is harder to >> convert single threaded applications into multithreaded apps than it >> would be to just rewrite applications with multithreading. >> >> >> >>> 2. IP Reputation Sharing >>> I want to feed these gigs of data I have and other projects have into my >>> security devices and let it use that data to make smarter decisions. IP >>> reputation isn't a new concept, but applying it in realtime will be a >>> challenge. But this also opens us up to the possibility of sharing >>> reputation data between ourselves. >>> >>> Imagine clouds of peer organizations sharing ip reputation between their >>> security devices. Each benefits from teh data gained and contributes >>> back what they encounter. All organizations become more safe. >>> >>> Then imagine organizations that collect this data for a living. We have >>> an avenue for this data to be more commercially viable. >>> >>> >>> >> I love this idea as well, but would not limit it to just IP data. The >> more data that is shared and collaborated on in a scoring system like >> you mention further down in your post the better in my opinion. I >> would like to see a correlation engine (ie you scoring scheme) factor >> in shared DNS data, Domain Reputations, ASN Reputations, ISP >> Reputations, Port Statistics, Protocol Statistics, Threat >> Indexes/Activities and statisitcs, vulnerability probabilities and >> activities, Network/Application/OS awareness, and GEO type statistics. >> I know this is kind of a big dream to have widely dispersed >> geographical networks sharing statistical data in real time that could >> be correlated to ensure even the smallest networks obtained the >> intelligence level and threat visibilities/awareness levels of the >> largest networks in the cloud... Obviously this is no trivial task! >> What I do see is if this idea/dream became reality, you would see AI >> implemented into IDS', Firewalls, Routers, Servers, Applications, and >> anything else that could listen for these correlated data statistics >> and adjust their configurations based off live data. Imagine an >> IDS/IPS that could auto magically load and unload specific rules to >> meet the threats seen on other networks in anticipation of attacks to >> come that may not have reached them yet or routers that could create >> null routes for troubled/bad IP subnets based off of data intelligence >> seen by this super cloud. Even web servers that could create mod >> rewrite rules or acls to prevent exploit bots from delivering their >> drive by badness, which I am sure have all grown to love in the last >> six months... I see limitless possibilities, but reality may play a >> factor here ;) >> >> >>> 3. Native ipv6 >>> Of course. No brainer there. >>> >>> >>> 4. Native Hardware acceleration support >>> There are a number of hardware acceleration technologies that could be >>> more effectively built into the engine from the start, versus the >>> back-asswards reverse engineering we have to do now to effectively >>> accelerate. >>> >>> >>> >> Really cool idea and would like to see this go forward, but again not >> a trivial mission. With hardware acceleration you must factor in >> cost. Why do we spend hours hacking the kernel, libpcap, and >> applications we use? Easy cause most of us are under budget >> constraints and can not afford to implement many of these hardware >> accelerators... I know we are going through the 10 Gb issues, and >> well buying a Bivio and Network General sniffer to do this is very >> painful for us and will break the bank.... that being said I welcome >> competitors in this area as I believe competition could cause some of >> these hardware acceleration technologies to fall in price. >> >> >>> 5. Scoring >>> Spam-assassin style point scoring. This would go a long way to >>> eliminating false positives. The absolutely sure 100% guaranteed true >>> positive rules of course would still hit. But the ones that are wrong as >>> often as right could be given a score, say a half a point. If something >>> else happens from that host within a certain timeframe that pushes that >>> over a threshold then all of these alerts come back and can be acted >>> upon with more confidence they're real. Complicated, but worthwhile. >>> >>> >>> >>> OK, those are my initial wish list items. Who has more? What else should >>> we do? Any problems with the above? >>> >>> Matt >>> >>> >>> >>> -- >>> -------------------------------------------- >>> Matthew Jonkman >>> Emerging Threats >>> Phone 765-429-0398 >>> Fax 312-264-0205 >>> http://www.emergingthreats.net >>> -------------------------------------------- >>> >>> PGP: http://www.jonkmans.com/mattjonkman.asc >>> >>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From andersen.mike at gmail.com Fri Oct 17 07:43:13 2008 From: andersen.mike at gmail.com (Mike Andersen) Date: Fri, 17 Oct 2008 13:43:13 +0200 Subject: [Discussion] [Mascot] Message-ID: <820C0FAF-1488-4775-A7B1-F91C8C6D8620@gmail.com> How about a Mole[1]? Cute little animal who digs around in the dirt, hunting for worms and other less likable creatures. Maybe "Mole" could be a working title for this project? ;-) [1] http://en.wikipedia.org/wiki/Mole Mike Andersen -- If I'd have asked my customers what they wanted, they would have told me "A faster horse". -- Henry Ford From thorsten.holz at informatik.uni-mannheim.de Fri Oct 17 08:09:35 2008 From: thorsten.holz at informatik.uni-mannheim.de (Thorsten Holz) Date: Fri, 17 Oct 2008 14:09:35 +0200 Subject: [Discussion] Why IDS sucks In-Reply-To: References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: On 17.10.2008, at 01:56, David Glosser wrote: > Matt, this begs for a wiki or something once you gather enough > information. Yes, a wiki would perhaps help to structure the information. > -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). > -ways to get alerts for fast-flux and other things where current sigs > are prone to FPs Just out of curiosity: what techniques do you use to detect fast-flux domains? And doesn't a whitelist help or do you see so many legitimate domains that look like fast-flux? Cheers, Thorsten From pepperjack at afferentsecurity.com Fri Oct 17 08:23:48 2008 From: pepperjack at afferentsecurity.com (Jack Pepper) Date: Fri, 17 Oct 2008 07:23:48 -0500 Subject: [Discussion] Welcome! In-Reply-To: <48F7D855.9020500@jonkmans.com> References: <48F7D855.9020500@jonkmans.com> Message-ID: <20081017072348.ovgzftj5wkkg8s8c@mail.afferentsecurity.com> Quoting Matt Jonkman : > 1. A mascot > Gotta have a mascot. The best suggestion we've seen to date I think is > an evil looking beaver. I'll try to take a picture the next time I visit my lawyers office .... -- Framework? I don't need no stinking framework! ---------------------------------------------------------------- @fferent Security Labs: Isolate/Insulate/Innovate http://www.afferentsecurity.com From david.glosser at gmail.com Fri Oct 17 10:17:46 2008 From: david.glosser at gmail.com (David Glosser) Date: Fri, 17 Oct 2008 10:17:46 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: I believe Matt tested some Fast-Flux sigs, but they were prone to FPs from legit sites such as amazon or cnn..... On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz wrote: > On 17.10.2008, at 01:56, David Glosser wrote: > >> Matt, this begs for a wiki or something once you gather enough >> information. > > Yes, a wiki would perhaps help to structure the information. > >> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >> -ways to get alerts for fast-flux and other things where current sigs >> are prone to FPs > > Just out of curiosity: what techniques do you use to detect fast-flux > domains? And doesn't a whitelist help or do you see so many legitimate > domains that look like fast-flux? > > Cheers, > Thorsten > > From aludwig at packetspy.com Fri Oct 17 10:24:26 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Fri, 17 Oct 2008 10:24:26 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: <48F8A01A.9090302@packetspy.com> Thanks to the work of Scott at nersc.gov there is a snort preproccessor that handles the DNS cache poisoning as well as FastFlux. http://www.nersc.gov/~scottc/software/snort/index.html Andre Ludwig David Glosser wrote: > I believe Matt tested some Fast-Flux sigs, but they were prone to FPs > from legit sites such as amazon or cnn..... > > On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz > wrote: > >> On 17.10.2008, at 01:56, David Glosser wrote: >> >> >>> Matt, this begs for a wiki or something once you gather enough >>> information. >>> >> Yes, a wiki would perhaps help to structure the information. >> >> >>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>> -ways to get alerts for fast-flux and other things where current sigs >>> are prone to FPs >>> >> Just out of curiosity: what techniques do you use to detect fast-flux >> domains? And doesn't a whitelist help or do you see so many legitimate >> domains that look like fast-flux? >> >> Cheers, >> Thorsten >> >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From jonkman at jonkmans.com Fri Oct 17 10:25:05 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 10:25:05 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> Message-ID: <48F8A041.9040605@jonkmans.com> David Glosser wrote: > I believe Matt tested some Fast-Flux sigs, but they were prone to FPs > from legit sites such as amazon or cnn..... Ya, we tried some things for low ttl dns, but it's used in legit apps too often. Only real way I see to effectively go after fast flux might be dns blacklisting. Most of them last for more than days, so blacklisting is relatively effective is we can keep effective sandnetting and intel gathering going. (Which is what we're trying to do at emerging threats for the long term) >>> Matt, this begs for a wiki or something once you gather enough >>> information. >> Yes, a wiki would perhaps help to structure the information. Absolutely. I'm making some space in the ET wiki, details soon. Matt >> >>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>> -ways to get alerts for fast-flux and other things where current sigs >>> are prone to FPs >> Just out of curiosity: what techniques do you use to detect fast-flux >> domains? And doesn't a whitelist help or do you see so many legitimate >> domains that look like fast-flux? >> >> Cheers, >> Thorsten >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 17 10:25:38 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 10:25:38 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F8A01A.9090302@packetspy.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F8A01A.9090302@packetspy.com> Message-ID: <48F8A062.6030105@jonkmans.com> Oh ya, I remember this work. I didn't ever test it myself. How has is fared in production? Matt Andre Ludwig wrote: > Thanks to the work of Scott at nersc.gov there is a snort preproccessor > that handles the DNS cache poisoning as well as FastFlux. > > http://www.nersc.gov/~scottc/software/snort/index.html > > Andre Ludwig > > David Glosser wrote: >> I believe Matt tested some Fast-Flux sigs, but they were prone to FPs >> from legit sites such as amazon or cnn..... >> >> On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz >> wrote: >> >>> On 17.10.2008, at 01:56, David Glosser wrote: >>> >>> >>>> Matt, this begs for a wiki or something once you gather enough >>>> information. >>>> >>> Yes, a wiki would perhaps help to structure the information. >>> >>> >>>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>>> -ways to get alerts for fast-flux and other things where current sigs >>>> are prone to FPs >>>> >>> Just out of curiosity: what techniques do you use to detect fast-flux >>> domains? And doesn't a whitelist help or do you see so many legitimate >>> domains that look like fast-flux? >>> >>> Cheers, >>> Thorsten >>> >>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From aludwig at packetspy.com Fri Oct 17 10:31:15 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Fri, 17 Oct 2008 10:31:15 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F8A062.6030105@jonkmans.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F8A01A.9090302@packetspy.com> <48F8A062.6030105@jonkmans.com> Message-ID: <48F8A1B3.4010301@packetspy.com> Initially there were a few memory leaks and core dumps, but they all got resolved. As for hard production use i have not heard anything negative yet. I am going to assume that is because we lack the proper marketing merits to get it out to enough people to test. (it worked during my quick testing) Andre Matt Jonkman wrote: > Oh ya, I remember this work. I didn't ever test it myself. How has is > fared in production? > > Matt > > Andre Ludwig wrote: > >> Thanks to the work of Scott at nersc.gov there is a snort preproccessor >> that handles the DNS cache poisoning as well as FastFlux. >> >> http://www.nersc.gov/~scottc/software/snort/index.html >> >> Andre Ludwig >> >> David Glosser wrote: >> >>> I believe Matt tested some Fast-Flux sigs, but they were prone to FPs >>> from legit sites such as amazon or cnn..... >>> >>> On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz >>> wrote: >>> >>> >>>> On 17.10.2008, at 01:56, David Glosser wrote: >>>> >>>> >>>> >>>>> Matt, this begs for a wiki or something once you gather enough >>>>> information. >>>>> >>>>> >>>> Yes, a wiki would perhaps help to structure the information. >>>> >>>> >>>> >>>>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>>>> -ways to get alerts for fast-flux and other things where current sigs >>>>> are prone to FPs >>>>> >>>>> >>>> Just out of curiosity: what techniques do you use to detect fast-flux >>>> domains? And doesn't a whitelist help or do you see so many legitimate >>>> domains that look like fast-flux? >>>> >>>> Cheers, >>>> Thorsten >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > > From jeremy at sudosecure.net Fri Oct 17 10:43:08 2008 From: jeremy at sudosecure.net (Jeremy) Date: Fri, 17 Oct 2008 09:43:08 -0500 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F8A1B3.4010301@packetspy.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F8A01A.9090302@packetspy.com> <48F8A062.6030105@jonkmans.com> <48F8A1B3.4010301@packetspy.com> Message-ID: <1a0f92ae0810170743v1edea24bjca6646d5829080af@mail.gmail.com> I too did limited lab testing with this DNS preproc and have had no real issues with it. I have planned to move this from the lab in the next few weeks to test on some real traffic, so I hope to have better feedback real soon. --jeremy On Fri, Oct 17, 2008 at 9:31 AM, Andre Ludwig wrote: > Initially there were a few memory leaks and core dumps, but they all got > resolved. As for hard production use i have not heard anything negative > yet. I am going to assume that is because we lack the proper marketing > merits to get it out to enough people to test. (it worked during my > quick testing) > > Andre > > Matt Jonkman wrote: >> Oh ya, I remember this work. I didn't ever test it myself. How has is >> fared in production? >> >> Matt >> >> Andre Ludwig wrote: >> >>> Thanks to the work of Scott at nersc.gov there is a snort preproccessor >>> that handles the DNS cache poisoning as well as FastFlux. >>> >>> http://www.nersc.gov/~scottc/software/snort/index.html >>> >>> Andre Ludwig >>> >>> David Glosser wrote: >>> >>>> I believe Matt tested some Fast-Flux sigs, but they were prone to FPs >>>> from legit sites such as amazon or cnn..... >>>> >>>> On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz >>>> wrote: >>>> >>>> >>>>> On 17.10.2008, at 01:56, David Glosser wrote: >>>>> >>>>> >>>>> >>>>>> Matt, this begs for a wiki or something once you gather enough >>>>>> information. >>>>>> >>>>>> >>>>> Yes, a wiki would perhaps help to structure the information. >>>>> >>>>> >>>>> >>>>>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>>>>> -ways to get alerts for fast-flux and other things where current sigs >>>>>> are prone to FPs >>>>>> >>>>>> >>>>> Just out of curiosity: what techniques do you use to detect fast-flux >>>>> domains? And doesn't a whitelist help or do you see so many legitimate >>>>> domains that look like fast-flux? >>>>> >>>>> Cheers, >>>>> Thorsten >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Discussion mailing list >>>> Discussion at openinfosecfoundation.org >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>>> >>>> >>>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >> >> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From aludwig at packetspy.com Fri Oct 17 10:51:54 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Fri, 17 Oct 2008 10:51:54 -0400 Subject: [Discussion] Why IDS sucks In-Reply-To: <1a0f92ae0810170743v1edea24bjca6646d5829080af@mail.gmail.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F8A01A.9090302@packetspy.com> <48F8A062.6030105@jonkmans.com> <48F8A1B3.4010301@packetspy.com> <1a0f92ae0810170743v1edea24bjca6646d5829080af@mail.gmail.com> Message-ID: <48F8A68A.8030901@packetspy.com> Feel free to send any feedback to scott (email at the link i sent out), or to me (ill send it to Scott). Andre Ludwig Jeremy wrote: > I too did limited lab testing with this DNS preproc and have had no > real issues with it. I have planned to move this from the lab in the > next few weeks to test on some real traffic, so I hope to have better > feedback real soon. > > --jeremy > > On Fri, Oct 17, 2008 at 9:31 AM, Andre Ludwig wrote: > >> Initially there were a few memory leaks and core dumps, but they all got >> resolved. As for hard production use i have not heard anything negative >> yet. I am going to assume that is because we lack the proper marketing >> merits to get it out to enough people to test. (it worked during my >> quick testing) >> >> Andre >> >> Matt Jonkman wrote: >> >>> Oh ya, I remember this work. I didn't ever test it myself. How has is >>> fared in production? >>> >>> Matt >>> >>> Andre Ludwig wrote: >>> >>> >>>> Thanks to the work of Scott at nersc.gov there is a snort preproccessor >>>> that handles the DNS cache poisoning as well as FastFlux. >>>> >>>> http://www.nersc.gov/~scottc/software/snort/index.html >>>> >>>> Andre Ludwig >>>> >>>> David Glosser wrote: >>>> >>>> >>>>> I believe Matt tested some Fast-Flux sigs, but they were prone to FPs >>>>> from legit sites such as amazon or cnn..... >>>>> >>>>> On Fri, Oct 17, 2008 at 8:09 AM, Thorsten Holz >>>>> wrote: >>>>> >>>>> >>>>> >>>>>> On 17.10.2008, at 01:56, David Glosser wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Matt, this begs for a wiki or something once you gather enough >>>>>>> information. >>>>>>> >>>>>>> >>>>>>> >>>>>> Yes, a wiki would perhaps help to structure the information. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -I would like to get alerts for L2 stuff (BPDUs, CDP, etc). >>>>>>> -ways to get alerts for fast-flux and other things where current sigs >>>>>>> are prone to FPs >>>>>>> >>>>>>> >>>>>>> >>>>>> Just out of curiosity: what techniques do you use to detect fast-flux >>>>>> domains? And doesn't a whitelist help or do you see so many legitimate >>>>>> domains that look like fast-flux? >>>>>> >>>>>> Cheers, >>>>>> Thorsten >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Discussion mailing list >>>>> Discussion at openinfosecfoundation.org >>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Discussion mailing list >>>> Discussion at openinfosecfoundation.org >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>>> >>>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > > From jonkman at jonkmans.com Fri Oct 17 11:24:55 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 11:24:55 -0400 Subject: [Discussion] Wiki Setup Message-ID: <48F8AE47.8090603@jonkmans.com> http://doc.emergingthreats.net/bin/view/Main/OpenInfosec I've got the discussion so far in there I believe. Please feel free to edit directly, but lets keep the similar discussion on the list. Gives us history as well as keeps everyone on the same sheet of code. Great ideas so far everyone! Please keep the discussion going, and lets explore some of the less traditional possibilities. The strangest idea wins a beer! Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jgimer at gmail.com Fri Oct 17 11:51:27 2008 From: jgimer at gmail.com (Joshua Gimer) Date: Fri, 17 Oct 2008 09:51:27 -0600 Subject: [Discussion] Features In-Reply-To: <48F7E3B0.20002@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> Message-ID: On Thu, Oct 16, 2008 at 7:00 PM, Matt Jonkman wrote: > OK, those are my initial wish list items. Who has more? What else should > we do? Any problems with the above? > > Matt > I like all of the ideas above. :) I would like to see more event correlation functionality built into detection engines. I give the example of all the recent distributed SSH brute force issue (or the distributed SQL Injection method used by the Asprox botnet) that is becoming more common. The majority of IDS/IPS engines will not trigger on this kind of distributed traffic. I would like to see the ability for the engine to track these connections to a common destination from multiple sources and then assess the weight of the traffic that is hitting the system within a given time window. I agree that this would be hard to do without blocking legitimate sources, but I see this as being one of the largest downfalls in modern IDS/IPS. -- Thx Joshua Gimer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/21b3e64e/attachment.html From jonkman at jonkmans.com Fri Oct 17 11:54:56 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 11:54:56 -0400 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> Message-ID: <48F8B550.5020709@jonkmans.com> Does the scoring ability cover what you're thinking? Or maybe if we see hits that satisfy a certain condition (like similar attacks from many sites, such as an ssh brute), then all similar hits for a timeframe get an automatic score bump up? matt Joshua Gimer wrote: > > On Thu, Oct 16, 2008 at 7:00 PM, Matt Jonkman > wrote: > > OK, those are my initial wish list items. Who has more? What else should > we do? Any problems with the above? > > Matt > > > I like all of the ideas above. :) > > I would like to see more event correlation functionality built into > detection engines. I give the example of all the recent distributed SSH > brute force issue (or the distributed SQL Injection method used by the > Asprox botnet) that is becoming more common. The majority of IDS/IPS > engines will not trigger on this kind of distributed traffic. > > I would like to see the ability for the engine to track these > connections to a common destination from multiple sources and then > assess the weight of the traffic that is hitting the system within a > given time window. I agree that this would be hard to do without > blocking legitimate sources, but I see this as being one of the largest > downfalls in modern IDS/IPS. > > -- > Thx > Joshua Gimer -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From lists at inliniac.net Fri Oct 17 11:58:08 2008 From: lists at inliniac.net (Victor Julien) Date: Fri, 17 Oct 2008 17:58:08 +0200 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> Message-ID: <48F8B610.1000109@inliniac.net> Joshua Gimer wrote: > > On Thu, Oct 16, 2008 at 7:00 PM, Matt Jonkman > wrote: > > OK, those are my initial wish list items. Who has more? What > else should > we do? Any problems with the above? > > Matt > > > I like all of the ideas above. :) > > I would like to see more event correlation functionality built into > detection engines. I give the example of all the recent distributed > SSH brute force issue (or the distributed SQL Injection method used > by the Asprox botnet) that is becoming more common. The majority of > IDS/IPS engines will not trigger on this kind of distributed traffic. > > I would like to see the ability for the engine to track these > connections to a common destination from multiple sources and then > assess the weight of the traffic that is hitting the system within a > given time window. I agree that this would be hard to do without > blocking legitimate sources, but I see this as being one of the > largest downfalls in modern IDS/IPS. > I like the idea to have a rule directive that can say: if this rule matches, add the (src|dst|both) ip to some set. Then use that set in other rules, etc... Regards, Victor From pepperjack at autoshun.org Fri Oct 17 12:23:23 2008 From: pepperjack at autoshun.org (Jack Pepper) Date: Fri, 17 Oct 2008 11:23:23 -0500 Subject: [Discussion] toward what goal? In-Reply-To: <48F8AE47.8090603@jonkmans.com> References: <48F8AE47.8090603@jonkmans.com> Message-ID: <20081017112323.sa5ln8sqkg8c488o@mail.afferentsecurity.com> Quoting Matt Jonkman : > > Great ideas so far everyone! Please keep the discussion going, and lets > explore some of the less traditional possibilities. The strangest idea > wins a beer! > Here's a strange idea: Rather than deciding how to build it or what to build, try to figure out what you would want to do with it. Otherwise you may end up building something which is totally, *totally* cool, but has no practical application (ala gnu/hurd). so here's my theoretical brain exercise: "If the " + [ insert your choice of constituency here, as in CIO / Network Admin / Switch Vendor / Defence Department ] + "had perfect visibility into the activities of " + [ insert sensor medium here, as in Network Traffic, server or firewall Event Logs, Network solutions' invoicing system, Paris hilton's Cell Phone ] + "with an AI engine and associated toolkit for data drilling and correlation, what defensive or offensive actions could they take that are not currently feasible or practicable?" When this is clear in our minds, we will know better what to build. If you just want to build a better Snort, that sounds quite pointless, given that snort has evolved already to become the best packet content scanner on the market. All the downsides of snort involve the input, output, actionable outputs, reporting actions, rule parsing limitations, limitations of the host OS, problems with PCI adapter limitations, DMA speeds, PCI-x limitations, etc, etc. The core engine and architecture are very nicely adapted (as in evolution/natural selection) to the task at hand. To be truly revolutionary (relative to current technologies) the problem needs to be narrowed toward some action we wish to take that is not currently possible or reasonable. consider: 1962: single goal of high altitude observation built the YF12/SR71 1969: single goal of highly survivable communications built the DarpaNet 1978: single goal of radar evasion built the F117/B2 (those three lines should trigger an off-topic flame war, I didn't go look up the dates, I'm just working from an old guy's memory) To be worthy of the challenge, the team goal needs to be highly focused. then it will be possible to achieve great things. what is the goal? what is the point? we all know how to build a mega-snort box, but no one does it because there is no reason to do it. Can't sell it, no one would buy it, don't really need it, and don't have $200,000 just lying about. Mega-snort doesn't really get us very far toward the goal. My 2c. jp ---------------------------------------------------------------- @fferent Security Labs: Isolate/Insulate/Innovate http://www.afferentsecurity.com From jonkman at jonkmans.com Fri Oct 17 12:46:08 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 12:46:08 -0400 Subject: [Discussion] [Fwd: Symantec's participation in IDP/IPS group] Message-ID: <48F8C150.6090308@jonkmans.com> Forwarded at Symantec's request -------- Original Message -------- Subject: Symantec's participation in IDP/IPS group Date: Fri, 17 Oct 2008 09:16:18 -0700 This is to confirm that Symantec would like to participate in the IDS/IPS project that was just announced and is being sponsored by DHS S&T. The primary POC for Symantec will be Gary Phillips, Senior Director of Development, from the Office of the CTO. Gary's contact info is: gary_phillips at symantec.com 407-357-5613 Please let me know if you have any additional questions. Best, Tiffany Jones Director, Government Relations Symantec Corporation 703-589-7402 -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 17 12:51:39 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 12:51:39 -0400 Subject: [Discussion] [Fwd: Symantec's participation in IDP/IPS group] In-Reply-To: <48F8C150.6090308@jonkmans.com> References: <48F8C150.6090308@jonkmans.com> Message-ID: <48F8C29B.6020008@jonkmans.com> We're VERY excited to have Symantec's involvement in the project. It's long term survival is 100% dependent on this kind of support! We'll be getting the consortium construction and bylaws ironed out very soon and get those public. Thanks very much!! Matt Matt Jonkman wrote: > Forwarded at Symantec's request > > -------- Original Message -------- > Subject: Symantec's participation in IDP/IPS group > Date: Fri, 17 Oct 2008 09:16:18 -0700 > > This is to confirm that Symantec would like to participate in the > IDS/IPS project that was just announced and is being sponsored by DHS S&T. > > The primary POC for Symantec will be Gary Phillips, Senior Director of > Development, from the Office of the CTO. > > Gary's contact info is: > gary_phillips at symantec.com > 407-357-5613 > > Please let me know if you have any additional questions. > > Best, > Tiffany Jones > Director, Government Relations > Symantec Corporation > 703-589-7402 > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 17 13:13:49 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 13:13:49 -0400 Subject: [Discussion] toward what goal? In-Reply-To: <20081017112323.sa5ln8sqkg8c488o@mail.afferentsecurity.com> References: <48F8AE47.8090603@jonkmans.com> <20081017112323.sa5ln8sqkg8c488o@mail.afferentsecurity.com> Message-ID: <48F8C7CD.7070501@jonkmans.com> Jack Pepper wrote: > Here's a strange idea: > Rather than deciding how to build it or what to build, try to > figure out what you would want to do with it. Otherwise you may end > up building something which is totally, *totally* cool, but has no > practical application (ala gnu/hurd). Very good point. Personally, I want a way to use all of the information we know about bad things. I want it to work well without having to read 3 books about it to configure it well, and give me actionable stuff and keep the rest to itself. What else can we add to that? Matt > so here's my theoretical brain exercise: > "If the " + > [ insert your choice of constituency here, as in CIO / Network > Admin / Switch Vendor / Defence Department ] + > "had perfect visibility into the activities of " + > [ insert sensor medium here, as in Network Traffic, server or > firewall Event Logs, Network solutions' invoicing system, Paris > hilton's Cell Phone ] + > "with an AI engine and associated toolkit for data drilling and > correlation, what defensive or offensive actions could they take that > are not currently feasible or practicable?" > > When this is clear in our minds, we will know better what to build. > If you just want to build a better Snort, that sounds quite pointless, > given that snort has evolved already to become the best packet content > scanner on the market. All the downsides of snort involve the input, > output, actionable outputs, reporting actions, rule parsing > limitations, limitations of the host OS, problems with PCI adapter > limitations, DMA speeds, PCI-x limitations, etc, etc. The core engine > and architecture are very nicely adapted (as in evolution/natural > selection) to the task at hand. > > To be truly revolutionary (relative to current technologies) the > problem needs to be narrowed toward some action we wish to take that > is not currently possible or reasonable. consider: > 1962: single goal of high altitude observation built the YF12/SR71 > 1969: single goal of highly survivable communications built the DarpaNet > 1978: single goal of radar evasion built the F117/B2 > (those three lines should trigger an off-topic flame war, I didn't go > look up the dates, I'm just working from an old guy's memory) > > To be worthy of the challenge, the team goal needs to be highly > focused. then it will be possible to achieve great things. what is > the goal? what is the point? > > we all know how to build a mega-snort box, but no one does it because > there is no reason to do it. Can't sell it, no one would buy it, > don't really need it, and don't have $200,000 just lying about. > Mega-snort doesn't really get us very far toward the goal. > > My 2c. > > jp > > ---------------------------------------------------------------- > @fferent Security Labs: Isolate/Insulate/Innovate > http://www.afferentsecurity.com > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 17 13:48:50 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 13:48:50 -0400 Subject: [Discussion] IRC Message-ID: <48F8D002.7000801@jonkmans.com> For those interested there is an official freenode channel: #oisf Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From rMslade at shaw.ca Fri Oct 17 15:44:15 2008 From: rMslade at shaw.ca (Rob, grandpa of Ryan, Trevor, Devon & Hannah) Date: Fri, 17 Oct 2008 11:44:15 -0800 Subject: [Discussion] Features - egress In-Reply-To: <48F7E3B0.20002@jonkmans.com> Message-ID: <48F87A8F.22439.119F3EE@localhost> Date sent: Thu, 16 Oct 2008 21:00:32 -0400 From: Matt Jonkman > Here's the big thread. And don't be afraid to start sub-threads for > specifics here. OK :-) > OK, those are my initial wish list items. Who has more? What else should > we do? Allow me to throw in a strong push for including egress scanning and analysis. We tend to get fixated on the traditional bastion position, with the bad guys all on the outside and everything inside is pure. In the current malware-rich environment that is untenable. We also can gain a lot more granular information (in addition to the defence-in-depth backstop) from egress scanning, since we have a much batter idea of what *should* be leaving our nets. ====================== (quote inserted randomly by Pegasus Mailer) rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org I appreciate the fact that this draft was done in haste, but some of the sentences that you are sending out in the world to do your work for you are loitering in taverns or asleep beside the highway. -- Dr. Dwight Van de Vate, Professor of Philosophy, University of Tennessee at Knoxville victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ From mcholste at gmail.com Fri Oct 17 14:53:08 2008 From: mcholste at gmail.com (Martin Holste) Date: Fri, 17 Oct 2008 13:53:08 -0500 Subject: [Discussion] Features and Design Strategy Message-ID: First off, many congrats, Matt! When you look at the threat landscape, it's clear that the most imminent threats are against web-facing clients and servers, so I think we should focus on client-side attacks and attacks against web applications. Detecting SMB protocol attacks, etc. is still important, but most attacks against non-web protocols need a foothold on the LAN first. Most IDS are historically built for defending servers from clients, and I think that idea is becoming a bit antiquated, though certainly far from obsolete. My group has had great success doing what John at Berkeley is/will be doing with file carving. We've been auto-extracting packed files from the network stream and auto-submitting to VirusTotal for over a year now. John's point about processing the layer-7 data inside the streams is dead-on. We need to move away from answering "what strings did the traffic contain" to answering "what did the host actually do?" We also need to keep in mind that canonically defining what "bad" behavior is can be incredibly difficult even if you're just trying to put it in words. The design strategy that it is important to me is plugin extensibility. One of the reasons I love perl so much is that there is a CPAN module for just about anything you could ever dream of needing. The extreme ease with which you can incorporate others' code into your own project is what has kept it thriving for so long. The differences between individual organization's security needs and preferences mandate that a successful project will have a long-term goal of being extremely modular, while also having a core functionality that is superb. Since Snort is what I'm familiar with, I will try to make my point by comparison: Snort has excellent core functionalilty, but despite Marty's best efforts, extending it is not a trivial task because the code is in C. When Jason Brevenik posted his SnortUnified.pm module which gives perl access to realtime Snort alerts, I realized that there was suddenly an avenue for decoupling the pure network-grepping core functionality of Snort with the action component. Moreover, I now had a hook for writing extremely customized and complicated event handling in my programming language of choice instead of trying to debug segfaults in preprocessors all day. So, while perl would be far too slow to do real-time content matching, it is certainly not too slow to receive an alert, check to see if this source IP has alerted recently, create an incident with some priority, see who needs to be notified of the incident, and then send an alert. So, the point is, we only need the really fast component to do a limited number of tasks, and if we decouple the real-time, per-packet tasks from the high level "what do we do with this" tasks, anyone is free to trivially write their own hooks into how they want things handled, which is the part that varies the most from org to org. Marty recognized all of this when he wrote the unified output module, but Sourcefire never had a commercial reason for extending it much since it would encroach on their own product offerings. OISF has the advantage of not being for-profit, and so we can make extensibility a priority. What I would like to see is a brand-new program with functions similar to Snort's HTTP preprocessor which has the sole purpose in life of sniffing the network and attaching HTTP header fields to the traffic, most importantly the host, content-type, and content-length fields. This daemon would read a config file which specifies rules more like scripts. The rules would have variables populated from a policy server, and it would update these variables very regularly (without a program restart of any kind). This would de-couple the work of sniffing from the work of policy setting. The policy server would get its variables from all kinds of sources, like remote XML files. Armed with this, very simple signatures which identify web applications could be written. Here's a pseudo-code example of what I think the configs would look like: ## On the policy server ## # Declare vars DECLARE $attacked = ARRAY; DECLARE $attackers = ARRAY; DECLARE $bad_hosts = { type=dns, url=blacklists.example.org/bad_dns.xml, refresh_list=30 }; DECLARE $bad_ips = { type=ip, url=more_blacklists.example.org/bad_ips.xml, refresh_list=30 }; DECLARE $our_windows_hosts = { type=ip, url= http://mycompany.example.org/our_windows_hosts.xml, refresh_list=900 }; ## On the HTTP sniffer ## # Detect our web apps ID $google AS [http,https ] FROM [ host='*.google.com' ]; ID $google_search_result AS [ $google ] MATCHING [ uri =~ /search/ ]; ID $our_web_clients AS [http,https] TO [ $our_windows_hosts ]; ID $us_sending_email AS [smtp] FROM [ $our_web_clients ]; ID $malware_webapp AS [ http, https ] FROM [ $bad_hosts OR $bad_ips ]; # Define our actions # Create an event and record a client as attacked if it is referred to a malware page by a Google search FORWARD EVENT IF { $google_search_result REFERS $malware_webapp } AND [ ADD $_CLIENT TO $attacked, ADD $_SERVER TO $attackers ]; The important thing is that our daemon doing the HTTP sniffing is agnostic in that it has no idea if things are bad or good, it just lets somebody know when a predefined set of criteria has occurred. It is up to an event policy server to receive these events and do something intelligent (and highly configurable) with it. So, we've decoupled any decision making from the component that has the most time-sensitive work to do. All of the other components can buffer their work, but the network sniffer needs as much work lifted from it as possible so that it can cope with high loads. The policy server components would be the only parts that would require much customization, and they would be written in a high-level language like perl for maximum extensibility. Let's also not forget the plethora of other apps out there that we can leverage. Of particular importance would be SANCP and PADS. SANCP's newest version contains a feature that John Curry added for me which writes out an index of the file position within a bulk pcap where a packet starts and stops. This means that you can do instantaenous bulk packet retrievals using the index. In my tests, I can pull an arbitrary host's traffic from a 30 GB pcap in under 5 seconds. This technique could be used for file carving, which would provide the ability to send to sandboxes, send to VirusTotal, Symantec, etc. PADS provides a ready-made daemon for identifying the protocols used between hosts. If PADS fed a policy server, we could automagically update the policy server when new Windows hosts come online, or when an SMTP session starts. Since the HTTP sniffer would be reading from the policy server on a regular basis, PADS would be able to keep it informed as to the state of the network. P0f could work as well. So, the main idea would be to have a very lightweight daemon with a specific task forward events to an event policy server written in a higher level language, and to remove almost all decision making from the sniffer. Comments? Thanks, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/f05d96c6/attachment-0001.html From Simon.Dredge at endace.com Fri Oct 17 14:57:42 2008 From: Simon.Dredge at endace.com (Simon Dredge) Date: Fri, 17 Oct 2008 14:57:42 -0400 Subject: [Discussion] Congratulations Message-ID: <942C74F45C7A844D9A63B58FD452D66F3CE041@usresdcn01.ad.endace.com> Contrats on getting this new initiative off the ground, guys. Please let me know how Endace can best participate / contribute. Simon. www.endace.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/b2d36892/attachment.html From jonkman at jonkmans.com Fri Oct 17 15:01:14 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 15:01:14 -0400 Subject: [Discussion] Congratulations In-Reply-To: <942C74F45C7A844D9A63B58FD452D66F3CE041@usresdcn01.ad.endace.com> References: <942C74F45C7A844D9A63B58FD452D66F3CE041@usresdcn01.ad.endace.com> Message-ID: <48F8E0FA.7030904@jonkmans.com> Hi Simon! Glad Endace is interested. As mentioned we're working up the bylaws for the consortium. We'll talk more about that very soon and make those public. Native HW acceleration support is a major goal for the engine, we'll have a lot to talk about! Matt Simon Dredge wrote: > Contrats on getting this new initiative off the ground, guys. Please let > me know how Endace can best participate / > > contribute. > > > > > > Simon. > > www.endace.com > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From mcholste at gmail.com Fri Oct 17 15:05:32 2008 From: mcholste at gmail.com (Martin Holste) Date: Fri, 17 Oct 2008 14:05:32 -0500 Subject: [Discussion] Features - egress In-Reply-To: <48F87A8F.22439.119F3EE@localhost> References: <48F7E3B0.20002@jonkmans.com> <48F87A8F.22439.119F3EE@localhost> Message-ID: Agreed! There's also the point that scanning clients' egress traffic can indicate whether a malware infection was successful or not. Customers don't care much about attacks, they care about compromises, and while being aware of unsuccessful attacks is important to us so that we can warn others, it's not the top priority. --Martin On Fri, Oct 17, 2008 at 2:44 PM, Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > Date sent: Thu, 16 Oct 2008 21:00:32 -0400 > From: Matt Jonkman > > > Here's the big thread. And don't be afraid to start sub-threads for > > specifics here. > > OK :-) > > > OK, those are my initial wish list items. Who has more? What else should > > we do? > > Allow me to throw in a strong push for including egress scanning and > analysis. We > tend to get fixated on the traditional bastion position, with the bad guys > all on the > outside and everything inside is pure. In the current malware-rich > environment > that is untenable. We also can gain a lot more granular information (in > addition > to the defence-in-depth backstop) from egress scanning, since we have a > much > batter idea of what *should* be leaving our nets. > > ====================== (quote inserted randomly by Pegasus Mailer) > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > I appreciate the fact that this draft was done in haste, but > some of the sentences that you are sending out in the world to > do your work for you are loitering in taverns or asleep beside > the highway. > -- Dr. Dwight Van de Vate, Professor of Philosophy, > University of Tennessee at Knoxville > victoria.tc.ca/techrev/rms.htm > blogs.securiteam.com/index.php/archives/author/p1/ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/270b47e3/attachment.html From jives at security.berkeley.edu Fri Oct 17 15:25:15 2008 From: jives at security.berkeley.edu (John Ives) Date: Fri, 17 Oct 2008 12:25:15 -0700 Subject: [Discussion] Features - egress In-Reply-To: <48F87A8F.22439.119F3EE@localhost> References: <48F87A8F.22439.119F3EE@localhost> Message-ID: <48F8E69B.1090307@security.berkeley.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > Date sent: Thu, 16 Oct 2008 21:00:32 -0400 > From: Matt Jonkman > >> Here's the big thread. And don't be afraid to start sub-threads for >> specifics here. > > OK :-) > >> OK, those are my initial wish list items. Who has more? What else should >> we do? > > Allow me to throw in a strong push for including egress scanning and analysis. We > tend to get fixated on the traditional bastion position, with the bad guys all on the > outside and everything inside is pure. In the current malware-rich environment > that is untenable. We also can gain a lot more granular information (in addition > to the defence-in-depth backstop) from egress scanning, since we have a much > batter idea of what *should* be leaving our nets. Speak for yourself :) My long standing and still somewhat accurate joke is that we try to protect our students from the Internet less than we try to protect the Internet from our students. Actually thanks to better policies and education, we are now more balanced in our detection aims than ever before, but I still spend a lot of my time looking at the outbound stuff as well. Having said that, you are correct, in that, out of the box, most IDS/IPS tools have a mentality that the bad stuff is coming from outside the border and that just isn't always true. Additionally, in most environments, the outbound traffic should be easier to diagnose and classify than the inbound (of course this isn't necessarily true in environments that lack strong central control - like academic institutions). Yours, John - -- - ------------------------------------------------------------------------- John Ives Phone (510) 642-7773 System & Network Security Cell (510) 229-8676 University of California, Berkeley - ------------------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEcBAEBAgAGBQJI+OabAAoJEJkidK6qbywsKhcIAJw8WbViCU0THP2wnw+r2vN8 spWUEibBujFWtS+rCHVnv1hMKieQEL+NwOEz6pxAvO/O+uJvIu3Ll7UFcnNsR5Kj O8OcHOmlFE7waSEmSn3UYmHCas6tMSz6+7Av3rMGg3z1Pj+vG0MJE9h1MyBVY0mR 192YsNt2cESdhN3JU2c7YoiqbbRCFYMplSpZN5+4WkuO8qcOJVxLGg0li4e5a0EG k8IGGA2wgpc9Yy+SxTVSe8NTC+nlPcEz7p5Yg7YYiOsvupyCiF5wkGC1cU0i2CXx V2cTEGu4bG7eYfudD81guiLAvtC3iXVPOvPKVUHPkHUZcbl1GYtP3tmy2GbGh58= =ES6o -----END PGP SIGNATURE----- From jonkman at jonkmans.com Fri Oct 17 15:36:16 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 15:36:16 -0400 Subject: [Discussion] Features - egress In-Reply-To: <48F8E69B.1090307@security.berkeley.edu> References: <48F87A8F.22439.119F3EE@localhost> <48F8E69B.1090307@security.berkeley.edu> Message-ID: <48F8E930.9040001@jonkmans.com> John Ives wrote: >> Allow me to throw in a strong push for including egress scanning and analysis. We >> tend to get fixated on the traditional bastion position, with the bad guys all on the >> outside and everything inside is pure. In the current malware-rich environment >> that is untenable. We also can gain a lot more granular information (in addition >> to the defence-in-depth backstop) from egress scanning, since we have a much >> batter idea of what *should* be leaving our nets. > > Speak for yourself :) > > My long standing and still somewhat accurate joke is that we try to > protect our students from the Internet less than we try to protect the > Internet from our students. Both valid security stances, which I think makes the point that we have to build tools to look both ways. Both to protect from outside direct attack, and warn us when we're the problem. It'd be an interesting alert to have your perimeter defenses pop up an alert telling you that the reputation of an IP in your range is suddenly getting a bad reputation in a certain category. But knowing the reputation of berkeley students, I think you SHOULD be protecting the internet from your students. :) Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 17 15:57:58 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 15:57:58 -0400 Subject: [Discussion] Features and Design Strategy In-Reply-To: References: Message-ID: <48F8EE46.1090709@jonkmans.com> Well put Martin. I definitely like the idea. It goes well with one of the core things we need to do, let everything thread off away from the flow of packets as possible. So to generalize a bit more, we should strive to make anything possible able to be hooked, or fed out to a side process as possible. Output of course, but anything else as well. Just leave generic hook points any process could pull data from fifo style. This would likely be where we could let the extracted binaries come out of the core processes as well. Great concept to add, I'll get this into the wiki so we don't lose the idea. What challenges does anyone see in this concept? What other advantages might we gain if we plan correctly? Matt Martin Holste wrote: > First off, many congrats, Matt! > > When you look at the threat landscape, it's clear that the most imminent > threats are against web-facing clients and servers, so I think we should > focus on client-side attacks and attacks against web applications. > Detecting SMB protocol attacks, etc. is still important, but most > attacks against non-web protocols need a foothold on the LAN first. > Most IDS are historically built for defending servers from clients, and > I think that idea is becoming a bit antiquated, though certainly far > from obsolete. > > My group has had great success doing what John at Berkeley is/will be > doing with file carving. We've been auto-extracting packed files from > the network stream and auto-submitting to VirusTotal for over a year > now. John's point about processing the layer-7 data inside the streams > is dead-on. We need to move away from answering "what strings did the > traffic contain" to answering "what did the host actually do?" We also > need to keep in mind that canonically defining what "bad" behavior is > can be incredibly difficult even if you're just trying to put it in words. > > The design strategy that it is important to me is plugin extensibility. > One of the reasons I love perl so much is that there is a CPAN module > for just about anything you could ever dream of needing. The extreme > ease with which you can incorporate others' code into your own project > is what has kept it thriving for so long. The differences between > individual organization's security needs and preferences mandate that a > successful project will have a long-term goal of being extremely > modular, while also having a core functionality that is superb. > > Since Snort is what I'm familiar with, I will try to make my point by > comparison: Snort has excellent core functionalilty, but despite > Marty's best efforts, extending it is not a trivial task because the > code is in C. When Jason Brevenik posted his SnortUnified.pm module > which gives perl access to realtime Snort alerts, I realized that there > was suddenly an avenue for decoupling the pure network-grepping core > functionality of Snort with the action component. Moreover, I now had a > hook for writing extremely customized and complicated event handling in > my programming language of choice instead of trying to debug segfaults > in preprocessors all day. So, while perl would be far too slow to do > real-time content matching, it is certainly not too slow to receive an > alert, check to see if this source IP has alerted recently, create an > incident with some priority, see who needs to be notified of the > incident, and then send an alert. So, the point is, we only need the > really fast component to do a limited number of tasks, and if we > decouple the real-time, per-packet tasks from the high level "what do we > do with this" tasks, anyone is free to trivially write their own hooks > into how they want things handled, which is the part that varies the > most from org to org. Marty recognized all of this when he wrote the > unified output module, but Sourcefire never had a commercial reason for > extending it much since it would encroach on their own product > offerings. OISF has the advantage of not being for-profit, and so we > can make extensibility a priority. > > What I would like to see is a brand-new program with functions similar > to Snort's HTTP preprocessor which has the sole purpose in life of > sniffing the network and attaching HTTP header fields to the traffic, > most importantly the host, content-type, and content-length fields. > This daemon would read a config file which specifies rules more like > scripts. The rules would have variables populated from a policy server, > and it would update these variables very regularly (without a program > restart of any kind). This would de-couple the work of sniffing from > the work of policy setting. The policy server would get its variables > from all kinds of sources, like remote XML files. Armed with this, very > simple signatures which identify web applications could be written. > Here's a pseudo-code example of what I think the configs would look like: > > ## On the policy server ## > > # Declare vars > > DECLARE $attacked = ARRAY; > > DECLARE $attackers = ARRAY; > > DECLARE $bad_hosts = { type=dns, url=blacklists.example.org/bad_dns.xml > , refresh_list=30 }; > > DECLARE $bad_ips = { type=ip, > url=more_blacklists.example.org/bad_ips.xml > , refresh_list=30 }; > > DECLARE $our_windows_hosts = { type=ip, > url=http://mycompany.example.org/our_windows_hosts.xml, refresh_list=900 }; > > ## On the HTTP sniffer ## > > # Detect our web apps > > ID $google AS [http,https ] FROM [ host='*.google.com > ' ]; > > ID $google_search_result AS [ $google ] MATCHING [ uri =~ /search/ ]; > > ID $our_web_clients AS [http,https] TO [ $our_windows_hosts ]; > > ID $us_sending_email AS [smtp] FROM [ $our_web_clients ]; > > ID $malware_webapp AS [ http, https ] FROM [ $bad_hosts OR $bad_ips ]; > > # Define our actions > > # Create an event and record a client as attacked if it is referred to a > malware page by a Google search > > FORWARD EVENT IF { $google_search_result REFERS $malware_webapp } AND [ > ADD $_CLIENT TO $attacked, ADD $_SERVER TO $attackers ]; > > The important thing is that our daemon doing the HTTP sniffing is > agnostic in that it has no idea if things are bad or good, it just lets > somebody know when a predefined set of criteria has occurred. It is up > to an event policy server to receive these events and do something > intelligent (and highly configurable) with it. So, we've decoupled any > decision making from the component that has the most time-sensitive work > to do. All of the other components can buffer their work, but the > network sniffer needs as much work lifted from it as possible so that it > can cope with high loads. The policy server components would be the > only parts that would require much customization, and they would be > written in a high-level language like perl for maximum extensibility. > > Let's also not forget the plethora of other apps out there that we can > leverage. Of particular importance would be SANCP and PADS. > > SANCP's newest version contains a feature that John Curry added for me > which writes out an index of the file position within a bulk pcap where > a packet starts and stops. This means that you can do instantaenous > bulk packet retrievals using the index. In my tests, I can pull an > arbitrary host's traffic from a 30 GB pcap in under 5 seconds. This > technique could be used for file carving, which would provide the > ability to send to sandboxes, send to VirusTotal, Symantec, etc. > > PADS provides a ready-made daemon for identifying the protocols used > between hosts. If PADS fed a policy server, we could automagically > update the policy server when new Windows hosts come online, or when an > SMTP session starts. Since the HTTP sniffer would be reading from the > policy server on a regular basis, PADS would be able to keep it informed > as to the state of the network. P0f could work as well. > > So, the main idea would be to have a very lightweight daemon with a > specific task forward events to an event policy server written in a > higher level language, and to remove almost all decision making from the > sniffer. > > Comments? > > Thanks, > > Martin > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From rMslade at shaw.ca Fri Oct 17 17:24:52 2008 From: rMslade at shaw.ca (Rob, grandpa of Ryan, Trevor, Devon & Hannah) Date: Fri, 17 Oct 2008 13:24:52 -0800 Subject: [Discussion] Mascot In-Reply-To: <48F7D855.9020500@jonkmans.com> Message-ID: <48F89224.21487.17611D7@localhost> Date sent: Thu, 16 Oct 2008 20:12:05 -0400 From: Matt Jonkman > 1. A mascot > Gotta have a mascot. The best suggestion we've seen to date I think is > an evil looking beaver. As a Canadian, I can't help but agree. Hardworking, industrious, tail-slap to warn the community of danger, damaging the envrionment under the guise of ... hmmm ... > The analogy being that the new engine is here to > dam up the flood of information and make it useful. Please, not the d-word. Channel, redirect, maybe ... > In the form of > hydroelectric power I guess. Building up pressure and force until somebody is fool enough to touch the barrier ... ====================== (quote inserted randomly by Pegasus Mailer) rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org Ignorance is never out of style. It was in fashion yesterday, it is the rage today, and it will set the pace tomorrow. -- Franklin K. Dane victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ From rMslade at shaw.ca Fri Oct 17 17:41:37 2008 From: rMslade at shaw.ca (Rob, grandpa of Ryan, Trevor, Devon & Hannah) Date: Fri, 17 Oct 2008 13:41:37 -0800 Subject: [Discussion] Features - Paran-o-meter In-Reply-To: <48F7E3B0.20002@jonkmans.com> Message-ID: <48F89611.1054.18567ED@localhost> Date sent: Thu, 16 Oct 2008 21:00:32 -0400 From: Matt Jonkman > 2. IP Reputation Sharing > 5. Scoring Along with both of these (and addressing some of the issues raised in regard to them), user-settable levels of paranoia. (Not too *easily* user-settable, mind you.) ====================== (quote inserted randomly by Pegasus Mailer) rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org What a waste it is to lose one's mind. Or not to have a mind is being very wasteful. How true that is. - Dan Quayle victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ From jonkman at jonkmans.com Fri Oct 17 16:49:06 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 17 Oct 2008 16:49:06 -0400 Subject: [Discussion] Features - Paran-o-meter In-Reply-To: <48F89611.1054.18567ED@localhost> References: <48F89611.1054.18567ED@localhost> Message-ID: <48F8FA42.2040709@jonkmans.com> Good idea. I had something in mind like there being categories for each ip, and an overall average. The user could if they like weight certain categories higher in that average and then make decisions on either that average or certain categories. A list of thresholds essentially and their level of pain acceptable. Similar again to spamassassin, warn at this level, block above this level. But in this case warn would be adding a few points to their alert threshold, sort of putting that IP on probation, slightest wrong move and they're out. And vice versa, a spectacularly good reputation (predefined partners, google, yahoo, etc) would have points reduced from their alert threshold so they'd have to REALLY screw up to get blocked. Make sense? matt Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > Date sent: Thu, 16 Oct 2008 21:00:32 -0400 > From: Matt Jonkman > >> 2. IP Reputation Sharing > >> 5. Scoring > > Along with both of these (and addressing some of the issues raised in regard to > them), user-settable levels of paranoia. (Not too *easily* user-settable, mind you.) > > ====================== (quote inserted randomly by Pegasus Mailer) > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > What a waste it is to lose one's mind. Or not to have a mind is > being very wasteful. How true that is. - Dan Quayle > victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From robert.jamison at bt.com Fri Oct 17 17:13:30 2008 From: robert.jamison at bt.com (robert.jamison@bt.com) Date: Fri, 17 Oct 2008 22:13:30 +0100 Subject: [Discussion] Features and Design Strategy In-Reply-To: <48F8EE46.1090709@jonkmans.com> References: <48F8EE46.1090709@jonkmans.com> Message-ID: <3839DD5BC9EE23459E6FAC536A2805370639D083@E03MVY2-UKDY.domain1.systemhost.net> Great e-mail Martin! Disclaimer: My comments and thoughts are coming from an MSSP perspective, not from a local network security administrator with strong knowledge of and view into a given environment (this is good and bad, I suppose...). Expanding on Martin's comments: " So, while perl would be far > too slow to do real-time content matching, it is certainly not too > slow to receive an alert, check to see if this source IP has alerted > recently, create an incident with some priority, see who needs to be > notified of the incident, and then send an alert." Checking to see if the SIP has alerted recently, and checking for derogatory information on the DIP gives opportunity to consider additional evidence outside the context of the observed session. While a blacklist provides a seed to begin session tracking, it's typically Boolean and doesn't provide historical or concurrent significance in establishing an event priority. The common framework is building on that by more effective session analysis and intelligent classification, resulting in better incident response--I'm not going to add to this as my perspective is much more lateral than focused. One of the projects I've been working on is the analysis and storage of firewall logs from functionally and geographically diverse environments. We are attempting to leverage cross customer firewall accept and deny data as well as security events from IDS/IPS systems which may or may not be on the same networks in which the firewalls exist. This could perhaps be viewed as filling gaps in a protective perimeter by harnessing detection resources that exist elsewhere. Parallelism vs. Serialism. This is to build confidence that a customer client is indeed infected, before reaction policy is even engaged. Even though SIP or client 'Alpha' may only be seen from a single network (which we, as a MSSP typically don't have control over) security alerts from disparate IDS/IPS platforms often implicate the same destination address as being a control node, or hosting malware. Some of the input we aim to feed the incident priority algorithm are: 1) How many other networks have connections to this blacklisted DIP 2) What ports have been accessed 3) What is the historical frequency of access to this DIP from all monitored environments 4) What are the IDS/IPS signatures which have been tripped 5) Is there a common security classification of those sigs 6) Has any forensic investigation of a client that has been in contact with this DIP uncovered a rootkit or malware instance 7) What is the historic significance in IP addresses in the DIP's allocated block (e.g. have other addresses in that block been listed as DIPs in other security alerts within the same or other attack classification type?) I hope to participate more in this project. Thanks, Rob Jamison -----Original Message----- From: discussion-bounces at openinfosecfoundation.org [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt Jonkman Sent: Friday, October 17, 2008 3:58 PM To: Martin Holste Cc: discussion at openinfosecfoundation.org Subject: Re: [Discussion] Features and Design Strategy Well put Martin. I definitely like the idea. It goes well with one of the core things we need to do, let everything thread off away from the flow of packets as possible. So to generalize a bit more, we should strive to make anything possible able to be hooked, or fed out to a side process as possible. Output of course, but anything else as well. Just leave generic hook points any process could pull data from fifo style. This would likely be where we could let the extracted binaries come out of the core processes as well. Great concept to add, I'll get this into the wiki so we don't lose the idea. What challenges does anyone see in this concept? What other advantages might we gain if we plan correctly? Matt Martin Holste wrote: > First off, many congrats, Matt! > > When you look at the threat landscape, it's clear that the most imminent > threats are against web-facing clients and servers, so I think we should > focus on client-side attacks and attacks against web applications. > Detecting SMB protocol attacks, etc. is still important, but most > attacks against non-web protocols need a foothold on the LAN first. > Most IDS are historically built for defending servers from clients, and > I think that idea is becoming a bit antiquated, though certainly far > from obsolete. > > My group has had great success doing what John at Berkeley is/will be > doing with file carving. We've been auto-extracting packed files from > the network stream and auto-submitting to VirusTotal for over a year > now. John's point about processing the layer-7 data inside the streams > is dead-on. We need to move away from answering "what strings did the > traffic contain" to answering "what did the host actually do?" We also > need to keep in mind that canonically defining what "bad" behavior is > can be incredibly difficult even if you're just trying to put it in words. > > The design strategy that it is important to me is plugin extensibility. > One of the reasons I love perl so much is that there is a CPAN module > for just about anything you could ever dream of needing. The extreme > ease with which you can incorporate others' code into your own project > is what has kept it thriving for so long. The differences between > individual organization's security needs and preferences mandate that a > successful project will have a long-term goal of being extremely > modular, while also having a core functionality that is superb. > > Since Snort is what I'm familiar with, I will try to make my point by > comparison: Snort has excellent core functionalilty, but despite > Marty's best efforts, extending it is not a trivial task because the > code is in C. When Jason Brevenik posted his SnortUnified.pm module > which gives perl access to realtime Snort alerts, I realized that there > was suddenly an avenue for decoupling the pure network-grepping core > functionality of Snort with the action component. Moreover, I now had a > hook for writing extremely customized and complicated event handling in > my programming language of choice instead of trying to debug segfaults > in preprocessors all day. So, while perl would be far too slow to do > real-time content matching, it is certainly not too slow to receive an > alert, check to see if this source IP has alerted recently, create an > incident with some priority, see who needs to be notified of the > incident, and then send an alert. So, the point is, we only need the > really fast component to do a limited number of tasks, and if we > decouple the real-time, per-packet tasks from the high level "what do we > do with this" tasks, anyone is free to trivially write their own hooks > into how they want things handled, which is the part that varies the > most from org to org. Marty recognized all of this when he wrote the > unified output module, but Sourcefire never had a commercial reason for > extending it much since it would encroach on their own product > offerings. OISF has the advantage of not being for-profit, and so we > can make extensibility a priority. > > What I would like to see is a brand-new program with functions similar > to Snort's HTTP preprocessor which has the sole purpose in life of > sniffing the network and attaching HTTP header fields to the traffic, > most importantly the host, content-type, and content-length fields. > This daemon would read a config file which specifies rules more like > scripts. The rules would have variables populated from a policy server, > and it would update these variables very regularly (without a program > restart of any kind). This would de-couple the work of sniffing from > the work of policy setting. The policy server would get its variables > from all kinds of sources, like remote XML files. Armed with this, very > simple signatures which identify web applications could be written. > Here's a pseudo-code example of what I think the configs would look like: > > ## On the policy server ## > > # Declare vars > > DECLARE $attacked = ARRAY; > > DECLARE $attackers = ARRAY; > > DECLARE $bad_hosts = { type=dns, url=blacklists.example.org/bad_dns.xml > , refresh_list=30 }; > > DECLARE $bad_ips = { type=ip, > url=more_blacklists.example.org/bad_ips.xml > , refresh_list=30 }; > > DECLARE $our_windows_hosts = { type=ip, > url=http://mycompany.example.org/our_windows_hosts.xml, refresh_list=900 }; > > ## On the HTTP sniffer ## > > # Detect our web apps > > ID $google AS [http,https ] FROM [ host='*.google.com > ' ]; > > ID $google_search_result AS [ $google ] MATCHING [ uri =~ /search/ ]; > > ID $our_web_clients AS [http,https] TO [ $our_windows_hosts ]; > > ID $us_sending_email AS [smtp] FROM [ $our_web_clients ]; > > ID $malware_webapp AS [ http, https ] FROM [ $bad_hosts OR $bad_ips ]; > > # Define our actions > > # Create an event and record a client as attacked if it is referred to a > malware page by a Google search > > FORWARD EVENT IF { $google_search_result REFERS $malware_webapp } AND [ > ADD $_CLIENT TO $attacked, ADD $_SERVER TO $attackers ]; > > The important thing is that our daemon doing the HTTP sniffing is > agnostic in that it has no idea if things are bad or good, it just lets > somebody know when a predefined set of criteria has occurred. It is up > to an event policy server to receive these events and do something > intelligent (and highly configurable) with it. So, we've decoupled any > decision making from the component that has the most time-sensitive work > to do. All of the other components can buffer their work, but the > network sniffer needs as much work lifted from it as possible so that it > can cope with high loads. The policy server components would be the > only parts that would require much customization, and they would be > written in a high-level language like perl for maximum extensibility. > > Let's also not forget the plethora of other apps out there that we can > leverage. Of particular importance would be SANCP and PADS. > > SANCP's newest version contains a feature that John Curry added for me > which writes out an index of the file position within a bulk pcap where > a packet starts and stops. This means that you can do instantaenous > bulk packet retrievals using the index. In my tests, I can pull an > arbitrary host's traffic from a 30 GB pcap in under 5 seconds. This > technique could be used for file carving, which would provide the > ability to send to sandboxes, send to VirusTotal, Symantec, etc. > > PADS provides a ready-made daemon for identifying the protocols used > between hosts. If PADS fed a policy server, we could automagically > update the policy server when new Windows hosts come online, or when an > SMTP session starts. Since the HTTP sniffer would be reading from the > policy server on a regular basis, PADS would be able to keep it informed > as to the state of the network. P0f could work as well. > > So, the main idea would be to have a very lightweight daemon with a > specific task forward events to an event policy server written in a > higher level language, and to remove almost all decision making from the > sniffer. > > Comments? > > Thanks, > > Martin > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc _______________________________________________ Discussion mailing list Discussion at openinfosecfoundation.org http://lists.openinfosecfoundation.org/mailman/listinfo/discussion From mcholste at gmail.com Fri Oct 17 17:56:20 2008 From: mcholste at gmail.com (Martin Holste) Date: Fri, 17 Oct 2008 16:56:20 -0500 Subject: [Discussion] Features and Design Strategy In-Reply-To: <3839DD5BC9EE23459E6FAC536A2805370639D083@E03MVY2-UKDY.domain1.systemhost.net> References: <48F8EE46.1090709@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370639D083@E03MVY2-UKDY.domain1.systemhost.net> Message-ID: Yes, yes, and yes. This is the part where a modular, pluggable architecture lets everyone contribute their various scoring mechanisms and checks. There could be a plugin for your SIP/DIP methodology, a plugin for the Google malware domain checker, to Shadowserver.org, etc., etc. But the key is that we don't have to make a web request on a per-packet basis, rather, we update lists for the sniffer periodically, and we only have to check external sources when we are handed an event from the sniffer. If we buffer the data from the sniffer to the event policy engine queue, then we can afford to do lots and lots of scoring, checking, re-checking, and ranking before a human ever has to wonder what's going on without dropping packets. As an added bonus, the source lists can be updated based on the collective decisions, so we get a positive cycle of pruning the source lists. Thanks for the feedback! I can't wait to all of the cool stuff this group is going to dream up. Martin On Fri, Oct 17, 2008 at 4:13 PM, wrote: > Great e-mail Martin! > > Disclaimer: My comments and thoughts are coming from an MSSP > perspective, not from a local network security administrator with strong > knowledge of and view into a given environment (this is good and bad, I > suppose...). > > Expanding on Martin's comments: > > " So, while perl would be far > > too slow to do real-time content matching, it is certainly not too > > slow to receive an alert, check to see if this source IP has alerted > > recently, create an incident with some priority, see who needs to be > > notified of the incident, and then send an alert." > > Checking to see if the SIP has alerted recently, and checking for > derogatory information on the DIP gives opportunity to consider > additional evidence outside the context of the observed session. While > a blacklist provides a seed to begin session tracking, it's typically > Boolean and doesn't provide historical or concurrent significance in > establishing an event priority. The common framework is building on > that by more effective session analysis and intelligent classification, > resulting in better incident response--I'm not going to add to this as > my perspective is much more lateral than focused. > > One of the projects I've been working on is the analysis and storage of > firewall logs from functionally and geographically diverse environments. > We are attempting to leverage cross customer firewall accept and deny > data as well as security events from IDS/IPS systems which may or may > not be on the same networks in which the firewalls exist. This could > perhaps be viewed as filling gaps in a protective perimeter by > harnessing detection resources that exist elsewhere. Parallelism vs. > Serialism. This is to build confidence that a customer client is > indeed infected, before reaction policy is even engaged. Even though > SIP or client 'Alpha' may only be seen from a single network (which we, > as a MSSP typically don't have control over) security alerts from > disparate IDS/IPS platforms often implicate the same destination address > as being a control node, or hosting malware. Some of the input we aim > to feed the incident priority algorithm are: > > 1) How many other networks have connections to this blacklisted DIP > 2) What ports have been accessed > 3) What is the historical frequency of access to this DIP from all > monitored environments > 4) What are the IDS/IPS signatures which have been tripped > 5) Is there a common security classification of those sigs > 6) Has any forensic investigation of a client that has been in contact > with this DIP uncovered a rootkit or malware instance > 7) What is the historic significance in IP addresses in the DIP's > allocated block (e.g. have other addresses in that block been listed as > DIPs in other security alerts within the same or other attack > classification type?) > > I hope to participate more in this project. > > Thanks, > > > Rob Jamison > > > > -----Original Message----- > From: discussion-bounces at openinfosecfoundation.org > [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt > Jonkman > Sent: Friday, October 17, 2008 3:58 PM > To: Martin Holste > Cc: discussion at openinfosecfoundation.org > Subject: Re: [Discussion] Features and Design Strategy > > Well put Martin. I definitely like the idea. It goes well with one of > the core things we need to do, let everything thread off away from the > flow of packets as possible. > > So to generalize a bit more, we should strive to make anything possible > able to be hooked, or fed out to a side process as possible. Output of > course, but anything else as well. Just leave generic hook points any > process could pull data from fifo style. > > This would likely be where we could let the extracted binaries come out > of the core processes as well. > > Great concept to add, I'll get this into the wiki so we don't lose the > idea. > > What challenges does anyone see in this concept? > > What other advantages might we gain if we plan correctly? > > Matt > > Martin Holste wrote: > > First off, many congrats, Matt! > > > > When you look at the threat landscape, it's clear that the most > imminent > > threats are against web-facing clients and servers, so I think we > should > > focus on client-side attacks and attacks against web applications. > > Detecting SMB protocol attacks, etc. is still important, but most > > attacks against non-web protocols need a foothold on the LAN first. > > Most IDS are historically built for defending servers from clients, > and > > I think that idea is becoming a bit antiquated, though certainly far > > from obsolete. > > > > My group has had great success doing what John at Berkeley is/will be > > doing with file carving. We've been auto-extracting packed files from > > the network stream and auto-submitting to VirusTotal for over a year > > now. John's point about processing the layer-7 data inside the > streams > > is dead-on. We need to move away from answering "what strings did the > > traffic contain" to answering "what did the host actually do?" We > also > > need to keep in mind that canonically defining what "bad" behavior is > > can be incredibly difficult even if you're just trying to put it in > words. > > > > The design strategy that it is important to me is plugin > extensibility. > > One of the reasons I love perl so much is that there is a CPAN module > > for just about anything you could ever dream of needing. The extreme > > ease with which you can incorporate others' code into your own project > > is what has kept it thriving for so long. The differences between > > individual organization's security needs and preferences mandate that > a > > successful project will have a long-term goal of being extremely > > modular, while also having a core functionality that is superb. > > > > Since Snort is what I'm familiar with, I will try to make my point by > > comparison: Snort has excellent core functionalilty, but despite > > Marty's best efforts, extending it is not a trivial task because the > > code is in C. When Jason Brevenik posted his SnortUnified.pm module > > which gives perl access to realtime Snort alerts, I realized that > there > > was suddenly an avenue for decoupling the pure network-grepping core > > functionality of Snort with the action component. Moreover, I now had > a > > hook for writing extremely customized and complicated event handling > in > > my programming language of choice instead of trying to debug segfaults > > in preprocessors all day. So, while perl would be far too slow to do > > real-time content matching, it is certainly not too slow to receive an > > alert, check to see if this source IP has alerted recently, create an > > incident with some priority, see who needs to be notified of the > > incident, and then send an alert. So, the point is, we only need the > > really fast component to do a limited number of tasks, and if we > > decouple the real-time, per-packet tasks from the high level "what do > we > > do with this" tasks, anyone is free to trivially write their own hooks > > into how they want things handled, which is the part that varies the > > most from org to org. Marty recognized all of this when he wrote the > > unified output module, but Sourcefire never had a commercial reason > for > > extending it much since it would encroach on their own product > > offerings. OISF has the advantage of not being for-profit, and so we > > can make extensibility a priority. > > > > What I would like to see is a brand-new program with functions similar > > to Snort's HTTP preprocessor which has the sole purpose in life of > > sniffing the network and attaching HTTP header fields to the traffic, > > most importantly the host, content-type, and content-length fields. > > This daemon would read a config file which specifies rules more like > > scripts. The rules would have variables populated from a policy > server, > > and it would update these variables very regularly (without a program > > restart of any kind). This would de-couple the work of sniffing from > > the work of policy setting. The policy server would get its variables > > from all kinds of sources, like remote XML files. Armed with this, > very > > simple signatures which identify web applications could be written. > > Here's a pseudo-code example of what I think the configs would look > like: > > > > ## On the policy server ## > > > > # Declare vars > > > > DECLARE $attacked = ARRAY; > > > > DECLARE $attackers = ARRAY; > > > > DECLARE $bad_hosts = { type=dns, > url=blacklists.example.org/bad_dns.xml > > , refresh_list=30 }; > > > > DECLARE $bad_ips = { type=ip, > > url=more_blacklists.example.org/bad_ips.xml > > , refresh_list=30 }; > > > > DECLARE $our_windows_hosts = { type=ip, > > url=http://mycompany.example.org/our_windows_hosts.xml, > refresh_list=900 }; > > > > ## On the HTTP sniffer ## > > > > # Detect our web apps > > > > ID $google AS [http,https ] FROM [ host='*.google.com > > ' ]; > > > > ID $google_search_result AS [ $google ] MATCHING [ uri =~ /search/ ]; > > > > ID $our_web_clients AS [http,https] TO [ $our_windows_hosts ]; > > > > ID $us_sending_email AS [smtp] FROM [ $our_web_clients ]; > > > > ID $malware_webapp AS [ http, https ] FROM [ $bad_hosts OR $bad_ips ]; > > > > # Define our actions > > > > # Create an event and record a client as attacked if it is referred to > a > > malware page by a Google search > > > > FORWARD EVENT IF { $google_search_result REFERS $malware_webapp } AND > [ > > ADD $_CLIENT TO $attacked, ADD $_SERVER TO $attackers ]; > > > > The important thing is that our daemon doing the HTTP sniffing is > > agnostic in that it has no idea if things are bad or good, it just > lets > > somebody know when a predefined set of criteria has occurred. It is > up > > to an event policy server to receive these events and do something > > intelligent (and highly configurable) with it. So, we've decoupled > any > > decision making from the component that has the most time-sensitive > work > > to do. All of the other components can buffer their work, but the > > network sniffer needs as much work lifted from it as possible so that > it > > can cope with high loads. The policy server components would be the > > only parts that would require much customization, and they would be > > written in a high-level language like perl for maximum extensibility. > > > > Let's also not forget the plethora of other apps out there that we can > > leverage. Of particular importance would be SANCP and PADS. > > > > SANCP's newest version contains a feature that John Curry added for me > > which writes out an index of the file position within a bulk pcap > where > > a packet starts and stops. This means that you can do instantaenous > > bulk packet retrievals using the index. In my tests, I can pull an > > arbitrary host's traffic from a 30 GB pcap in under 5 seconds. This > > technique could be used for file carving, which would provide the > > ability to send to sandboxes, send to VirusTotal, Symantec, etc. > > > > PADS provides a ready-made daemon for identifying the protocols used > > between hosts. If PADS fed a policy server, we could automagically > > update the policy server when new Windows hosts come online, or when > an > > SMTP session starts. Since the HTTP sniffer would be reading from the > > policy server on a regular basis, PADS would be able to keep it > informed > > as to the state of the network. P0f could work as well. > > > > So, the main idea would be to have a very lightweight daemon with a > > specific task forward events to an event policy server written in a > > higher level language, and to remove almost all decision making from > the > > sniffer. > > > > Comments? > > > > Thanks, > > > > Martin > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/363e5ef7/attachment-0001.html From lists at inliniac.net Fri Oct 17 18:52:35 2008 From: lists at inliniac.net (Victor Julien) Date: Sat, 18 Oct 2008 00:52:35 +0200 Subject: [Discussion] Features In-Reply-To: <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> Message-ID: <48F91733.6050008@inliniac.net> Jeremy wrote: >> 4. Native Hardware acceleration support >> There are a number of hardware acceleration technologies that could be >> more effectively built into the engine from the start, versus the >> back-asswards reverse engineering we have to do now to effectively >> accelerate. >> >> > Really cool idea and would like to see this go forward, but again not > a trivial mission. With hardware acceleration you must factor in > cost. Why do we spend hours hacking the kernel, libpcap, and > applications we use? Easy cause most of us are under budget > constraints and can not afford to implement many of these hardware > accelerators... I know we are going through the 10 Gb issues, and > well buying a Bivio and Network General sniffer to do this is very > painful for us and will break the bank.... that being said I welcome > competitors in this area as I believe competition could cause some of > these hardware acceleration technologies to fall in price. > I've read some stuff about using (Nvidia) GPU's for hw acceleration. There are some papers about it. I also read about using GPU's to speed up ClamAV scanning. So maybe that would be useful to support as 'poormans' hw acceleration... Regards, Victor From jxs7886 at rit.edu Fri Oct 17 21:20:08 2008 From: jxs7886 at rit.edu (Josh Smith) Date: Fri, 17 Oct 2008 21:20:08 -0400 Subject: [Discussion] Hey everyone Message-ID: Hey, just thought I would introduce myself to the discussion group. I'm currently a third year student at RIT double majoring in "Information Security and Forensics" and "Applied Networking and System Administration". I also work remotely during school for iDefense on the malcode team, and work in the office during spring and summer quarters. I look forward to contributing whatever I can to the OISF. -Josh Smith From thorsten.holz at informatik.uni-mannheim.de Sat Oct 18 10:35:39 2008 From: thorsten.holz at informatik.uni-mannheim.de (Thorsten Holz) Date: Sat, 18 Oct 2008 16:35:39 +0200 Subject: [Discussion] Why IDS sucks In-Reply-To: <48F8A041.9040605@jonkmans.com> References: <48F77E92.4090607@support-intelligence.com> <48F78763.5030008@security.berkeley.edu> <48F7989A.1030306@packetspy.com> <48F8A041.9040605@jonkmans.com> Message-ID: <61678C76-D850-4A4E-9077-6210E7FAD8DB@informatik.uni-mannheim.de> On 17.10.2008, at 16:25, Matt Jonkman wrote: > Ya, we tried some things for low ttl dns, but it's used in legit apps > too often. > > Only real way I see to effectively go after fast flux might be dns > blacklisting. Most of them last for more than days, so blacklisting is > relatively effective is we can keep effective sandnetting and intel > gathering going. (Which is what we're trying to do at emerging threats > for the long term) Or use a scoring approach (developed by Robert Danford) similar to what Arbor uses for their fast-flux identification (http:// atlas.arbor.net/summary/fastflux). This seems to work well in practice, you need only a small whitelist for hosts like db.us.big.clamav.net More details in our MALWARE'08 paper: https://honeyblog.org/archives/ 206-MALWARE08-As-the-Net-Churns-Fast-Flux-Botnet-Observations.html Cheers, Thorsten From thorsten.holz at informatik.uni-mannheim.de Sat Oct 18 10:54:24 2008 From: thorsten.holz at informatik.uni-mannheim.de (Thorsten Holz) Date: Sat, 18 Oct 2008 16:54:24 +0200 Subject: [Discussion] Features In-Reply-To: <48F91733.6050008@inliniac.net> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> Message-ID: On 18.10.2008, at 00:52, Victor Julien wrote: > I've read some stuff about using (Nvidia) GPU's for hw acceleration. > There are some papers about it. I also read about using GPU's to speed > up ClamAV scanning. So maybe that would be useful to support as > 'poormans' hw acceleration... Pointers to the papers: "Offloading IDS Computation to the GPU" - http://www.acsac.org/2006/ papers/74.pdf "Gnort: High Performance Network Intrusion Detection Using Graphics Processors" - http://www.ics.forth.gr/dcs/Activities/papers/ gnort.raid08.pdf "A GPU-Based Multiple-Pattern Matching Algorithm for Network Intrusion Detection Systems" - http://ieeexplore.ieee.org/ iel5/4482830/4482831/04482891.pdf?arnumber=4482891 I think the source code for these projects is not (yet?) public, but perhaps the authors can publish their code. Cheers, Thorsten From th3m0nq at gmail.com Sat Oct 18 17:28:02 2008 From: th3m0nq at gmail.com (th3 m0nq) Date: Sat, 18 Oct 2008 16:28:02 -0500 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> Message-ID: Hey all, Just want to introduce myself. My name is Adrian Vogeltanz. My knowledge spans from systems administration to programming as well as how security applies to them. I'm interested in helping out in any way I can. Since this is the ideas portion I guess I will put my 2 cents in. What I personally would like to see in the evolution of IDS/IPS software is something closer to an expert system, but not exactly an expert system. I assume most people here have written tons of snort rules and have an amazing amount of harvested information about "bad" activity. I would like to see a lot of this information standardized and shared in an IDS/IPS solution. I think there is room, at minimum, to have packets that do not necessarily trigger a rule "flagged" in some way in an IDS. This could be implemented as a plug-in component of the system. This may be way outside of the scope of what you guys are looking at, but that's my 2 cents. Adrian On Sat, Oct 18, 2008 at 9:54 AM, Thorsten Holz wrote: > On 18.10.2008, at 00:52, Victor Julien wrote: > >> I've read some stuff about using (Nvidia) GPU's for hw acceleration. >> There are some papers about it. I also read about using GPU's to speed >> up ClamAV scanning. So maybe that would be useful to support as >> 'poormans' hw acceleration... > > Pointers to the papers: > > "Offloading IDS Computation to the GPU" - http://www.acsac.org/2006/ > papers/74.pdf > "Gnort: High Performance Network Intrusion Detection Using Graphics > Processors" - http://www.ics.forth.gr/dcs/Activities/papers/ > gnort.raid08.pdf > "A GPU-Based Multiple-Pattern Matching Algorithm for Network > Intrusion Detection Systems" - http://ieeexplore.ieee.org/ > iel5/4482830/4482831/04482891.pdf?arnumber=4482891 > > I think the source code for these projects is not (yet?) public, but > perhaps the authors can publish their code. > > Cheers, > Thorsten > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From famousjs at gmail.com Sat Oct 18 17:40:14 2008 From: famousjs at gmail.com (Josh Smith) Date: Sat, 18 Oct 2008 17:40:14 -0400 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> Message-ID: What about the technique people have used to harness GPU power to crack WPA2 with parallel processing? This might be open source. -Josh Smith On Sat, Oct 18, 2008 at 10:54 AM, Thorsten Holz wrote: > On 18.10.2008, at 00:52, Victor Julien wrote: > >> I've read some stuff about using (Nvidia) GPU's for hw acceleration. >> There are some papers about it. I also read about using GPU's to speed >> up ClamAV scanning. So maybe that would be useful to support as >> 'poormans' hw acceleration... > > Pointers to the papers: > > "Offloading IDS Computation to the GPU" - http://www.acsac.org/2006/ > papers/74.pdf > "Gnort: High Performance Network Intrusion Detection Using Graphics > Processors" - http://www.ics.forth.gr/dcs/Activities/papers/ > gnort.raid08.pdf > "A GPU-Based Multiple-Pattern Matching Algorithm for Network > Intrusion Detection Systems" - http://ieeexplore.ieee.org/ > iel5/4482830/4482831/04482891.pdf?arnumber=4482891 > > I think the source code for these projects is not (yet?) public, but > perhaps the authors can publish their code. > > Cheers, > Thorsten > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From frank at knobbe.us Sat Oct 18 18:01:42 2008 From: frank at knobbe.us (Frank Knobbe) Date: Sat, 18 Oct 2008 17:01:42 -0500 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> Message-ID: <1224367303.13106.17.camel@localhost> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: > [..] I think there is > room, at minimum, to have packets that do not necessarily trigger a > rule "flagged" in some way in an IDS. This could be implemented as a > plug-in component of the system. This may be way outside of the scope > of what you guys are looking at, but that's my 2 cents. Receiving alerts (with session content) on IP's that don't trip existing signatures, but are present in a hostile-IP list, is a good idea (for example, to detect new evasion techniques of existing badware). However, I don't think the sensor is the proper place for that due to the immense IP-matching workload that would have to occur on a per-packet or per-session basis (remember, we're talking at least tens of thousands hostile IP's). As Robert and Martin eluded in an other thread, any sort of correlation and IP analysis, scoring, severity-bumping, etc, is best done in the analysis or portal engine. (We do those things in our shop as well, with back-end scripts and our portal). While you can make decisions on the IP "badness value" for analysis, you would of course miss the content of the IP not receiving an alert. I don't think any existing detection engine of an IDS is the proper place. But perhaps a parallel engine that collects network stream data and does matching on hostile IP's, and then checks (perhaps by packet/session timestamp) if the IDS did produce an alert in a recent time window (since the session-IP-eval-engine will surely lag behind). That check would probably be best done against the IDS/portal database. A simple bpf filter with hostile 20,000+ IP's to capture content will probably not cut it :) -Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081018/14dbc91a/attachment.bin From rMslade at shaw.ca Sat Oct 18 21:09:13 2008 From: rMslade at shaw.ca (Rob, grandpa of Ryan, Trevor, Devon & Hannah) Date: Sat, 18 Oct 2008 17:09:13 -0800 Subject: [Discussion] What are we making? In-Reply-To: <48F8AE47.8090603@jonkmans.com> Message-ID: <48FA1839.14684.23E77B8@localhost> Question: what are we making? Oh, yeah, I read the blurb: "The OISF has been chartered and funded to build a next-generation intrusion detection and prevention engine. This project will consider every new and existing technology, concept and idea to build a completely open source licensed engine." OK, we're making an IDS. But I think we need to be more specific. In particular, we need to answer the question of "who." Since the DHS has provided money, I suspect there would be an automatic assumption that this is a heavy-duty device intended for use to protect major servers and nodes in the critical information infrastructure. (Whatever that means.) This kind of thing is built by professionals, for professionals. Trained people. However, given the current computing environment, I think it would be relatively easy to make a case that such a device is not going to do all that much good. That a more accessible device, intended for "Grannyx" users, would actually do more to protect the infrastructure. After all, it isn't major nodes on the net that make up botnets, it's the little guys. Protect them, and you reduce the threat. This is the "low hanging fruit" for the blackhats, so protecting that crop is going to give us the greatest benefit for the commitment of resources. This makes a difference. Not, perhaps, in terms of questions about multithreading analysis streams using graphics co-processors. But certainly in most other areas. We've talked about extensibility. If we create a standard template or format for signatures, the "who" makes a difference. Professionals need a warning and a packet. Grannyx users need a warning, no packet, a clear explanation of what and how important, and a recommended course of action. Makes a difference to the template. In terms of my recommendation of a paran-o-meter, it makes a difference. Actually, I see huge debates over initial settings: do we keep it low to keep from crying wolf, or keep it high to keep people as safe as possible. But one thing that should be done is make the paranoia settings not-quite-obvious up front, so that somebody needs to know a little about the implications before they start fiddling with settings. Heck, if it's a professional device, we don't need to worry about the interface at all. If it's for Granny, we definitely do. It also makes a difference in terms of the technology to be included. If it is for professionals, we can throw in everything. If for Granny, we need to make a careful choice about maximum protection for minimum performance drain. ====================== (quote inserted randomly by Pegasus Mailer) rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org I'm getting so absent-minded that sometimes in the middle of victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ From th3m0nq at gmail.com Sat Oct 18 22:17:16 2008 From: th3m0nq at gmail.com (th3 m0nq) Date: Sat, 18 Oct 2008 21:17:16 -0500 Subject: [Discussion] Features In-Reply-To: <1224367303.13106.17.camel@localhost> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> Message-ID: As Robert and Martin eluded in an other thread, any sort of correlation and IP analysis, scoring, severity-bumping, etc, is best done in the analysis or portal engine. I agree here, but does next generation mean an advanced version of snort? Does next generation mean an evolution in terms of thinking about IDS/IPS? Do we leave things as they are and say that there exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system for my network? I would like to see an evolution of such technologies be more biological in nature. I don't think any existing detection engine of an IDS is the proper place. But perhaps a parallel engine that collects network stream data and does matching on hostile IP's, and then checks (perhaps by packet/session timestamp) if the IDS did produce an alert in a recent time window (since the session-IP-eval-engine will surely lag behind) I am with you here as well. Why is the parallel engine not part of the IDS? Why should next generation IDS mean a better snort? Why can't it mean a better way of looking at network security? http://en.wikipedia.org/wiki/Immune_system#Layered_defense What about an all encompassing IDS/IPS _system_? I am not saying I have the answers, or something similar to an expert system is the answer. I am saying I think the issue should be looked at as we dump the current way we do things in a temporary holding cell, and think in evolutionary terms. This makes 4 cents ;-). Adrian On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe wrote: > On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: >> [..] I think there is >> room, at minimum, to have packets that do not necessarily trigger a >> rule "flagged" in some way in an IDS. This could be implemented as a >> plug-in component of the system. This may be way outside of the scope >> of what you guys are looking at, but that's my 2 cents. > > Receiving alerts (with session content) on IP's that don't trip existing > signatures, but are present in a hostile-IP list, is a good idea (for > example, to detect new evasion techniques of existing badware). > > However, I don't think the sensor is the proper place for that due to > the immense IP-matching workload that would have to occur on a > per-packet or per-session basis (remember, we're talking at least tens > of thousands hostile IP's). As Robert and Martin eluded in an other > thread, any sort of correlation and IP analysis, scoring, > severity-bumping, etc, is best done in the analysis or portal engine. > (We do those things in our shop as well, with back-end scripts and our > portal). > > While you can make decisions on the IP "badness value" for analysis, you > would of course miss the content of the IP not receiving an alert. I > don't think any existing detection engine of an IDS is the proper place. > But perhaps a parallel engine that collects network stream data and does > matching on hostile IP's, and then checks (perhaps by packet/session > timestamp) if the IDS did produce an alert in a recent time window > (since the session-IP-eval-engine will surely lag behind). That check > would probably be best done against the IDS/portal database. A simple > bpf filter with hostile 20,000+ IP's to capture content will probably > not cut it :) > > -Frank > > > -- > It is said that the Internet is a public utility. As such, it is best > compared to a sewer. A big, fat pipe with a bunch of crap sloshing > against your ports. > > From aludwig at packetspy.com Sat Oct 18 22:31:49 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Sat, 18 Oct 2008 22:31:49 -0400 Subject: [Discussion] What are we making? In-Reply-To: <48FA1839.14684.23E77B8@localhost> References: <48FA1839.14684.23E77B8@localhost> Message-ID: <48FA9C15.3040505@packetspy.com> I doubt the intent of the DHS is to simply do good, they are most likely much more focused on producing technology that allows them to detect/mitigate/prevent attacks against critical components. This of course means detecting attacks that fly below the threshold of detection for todays technology. If it comes to "doing good" or detecting state sponsored attacks against critical components (think custom attacks against unknown vulns), i'm going to go out on a limb and say they would rather protect the critical component vs the enduser. What you are discussing still has value and merit but im not so sure it is what should be focused on, but of course I am not the person to decide such things. Andre Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > Question: what are we making? Oh, yeah, I read the blurb: "The OISF has been > chartered and funded to build a next-generation intrusion detection and prevention > engine. This project will consider every new and existing technology, concept and > idea to build a completely open source licensed engine." > > OK, we're making an IDS. But I think we need to be more specific. In particular, > we need to answer the question of "who." > > Since the DHS has provided money, I suspect there would be an automatic > assumption that this is a heavy-duty device intended for use to protect major > servers and nodes in the critical information infrastructure. (Whatever that > means.) This kind of thing is built by professionals, for professionals. Trained > people. > > However, given the current computing environment, I think it would be relatively > easy to make a case that such a device is not going to do all that much good. That > a more accessible device, intended for "Grannyx" users, would actually do more to > protect the infrastructure. After all, it isn't major nodes on the net that make up > botnets, it's the little guys. Protect them, and you reduce the threat. This is the > "low hanging fruit" for the blackhats, so protecting that crop is going to give us > the greatest benefit for the commitment of resources. > > This makes a difference. Not, perhaps, in terms of questions about multithreading > analysis streams using graphics co-processors. But certainly in most other areas. > > We've talked about extensibility. If we create a standard template or format for > signatures, the "who" makes a difference. Professionals need a warning and a > packet. Grannyx users need a warning, no packet, a clear explanation of what and > how important, and a recommended course of action. Makes a difference to the > template. > > In terms of my recommendation of a paran-o-meter, it makes a difference. > Actually, I see huge debates over initial settings: do we keep it low to keep from > crying wolf, or keep it high to keep people as safe as possible. But one thing that > should be done is make the paranoia settings not-quite-obvious up front, so that > somebody needs to know a little about the implications before they start fiddling > with settings. > > Heck, if it's a professional device, we don't need to worry about the interface at > all. If it's for Granny, we definitely do. > > It also makes a difference in terms of the technology to be included. If it is for > professionals, we can throw in everything. If for Granny, we need to make a > careful choice about maximum protection for minimum performance drain. > > ====================== (quote inserted randomly by Pegasus Mailer) > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > I'm getting so absent-minded that sometimes in the middle of > victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From hall.692 at osu.edu Sun Oct 19 00:25:34 2008 From: hall.692 at osu.edu (Seth Hall) Date: Sun, 19 Oct 2008 00:25:34 -0400 Subject: [Discussion] I think everyone is describing Bro Message-ID: Sorry, I just joined the list so I'm going to be doing some odd quoting from the list archive :) I do want to point out too, that I'm not writing this email to downplay OISFs goals but rather to hopefully guide OISF toward improving an existing opensource project (Bro - http://www.bro-ids.org/ ) that already does much of what is being discussed on this list. Andre Ludwig wrote: > 1. Signature matching has its use but can not be an end all for detection of attacks. Completely agreed and to be honest, at OSU (The Ohio State University) we really don't run many signatures with Bro. The only signature that we get any real use out of is one that matches remote Windows shells on arbitrary ports. We found that the scripts Bro ships with and the scripts that I was writing were detecting all of the activity that we were interested in. It has continued that it's usually easy to write scripts to detect new activity that we might be interested in. > 2. Performance and wire speed is always going to be an issue Agreed. The main Bro developers are already working on several techniques to scale Bro's performance. There is the cluster deployment which already works (http://www.icir.org/robin/bro-cluster/README.html ). We (OSU) run a cluster deployment of 21 worker nodes currently and tests have been done that show that it scales linearly to 24 nodes. Bro has a communications framework so that the cluster workers can communicate with each other too. This is useful for the times when you need some information shared around to all of the workers (because they each only see their portion of all traffic) like when one host identifies a host that is SSH scanning/bruteforcing, it can let all of the other nodes know about that ip address automatically. The best part is that it's all scalable with commodity hardware, there isn't any non-commodity hardware in our entire cluster. :) The Bro developers have also begun to parallelize their core engine and once that is working (not yet) Bro users will see even greater performance increases if they're running multi core/processor machines. David Glosser wrote: > ways to get alerts for fast-flux and other things where current sigs are prone to FPs I implemented the algorithm behind the Holz et al paper in just a day or two after the paper was released. It was all done in the Bro scripting language. To be fair, the guy that did the snort preprocessor (Scott Campbell) for detecting fastflux also wrote a Bro script for detecting fastflux at the same time as me. :) Matt Jonkman wrote: > We can't give huge lists of bad IPs to most tools, we can't feed behavior patterns to existing tools Bro is a perfect fit for this use case. It can take fairly gigantic lists of IP addresses and check for membership in the lists very frequently without suffering performance-wise. > 1. Native multithreading. Like I wrote earlier, that's being worked on now. :) > 2. IP Reputation Sharing It's only waiting for someone to extend the Bro communications framework to do inter-site communications. There isn't currently any way to specify what you want to receive from remote sites or how much you trust them. It would be great project for someone to pick up. > 3. Native ipv6 Done. Needs more real world testing though, since high volume real world testing with IPv6 is a little difficult to come by still. > 5. Scoring I can definitely imagine implementing something like this with Bro, but you might realize that this isn't quite as important as you thought once you move away from the IDS==signatures mindset. Jeremy wrote: > Network intelligence through passive identification. > a windows xp home > computer correlated with network flow data and port data could easily > be alerted on for any outgoing port 25 connections That's a great idea Jeremy! There's no reason it couldn't be implemented in Bro too. I might do that soon, it would be useful for us. Andre Ludwig wrote: >7. Ability to look across packets in a stream, be it 3 packets before or 4 packets ahead. Bro doesn't tend to work at the individual packet level, so this is not really an issue. > 8. Some sort of meta language needs to be created that easily and effectively can communicate any applications "state". If I'm understanding correctly, you want a language that allows you to program your analysis? Here are the docs for the Bro policy language: http://www.bro-ids.org/wiki/index.php/Reference_Manual John Ives wrote: > 9. The ability to pull files out of the stream in real-time. There is a new script in Bro 1.4 that was just released on Friday. This is from the changelog... - The new script http-extract-items.bro extracts the items from HTTP traffic into individual files (Vern Paxson). At OSU, we have also been running a script I wrote called http- identified-files.bro (included with Bro 1.4) that identifies file types on all http transfers with libmagic. The result is that we have a log of every Windows and UNIX executable that is downloaded in or out of our network over HTTP (on any port and regardless of URL). Bro can identify any file type that the libmagic database you are using can identify. > a language/protocol for it to interact with other security devices Bro already has this, it's called Broccoli and it's included with the main Bro distribution. Here are the docs: http://www.bro-ids.org/broccoli/index.html As some examples of what the library can do, I'm using it in two ways currently. 1. I'm pushing authentication data from a number of different OSU services into our Bro cluster so that it can be analyzed along with the network traffic. We get the benefit of continuing to do our anomaly detection in a single language that is very well suited for detecting odd activity over time. 2. I just released a tool that connects directly to my master node of my cluster and logs data from the cluster directly to a postgresql database in realtime. There is no hacky file tailing being done. Bro sends an event to my tool it logs the data in the message to the database. Here's the bro-dblogger: http://github.com/sethhall/bro-dblogger/tree/master There are also currently functioning bindings for python and ruby. Python bindings are included in the Bro distribution now I believe and the Ruby bindings can be installed with the "broccoli" gem. Victor Julien wrote: > For example in ModSecurity you can match on the password on HTTP basic > auth by capturing the string containing the username and password from > the HTTP header, splitting it, decoding it's base64 This is absolutely possible today with Bro. If you're interested in it for some reason, I could write the script to show how straight forward it is. Whew, sorry about the long email but I hadn't seen a reference to Bro in the archives yet and there are so many suggestions that are already possible in Bro or would be easy to implement that I couldn't pass it up. I'd certainly be willing to answer any questions people have about Bro. It's getting late here and I might not make much sense in part of my email. :) .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From jlewis at packetnexus.com Sun Oct 19 01:06:38 2008 From: jlewis at packetnexus.com (Jason Lewis) Date: Sun, 19 Oct 2008 01:06:38 -0400 Subject: [Discussion] I think everyone is describing Bro In-Reply-To: References: Message-ID: <48FAC05E.4010703@packetnexus.com> Food for thought.... would efforts be better spent developing a common format for these types of devices to talk together? The IDMEF was an attempt, but I haven't seen a standard come out. http://www.ietf.org/rfc/rfc4765.txt jas From mcholste at gmail.com Sun Oct 19 02:16:45 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 01:16:45 -0500 Subject: [Discussion] What are we making? In-Reply-To: <48FA9C15.3040505@packetspy.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> Message-ID: I'm guessing that the intent is to be the "detection" component of the Trusted Internet Connection (TIC) ( http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), in which the mandate is to protect all federal assets, especially the federal desktop environment. Even if DHS has different goals, I think that this group could do a great service by providing the TIC's protection geared toward the client-side, and I think for most sectors, that's where the most imminent threats lie. Again, I'm certainly not saying that defending servers is out of scope, just that IDS has historically been about defending servers as the primary goal, with clients secondary, and I think that needs to be reversed now. --Martin On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig wrote: > I doubt the intent of the DHS is to simply do good, they are most likely > much more focused on producing technology that allows them to > detect/mitigate/prevent attacks against critical components. This of > course means detecting attacks that fly below the threshold of detection > for todays technology. If it comes to "doing good" or detecting state > sponsored attacks against critical components (think custom attacks > against unknown vulns), i'm going to go out on a limb and say they would > rather protect the critical component vs the enduser. > > What you are discussing still has value and merit but im not so sure it > is what should be focused on, but of course I am not the person to > decide such things. > > Andre > > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > > Question: what are we making? Oh, yeah, I read the blurb: "The OISF has > been > > chartered and funded to build a next-generation intrusion detection and > prevention > > engine. This project will consider every new and existing technology, > concept and > > idea to build a completely open source licensed engine." > > > > OK, we're making an IDS. But I think we need to be more specific. In > particular, > > we need to answer the question of "who." > > > > Since the DHS has provided money, I suspect there would be an automatic > > assumption that this is a heavy-duty device intended for use to protect > major > > servers and nodes in the critical information infrastructure. (Whatever > that > > means.) This kind of thing is built by professionals, for professionals. > Trained > > people. > > > > However, given the current computing environment, I think it would be > relatively > > easy to make a case that such a device is not going to do all that much > good. That > > a more accessible device, intended for "Grannyx" users, would actually do > more to > > protect the infrastructure. After all, it isn't major nodes on the net > that make up > > botnets, it's the little guys. Protect them, and you reduce the threat. > This is the > > "low hanging fruit" for the blackhats, so protecting that crop is going > to give us > > the greatest benefit for the commitment of resources. > > > > This makes a difference. Not, perhaps, in terms of questions about > multithreading > > analysis streams using graphics co-processors. But certainly in most > other areas. > > > > We've talked about extensibility. If we create a standard template or > format for > > signatures, the "who" makes a difference. Professionals need a warning > and a > > packet. Grannyx users need a warning, no packet, a clear explanation of > what and > > how important, and a recommended course of action. Makes a difference to > the > > template. > > > > In terms of my recommendation of a paran-o-meter, it makes a difference. > > Actually, I see huge debates over initial settings: do we keep it low to > keep from > > crying wolf, or keep it high to keep people as safe as possible. But one > thing that > > should be done is make the paranoia settings not-quite-obvious up front, > so that > > somebody needs to know a little about the implications before they start > fiddling > > with settings. > > > > Heck, if it's a professional device, we don't need to worry about the > interface at > > all. If it's for Granny, we definitely do. > > > > It also makes a difference in terms of the technology to be included. If > it is for > > professionals, we can throw in everything. If for Granny, we need to > make a > > careful choice about maximum protection for minimum performance drain. > > > > ====================== (quote inserted randomly by Pegasus Mailer) > > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > > I'm getting so absent-minded that sometimes in the middle of > > victoria.tc.ca/techrev/rms.htm > blogs.securiteam.com/index.php/archives/author/p1/ > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/2d472e50/attachment.html From thorsten.holz at informatik.uni-mannheim.de Sun Oct 19 05:47:13 2008 From: thorsten.holz at informatik.uni-mannheim.de (Thorsten Holz) Date: Sun, 19 Oct 2008 11:47:13 +0200 Subject: [Discussion] I think everyone is describing Bro In-Reply-To: References: Message-ID: On 19.10.2008, at 06:25, Seth Hall wrote: > Sorry, I just joined the list so I'm going to be doing some odd > quoting from the list archive :) I do want to point out too, that I'm > not writing this email to downplay OISFs goals but rather to hopefully > guide OISF toward improving an existing opensource project (Bro - > http://www.bro-ids.org/ > ) that already does much of what is being discussed on this list. I agree with Seth: why implement something completely new when you can extend an existing project that already contains many features that should be included in the resulting tool? Bro has some cool features and could be considered as a starting point. Cheers, Thorsten From jonkman at jonkmans.com Sun Oct 19 11:27:05 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 11:27:05 -0400 Subject: [Discussion] I think everyone is describing Bro In-Reply-To: References: Message-ID: <48FB51C9.5030709@jonkmans.com> I definitely agree. There are a great number of features in bro that we can learn from or steal. :) Matt Thorsten Holz wrote: > On 19.10.2008, at 06:25, Seth Hall wrote: > >> Sorry, I just joined the list so I'm going to be doing some odd >> quoting from the list archive :) I do want to point out too, that I'm >> not writing this email to downplay OISFs goals but rather to hopefully >> guide OISF toward improving an existing opensource project (Bro - >> http://www.bro-ids.org/ >> ) that already does much of what is being discussed on this list. > > I agree with Seth: why implement something completely new when you > can extend an existing project that already contains many features > that should be included in the resulting tool? Bro has some cool > features and could be considered as a starting point. > > Cheers, > Thorsten > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Oct 19 11:31:30 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 11:31:30 -0400 Subject: [Discussion] Output Format In-Reply-To: <48FAC05E.4010703@packetnexus.com> References: <48FAC05E.4010703@packetnexus.com> Message-ID: <48FB52D2.7070909@jonkmans.com> I think we may need something like that. Especially if we're moving toward both IP reputation/history sharing and having a lot of output points in the engine. I've also seen JSONs lately for describing bad things in malware. Is anyone aware of other standards we ought to consider or learn from? Matt Jason Lewis wrote: > Food for thought.... would efforts be better spent developing a common > format for these types of devices to talk together? The IDMEF was an > attempt, but I haven't seen a standard come out. > > http://www.ietf.org/rfc/rfc4765.txt > > jas > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Oct 19 12:17:20 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 12:17:20 -0400 Subject: [Discussion] What are we making? -- FUNDING In-Reply-To: <48FA1839.14684.23E77B8@localhost> References: <48FA1839.14684.23E77B8@localhost> Message-ID: <48FB5D90.2060709@jonkmans.com> Very good question. Comments inline: Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > Question: what are we making? > OK, we're making an IDS. But I think we need to be more specific. In particular, > we need to answer the question of "who." > > Since the DHS has provided money, I suspect there would be an automatic > assumption that this is a heavy-duty device intended for use to protect major > servers and nodes in the critical information infrastructure. (Whatever that > means.) This kind of thing is built by professionals, for professionals. Trained > people. I should explain how we got to this point and in our funding. It'll shed a lot of light. A year or more ago Victor, Will and I had the idea that we wanted to do more with Snort than we could. Not just faster or more complex signatures, but very different ideas. Like what we've all been talking about on the list to date. We wanted to modify Snort originally, but contributing to it is a significant challenge, as these would be a major change to it's development goals. Now snort 3 is on the table. Snort and Snort 3 are Sourcefire products, and they SHOULD be. SF dumps huge money into development and they need that money to buy them a tool that will 100% satisfy their commercial needs. Which it should. SF is a company and has always been generous to the community. But they don't owe us anything, nor we them (unless you're a customer of theirs, then you may owe them something :) ). Very few of the things we wanted to go after were going to be immediate, or possibly ever, commercially interesting. What I mean by "commercially interesting" is that if you invest 500k in coding feature X there will be a 10 million dollar return selling it over 5 years. Thus the 500k investment is worthwhile considering the risk of failing to produce feature X. So we sat on the ideas for a bit not having the means to go forward, but eventually when the emerging threats grants came through to keep that project alive we got to know the folks that had money to grant out. Eventually we got our ideas heard by our guy at DHS. He liked the ideas and here we go. (a year later after some "interesting" bureaucracy :) ) I want to make clear what this grant is. Knowing how we got here helps, but knowing what the grant is for is more important. That blurb in the press release "The OISF has been chartered and funded to build a next-generation intrusion detection and prevention engine" is true. And that's about all we were chartered to do! Literally. That's a cut and paste from the paperwork. Let me explain that further. We weren't contacted by DHS to go out and build product X to satisfy a need for Einstein II, or any other gov't tool. We were not asked to go out and explore the feasibility of technology or idea X by DHS and bring back a paper on it. We approached DHS with our ideas. We talked to only one person there, a very forward thinking program manager with a research grant budget he is supposed to apply to good things. He liked our ideas, didn't modify or change them, and has set us loose with the budget we asked for! Sorry for the long winded way to get here, but I thought it important that the community understands the intent of DHS. Their intent was to let us as a community explore these ideas and produce a new way to do approach network security. If the resulting engine helps DHS (and other gov't agencies) I'm sure they'll adopt it. But we're not building anything specifically FOR them, or anyone. We've got resources to use for us, the community in general. Our mandate is essentially to "go do something good, send in your receipts when you're done". I hope that helps explain what we're up to, and why it's so important that we get all of our ideas out there. Making this a separate thread as I'd like to talk about other things as well... Matt > > However, given the current computing environment, I think it would be relatively > easy to make a case that such a device is not going to do all that much good. That > a more accessible device, intended for "Grannyx" users, would actually do more to > protect the infrastructure. After all, it isn't major nodes on the net that make up > botnets, it's the little guys. Protect them, and you reduce the threat. This is the > "low hanging fruit" for the blackhats, so protecting that crop is going to give us > the greatest benefit for the commitment of resources. > > This makes a difference. Not, perhaps, in terms of questions about multithreading > analysis streams using graphics co-processors. But certainly in most other areas. > > We've talked about extensibility. If we create a standard template or format for > signatures, the "who" makes a difference. Professionals need a warning and a > packet. Grannyx users need a warning, no packet, a clear explanation of what and > how important, and a recommended course of action. Makes a difference to the > template. > > In terms of my recommendation of a paran-o-meter, it makes a difference. > Actually, I see huge debates over initial settings: do we keep it low to keep from > crying wolf, or keep it high to keep people as safe as possible. But one thing that > should be done is make the paranoia settings not-quite-obvious up front, so that > somebody needs to know a little about the implications before they start fiddling > with settings. > > Heck, if it's a professional device, we don't need to worry about the interface at > all. If it's for Granny, we definitely do. > > It also makes a difference in terms of the technology to be included. If it is for > professionals, we can throw in everything. If for Granny, we need to make a > careful choice about maximum protection for minimum performance drain. > > ====================== (quote inserted randomly by Pegasus Mailer) > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > I'm getting so absent-minded that sometimes in the middle of > victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Oct 19 12:32:19 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 12:32:19 -0400 Subject: [Discussion] What are we making? - Target User In-Reply-To: <48FA1839.14684.23E77B8@localhost> References: <48FA1839.14684.23E77B8@localhost> Message-ID: <48FB6113.6070806@jonkmans.com> Splitting to s second thread as there are many good ideas here: Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > However, given the current computing environment, I think it would be relatively > easy to make a case that such a device is not going to do all that much good. That > a more accessible device, intended for "Grannyx" users, would actually do more to > protect the infrastructure. After all, it isn't major nodes on the net that make up > botnets, it's the little guys. Protect them, and you reduce the threat. This is the > "low hanging fruit" for the blackhats, so protecting that crop is going to give us > the greatest benefit for the commitment of resources. Very well put. We've always left the home user to fend for themselves because it's just too complicated to run IDS unless you're a security professional. Thus in the botnet world we chase command and control servers and leave the bots infected. Not the best approach. So if we were to make this tool capable of being run "out of the box" as a simple install and it'll do the rest on it's own, what would that mean? Would we need it to run on a WRTG style router OS? Would we need to approach the home router makers about a plugin? Would we want to go desktop stuff? (not my preference as the fox can't be trusted to watch the henhouse IMHO) Or do we go with just pushing reputational data to the home user? What I mean is if we build this engine to generate and act upon IP reputation data could we know enough about the Internet collectively to simply push a blacklist to the home user's router/firewall? On the more sophisticated devices where software could be installed maybe it does run a stripped down detection engine and help feed IP data back to the group. But overall it's still primarily benefiting only from the blacklisting and whitelisting of the whole? How many false positives would we encounter that might actually affect a home user? I think it'd be a very interesting day if we were to have essentially a Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even more IP mobile than they are now. Take atrivo/intercare/mccolo for example. Infested with crap, and have been for years. But since they can't really be blocked on the backbone home users still hit the same scam AV sites, give their credit card info, and get screwed. We know the sites are there, the registrars won't take them down, the ISP is colluding with the bad guys so they'll stay online. What can we do? (besides scream to our representatives for more effective laws) We can block those bad IPs at the home user's level. That'll make them start moving of course, just like bots being used to spam until they're listed. So we have to be able to immediately move quickly with the. What does everyone think there? The basic idea being to use a normal engine model by most security pro's to feed IP reputation into a global database, and then the home user gets some sort of very basic tool or button they can click on to benefit from that data? Maybe even feed back to us. > In terms of my recommendation of a paran-o-meter, it makes a difference. > Actually, I see huge debates over initial settings: do we keep it low to keep from > crying wolf, or keep it high to keep people as safe as possible. But one thing that > should be done is make the paranoia settings not-quite-obvious up front, so that > somebody needs to know a little about the implications before they start fiddling > with settings. > > Heck, if it's a professional device, we don't need to worry about the interface at > all. If it's for Granny, we definitely do. Agreed. I don't think we can satisfy any of the needs of either granny or us in the same tool. It'd either be too dumbed down for us, or too complex for granny. I don't see any middle ground personally. I like the home router plugin thing though. If it could feed back to us what IPs it was blocking we'd learn a lot! Matt > > It also makes a difference in terms of the technology to be included. If it is for > professionals, we can throw in everything. If for Granny, we need to make a > careful choice about maximum protection for minimum performance drain. > > ====================== (quote inserted randomly by Pegasus Mailer) > rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org > I'm getting so absent-minded that sometimes in the middle of > victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Oct 19 12:36:05 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 12:36:05 -0400 Subject: [Discussion] What are we making? In-Reply-To: <48FA9C15.3040505@packetspy.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> Message-ID: <48FB61F5.8020107@jonkmans.com> Andre Ludwig wrote: > I doubt the intent of the DHS is to simply do good, they are most likely > much more focused on producing technology that allows them to > detect/mitigate/prevent attacks against critical components. Oh so cynical. :) I'd have said the same thing a year ago. I can't say enough good things about our DHS representative. Forward thinking, willing to trust, and understands the open source community and what's possible when we have the resources. He may be the only guy there that thinks this way, but he's OUR guy. Glad these questions were asked. I'd have never believed what we've gotten here with DHS were possible a year ago. But I've got the paperwork to prove it!!! Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From famousjs at gmail.com Sun Oct 19 12:43:00 2008 From: famousjs at gmail.com (Josh Smith) Date: Sun, 19 Oct 2008 12:43:00 -0400 Subject: [Discussion] What are we making? - Target User In-Reply-To: <48FB6113.6070806@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FB6113.6070806@jonkmans.com> Message-ID: The home router plugin seems like a great idea. That way the majority of people would be able to use existing hardware they have and not have to buy a separate machine just to put in between them and the internet. This wouldn't be the full product in my mind, just a small firmware upgrade like DDWRT or something like that. But it would be highly configurable with options such as remote logging back to your machine. -Josh On Sun, Oct 19, 2008 at 12:32 PM, Matt Jonkman wrote: > Splitting to s second thread as there are many good ideas here: > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: >> However, given the current computing environment, I think it would be relatively >> easy to make a case that such a device is not going to do all that much good. That >> a more accessible device, intended for "Grannyx" users, would actually do more to >> protect the infrastructure. After all, it isn't major nodes on the net that make up >> botnets, it's the little guys. Protect them, and you reduce the threat. This is the >> "low hanging fruit" for the blackhats, so protecting that crop is going to give us >> the greatest benefit for the commitment of resources. > > Very well put. We've always left the home user to fend for themselves > because it's just too complicated to run IDS unless you're a security > professional. Thus in the botnet world we chase command and control > servers and leave the bots infected. Not the best approach. > > So if we were to make this tool capable of being run "out of the box" as > a simple install and it'll do the rest on it's own, what would that mean? > > Would we need it to run on a WRTG style router OS? > > Would we need to approach the home router makers about a plugin? > > Would we want to go desktop stuff? (not my preference as the fox can't > be trusted to watch the henhouse IMHO) > > Or do we go with just pushing reputational data to the home user? What I > mean is if we build this engine to generate and act upon IP reputation > data could we know enough about the Internet collectively to simply push > a blacklist to the home user's router/firewall? > > On the more sophisticated devices where software could be installed > maybe it does run a stripped down detection engine and help feed IP data > back to the group. But overall it's still primarily benefiting only from > the blacklisting and whitelisting of the whole? > > How many false positives would we encounter that might actually affect a > home user? > > I think it'd be a very interesting day if we were to have essentially a > Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even > more IP mobile than they are now. > > Take atrivo/intercare/mccolo for example. Infested with crap, and have > been for years. But since they can't really be blocked on the backbone > home users still hit the same scam AV sites, give their credit card > info, and get screwed. We know the sites are there, the registrars won't > take them down, the ISP is colluding with the bad guys so they'll stay > online. What can we do? (besides scream to our representatives for more > effective laws) > > We can block those bad IPs at the home user's level. That'll make them > start moving of course, just like bots being used to spam until they're > listed. So we have to be able to immediately move quickly with the. > > What does everyone think there? The basic idea being to use a normal > engine model by most security pro's to feed IP reputation into a global > database, and then the home user gets some sort of very basic tool or > button they can click on to benefit from that data? Maybe even feed back > to us. > > > >> In terms of my recommendation of a paran-o-meter, it makes a difference. >> Actually, I see huge debates over initial settings: do we keep it low to keep from >> crying wolf, or keep it high to keep people as safe as possible. But one thing that >> should be done is make the paranoia settings not-quite-obvious up front, so that >> somebody needs to know a little about the implications before they start fiddling >> with settings. >> >> Heck, if it's a professional device, we don't need to worry about the interface at >> all. If it's for Granny, we definitely do. > > Agreed. I don't think we can satisfy any of the needs of either granny > or us in the same tool. It'd either be too dumbed down for us, or too > complex for granny. I don't see any middle ground personally. > > I like the home router plugin thing though. If it could feed back to us > what IPs it was blocking we'd learn a lot! > > Matt > >> >> It also makes a difference in terms of the technology to be included. If it is for >> professionals, we can throw in everything. If for Granny, we need to make a >> careful choice about maximum protection for minimum performance drain. >> >> ====================== (quote inserted randomly by Pegasus Mailer) >> rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org >> I'm getting so absent-minded that sometimes in the middle of >> victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From jonkman at jonkmans.com Sun Oct 19 12:48:01 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 19 Oct 2008 12:48:01 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> Message-ID: <48FB64C1.4000803@jonkmans.com> Martin Holste wrote: > I'm guessing that the intent is to be the "detection" component of the > Trusted Internet Connection (TIC) > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), > in which the mandate is to protect all federal assets, especially the > federal desktop environment. Certainly a sound approach to tackle such a problem. Glad I don't have to solve that one, it's huge among gov't agencies. Especially in the US, but surely similar everywhere. Even if DHS has different goals, I think > that this group could do a great service by providing the TIC's > protection geared toward the client-side, and I think for most sectors, > that's where the most imminent threats lie. Again, I'm certainly not > saying that defending servers is out of scope, just that IDS has > historically been about defending servers as the primary goal, with > clients secondary, and I think that needs to be reversed now. I agree. At emerging threats we've been lately heavy into post-infection signatures in our research. There are many vulnerabilities out there in client apps, but I'd venture to say that a very high percentage of the infections out there aren't the result of a remotely exploitable vulnerability. Most come from users clicking on email attachments, installing fake software/codecs, and visiting websites with hostile code. We've tried many times to write effective signatures to detect hostile html/java/gifs, etc. It's just not feasible as is. Code is too flexible for signature-based approaches. You can say the same thing a hundred ways, especially in html. So how can we go after client side? I really REALLY am not excited about trying to make a windows client. Not only does that open up a huge responsibility in support and the inevitable bluescreens, but I have had a difficult time over the years believing that any process on ANY os (especially windows) could be trusted and independant enough to watch itself. Take into account how easy it is for trojans and rootkits to shut down antivirus, or blind it. And these are products with hundreds of the most skilled coders around working on them. I know we're sharp as a community, but I don't think that's a battle we want to get in to. So how can we do it at the network layer? Sandboxing? Virtual emulators? Or do we continue the thinking that for any infection to be of any use to anyone it has to generate traffic? It has to call home, send spam, report stolen information, something. So we concentrate on detecting and stopping the post infection? What does everyone think about that? Matt > > --Martin > > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > wrote: > > I doubt the intent of the DHS is to simply do good, they are most likely > much more focused on producing technology that allows them to > detect/mitigate/prevent attacks against critical components. This of > course means detecting attacks that fly below the threshold of detection > for todays technology. If it comes to "doing good" or detecting state > sponsored attacks against critical components (think custom attacks > against unknown vulns), i'm going to go out on a limb and say they would > rather protect the critical component vs the enduser. > > What you are discussing still has value and merit but im not so sure it > is what should be focused on, but of course I am not the person to > decide such things. > > Andre > > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > > Question: what are we making? Oh, yeah, I read the blurb: "The > OISF has been > > chartered and funded to build a next-generation intrusion > detection and prevention > > engine. This project will consider every new and existing > technology, concept and > > idea to build a completely open source licensed engine." > > > > OK, we're making an IDS. But I think we need to be more specific. > In particular, > > we need to answer the question of "who." > > > > Since the DHS has provided money, I suspect there would be an > automatic > > assumption that this is a heavy-duty device intended for use to > protect major > > servers and nodes in the critical information infrastructure. > (Whatever that > > means.) This kind of thing is built by professionals, for > professionals. Trained > > people. > > > > However, given the current computing environment, I think it would > be relatively > > easy to make a case that such a device is not going to do all that > much good. That > > a more accessible device, intended for "Grannyx" users, would > actually do more to > > protect the infrastructure. After all, it isn't major nodes on > the net that make up > > botnets, it's the little guys. Protect them, and you reduce the > threat. This is the > > "low hanging fruit" for the blackhats, so protecting that crop is > going to give us > > the greatest benefit for the commitment of resources. > > > > This makes a difference. Not, perhaps, in terms of questions > about multithreading > > analysis streams using graphics co-processors. But certainly in > most other areas. > > > > We've talked about extensibility. If we create a standard > template or format for > > signatures, the "who" makes a difference. Professionals need a > warning and a > > packet. Grannyx users need a warning, no packet, a clear > explanation of what and > > how important, and a recommended course of action. Makes a > difference to the > > template. > > > > In terms of my recommendation of a paran-o-meter, it makes a > difference. > > Actually, I see huge debates over initial settings: do we keep it > low to keep from > > crying wolf, or keep it high to keep people as safe as possible. > But one thing that > > should be done is make the paranoia settings not-quite-obvious up > front, so that > > somebody needs to know a little about the implications before they > start fiddling > > with settings. > > > > Heck, if it's a professional device, we don't need to worry about > the interface at > > all. If it's for Granny, we definitely do. > > > > It also makes a difference in terms of the technology to be > included. If it is for > > professionals, we can throw in everything. If for Granny, we need > to make a > > careful choice about maximum protection for minimum performance drain. > > > > ====================== (quote inserted randomly by Pegasus Mailer) > > rslade at vcn.bc.ca > slade at victoria.tc.ca > rslade at computercrime.org > > I'm getting so absent-minded that sometimes in the middle of > > victoria.tc.ca/techrev/rms.htm > > blogs.securiteam.com/index.php/archives/author/p1/ > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From david.glosser at gmail.com Sun Oct 19 13:13:06 2008 From: david.glosser at gmail.com (David Glosser) Date: Sun, 19 Oct 2008 13:13:06 -0400 Subject: [Discussion] What are we making? - Target User In-Reply-To: <48FB6113.6070806@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FB6113.6070806@jonkmans.com> Message-ID: This is why I asked earlier about something that runs on desktops or via browser plug-in, for the home user..... There are so many home router types out there, would approaching each vendor be feasible? Or is this something to be run at the ISP-level? There are several reputation services out there (siteadvisor, finjan, WOT, etc), how would this differ? Or just partner with those companies to provide data to help the end user? > > Very well put. We've always left the home user to fend for themselves > because it's just too complicated to run IDS unless you're a security > professional. Thus in the botnet world we chase command and control > servers and leave the bots infected. Not the best approach. > > So if we were to make this tool capable of being run "out of the box" as > a simple install and it'll do the rest on it's own, what would that mean? > > Would we need it to run on a WRTG style router OS? > > Would we need to approach the home router makers about a plugin? > > Would we want to go desktop stuff? (not my preference as the fox can't > be trusted to watch the henhouse IMHO) > > Or do we go with just pushing reputational data to the home user? What I > mean is if we build this engine to generate and act upon IP reputation > data could we know enough about the Internet collectively to simply push > a blacklist to the home user's router/firewall? > > On the more sophisticated devices where software could be installed > maybe it does run a stripped down detection engine and help feed IP data > back to the group. But overall it's still primarily benefiting only from > the blacklisting and whitelisting of the whole? > > How many false positives would we encounter that might actually affect a > home user? > From mcholste at gmail.com Sun Oct 19 13:18:59 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 12:18:59 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FB64C1.4000803@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: I vehemently agree with Matt about the necessity of staying out of the client-side arena. I think that if we're going to actually produce anything of value, we need to narrow the scope to start with. My vote is that we consider notifying Granny that there's something bad on her PC out of scope until we figure out how to notify Granny's ISP's network security admin that there's something bad. If we can figure out how to reliably find successful compromises and get that information output as actionable intelligence to the netsec community, the rest will come in time. For instance, if we can get to the point where we're %99.99 sure that a given URL is bad, then we could work on setting up an alliance with a place like Google, which already has an infrastructure for end-user notification of bad sites (as in, the "this site may harm your computer" displayed under links to sql injected pages). I'm sure Neils Provos would be interested in hints from this group. I would say that blackhat SEO is 80% of the malware infection vector right now, so that would be huge. Someday even farther in the future, we could use a platform like OSSEC HIDS to deliver our blacklists, or a browser plugin, but that is a carrot on a stick for another day. The point is that we need to start by doing one thing and doing it well: provide the most accurate method of network-based detection and prevention of compromises to date by leveraging the collective wisdom of the community. On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman wrote: > Martin Holste wrote: > > I'm guessing that the intent is to be the "detection" component of the > > Trusted Internet Connection (TIC) > > ( > http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html > ), > > in which the mandate is to protect all federal assets, especially the > > federal desktop environment. > > Certainly a sound approach to tackle such a problem. Glad I don't have > to solve that one, it's huge among gov't agencies. Especially in the US, > but surely similar everywhere. > > > Even if DHS has different goals, I think > > that this group could do a great service by providing the TIC's > > protection geared toward the client-side, and I think for most sectors, > > that's where the most imminent threats lie. Again, I'm certainly not > > saying that defending servers is out of scope, just that IDS has > > historically been about defending servers as the primary goal, with > > clients secondary, and I think that needs to be reversed now. > > I agree. At emerging threats we've been lately heavy into post-infection > signatures in our research. There are many vulnerabilities out there in > client apps, but I'd venture to say that a very high percentage of the > infections out there aren't the result of a remotely exploitable > vulnerability. Most come from users clicking on email attachments, > installing fake software/codecs, and visiting websites with hostile code. > > We've tried many times to write effective signatures to detect hostile > html/java/gifs, etc. It's just not feasible as is. Code is too flexible > for signature-based approaches. You can say the same thing a hundred > ways, especially in html. > > So how can we go after client side? > > I really REALLY am not excited about trying to make a windows client. > Not only does that open up a huge responsibility in support and the > inevitable bluescreens, but I have had a difficult time over the years > believing that any process on ANY os (especially windows) could be > trusted and independant enough to watch itself. Take into account how > easy it is for trojans and rootkits to shut down antivirus, or blind it. > And these are products with hundreds of the most skilled coders around > working on them. > > I know we're sharp as a community, but I don't think that's a battle we > want to get in to. So how can we do it at the network layer? > > Sandboxing? > > Virtual emulators? > > Or do we continue the thinking that for any infection to be of any use > to anyone it has to generate traffic? It has to call home, send spam, > report stolen information, something. So we concentrate on detecting and > stopping the post infection? > > What does everyone think about that? > > Matt > > > > > > --Martin > > > > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > > wrote: > > > > I doubt the intent of the DHS is to simply do good, they are most > likely > > much more focused on producing technology that allows them to > > detect/mitigate/prevent attacks against critical components. This of > > course means detecting attacks that fly below the threshold of > detection > > for todays technology. If it comes to "doing good" or detecting > state > > sponsored attacks against critical components (think custom attacks > > against unknown vulns), i'm going to go out on a limb and say they > would > > rather protect the critical component vs the enduser. > > > > What you are discussing still has value and merit but im not so sure > it > > is what should be focused on, but of course I am not the person to > > decide such things. > > > > Andre > > > > > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > > > Question: what are we making? Oh, yeah, I read the blurb: "The > > OISF has been > > > chartered and funded to build a next-generation intrusion > > detection and prevention > > > engine. This project will consider every new and existing > > technology, concept and > > > idea to build a completely open source licensed engine." > > > > > > OK, we're making an IDS. But I think we need to be more specific. > > In particular, > > > we need to answer the question of "who." > > > > > > Since the DHS has provided money, I suspect there would be an > > automatic > > > assumption that this is a heavy-duty device intended for use to > > protect major > > > servers and nodes in the critical information infrastructure. > > (Whatever that > > > means.) This kind of thing is built by professionals, for > > professionals. Trained > > > people. > > > > > > However, given the current computing environment, I think it would > > be relatively > > > easy to make a case that such a device is not going to do all that > > much good. That > > > a more accessible device, intended for "Grannyx" users, would > > actually do more to > > > protect the infrastructure. After all, it isn't major nodes on > > the net that make up > > > botnets, it's the little guys. Protect them, and you reduce the > > threat. This is the > > > "low hanging fruit" for the blackhats, so protecting that crop is > > going to give us > > > the greatest benefit for the commitment of resources. > > > > > > This makes a difference. Not, perhaps, in terms of questions > > about multithreading > > > analysis streams using graphics co-processors. But certainly in > > most other areas. > > > > > > We've talked about extensibility. If we create a standard > > template or format for > > > signatures, the "who" makes a difference. Professionals need a > > warning and a > > > packet. Grannyx users need a warning, no packet, a clear > > explanation of what and > > > how important, and a recommended course of action. Makes a > > difference to the > > > template. > > > > > > In terms of my recommendation of a paran-o-meter, it makes a > > difference. > > > Actually, I see huge debates over initial settings: do we keep it > > low to keep from > > > crying wolf, or keep it high to keep people as safe as possible. > > But one thing that > > > should be done is make the paranoia settings not-quite-obvious up > > front, so that > > > somebody needs to know a little about the implications before they > > start fiddling > > > with settings. > > > > > > Heck, if it's a professional device, we don't need to worry about > > the interface at > > > all. If it's for Granny, we definitely do. > > > > > > It also makes a difference in terms of the technology to be > > included. If it is for > > > professionals, we can throw in everything. If for Granny, we need > > to make a > > > careful choice about maximum protection for minimum performance > drain. > > > > > > ====================== (quote inserted randomly by Pegasus Mailer) > > > rslade at vcn.bc.ca > > slade at victoria.tc.ca > > rslade at computercrime.org > > > I'm getting so absent-minded that sometimes in the middle > of > > > victoria.tc.ca/techrev/rms.htm > > > > blogs.securiteam.com/index.php/archives/author/p1/ > > > > > _______________________________________________ > > > Discussion mailing list > > > Discussion at openinfosecfoundation.org > > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/d6f8de5d/attachment-0001.html From david.glosser at gmail.com Sun Oct 19 13:50:18 2008 From: david.glosser at gmail.com (David Glosser) Date: Sun, 19 Oct 2008 13:50:18 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: How about if a subset of the data (IP blacklists, domain blacklists, etc) is made available to the googles of the world, Granny's isps, browser-plug-in writers of the world, and desktop-based software. Then granny can be protected on the desktop, her ISP, etc. On Sun, Oct 19, 2008 at 1:18 PM, Martin Holste wrote: > I vehemently agree with Matt about the necessity of staying out of the > client-side arena. I think that if we're going to actually produce anything > of value, we need to narrow the scope to start with. My vote is that we > consider notifying Granny that there's something bad on her PC out of scope > until we figure out how to notify Granny's ISP's network security admin that > there's something bad. If we can figure out how to reliably find successful > compromises and get that information output as actionable intelligence to > the netsec community, the rest will come in time. > > For instance, if we can get to the point where we're %99.99 sure that a > given URL is bad, then we could work on setting up an alliance with a place > like Google, which already has an infrastructure for end-user notification > of bad sites (as in, the "this site may harm your computer" displayed under > links to sql injected pages). I'm sure Neils Provos would be interested in > hints from this group. I would say that blackhat SEO is 80% of the malware > infection vector right now, so that would be huge. > > Someday even farther in the future, we could use a platform like OSSEC HIDS > to deliver our blacklists, or a browser plugin, but that is a carrot on a > stick for another day. > > The point is that we need to start by doing one thing and doing it well: > provide the most accurate method of network-based detection and prevention > of compromises to date by leveraging the collective wisdom of the community. > > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman wrote: >> >> Martin Holste wrote: >> > I'm guessing that the intent is to be the "detection" component of the >> > Trusted Internet Connection (TIC) >> > >> > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), >> > in which the mandate is to protect all federal assets, especially the >> > federal desktop environment. >> >> Certainly a sound approach to tackle such a problem. Glad I don't have >> to solve that one, it's huge among gov't agencies. Especially in the US, >> but surely similar everywhere. >> >> >> Even if DHS has different goals, I think >> > that this group could do a great service by providing the TIC's >> > protection geared toward the client-side, and I think for most sectors, >> > that's where the most imminent threats lie. Again, I'm certainly not >> > saying that defending servers is out of scope, just that IDS has >> > historically been about defending servers as the primary goal, with >> > clients secondary, and I think that needs to be reversed now. >> >> I agree. At emerging threats we've been lately heavy into post-infection >> signatures in our research. There are many vulnerabilities out there in >> client apps, but I'd venture to say that a very high percentage of the >> infections out there aren't the result of a remotely exploitable >> vulnerability. Most come from users clicking on email attachments, >> installing fake software/codecs, and visiting websites with hostile code. >> >> We've tried many times to write effective signatures to detect hostile >> html/java/gifs, etc. It's just not feasible as is. Code is too flexible >> for signature-based approaches. You can say the same thing a hundred >> ways, especially in html. >> >> So how can we go after client side? >> >> I really REALLY am not excited about trying to make a windows client. >> Not only does that open up a huge responsibility in support and the >> inevitable bluescreens, but I have had a difficult time over the years >> believing that any process on ANY os (especially windows) could be >> trusted and independant enough to watch itself. Take into account how >> easy it is for trojans and rootkits to shut down antivirus, or blind it. >> And these are products with hundreds of the most skilled coders around >> working on them. >> >> I know we're sharp as a community, but I don't think that's a battle we >> want to get in to. So how can we do it at the network layer? >> >> Sandboxing? >> >> Virtual emulators? >> >> Or do we continue the thinking that for any infection to be of any use >> to anyone it has to generate traffic? It has to call home, send spam, >> report stolen information, something. So we concentrate on detecting and >> stopping the post infection? >> >> What does everyone think about that? >> >> Matt >> >> >> > >> > --Martin >> > >> > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > > > wrote: >> > >> > I doubt the intent of the DHS is to simply do good, they are most >> > likely >> > much more focused on producing technology that allows them to >> > detect/mitigate/prevent attacks against critical components. This >> > of >> > course means detecting attacks that fly below the threshold of >> > detection >> > for todays technology. If it comes to "doing good" or detecting >> > state >> > sponsored attacks against critical components (think custom attacks >> > against unknown vulns), i'm going to go out on a limb and say they >> > would >> > rather protect the critical component vs the enduser. >> > >> > What you are discussing still has value and merit but im not so sure >> > it >> > is what should be focused on, but of course I am not the person to >> > decide such things. >> > >> > Andre >> > >> > >> > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: >> > > Question: what are we making? Oh, yeah, I read the blurb: "The >> > OISF has been >> > > chartered and funded to build a next-generation intrusion >> > detection and prevention >> > > engine. This project will consider every new and existing >> > technology, concept and >> > > idea to build a completely open source licensed engine." >> > > >> > > OK, we're making an IDS. But I think we need to be more specific. >> > In particular, >> > > we need to answer the question of "who." >> > > >> > > Since the DHS has provided money, I suspect there would be an >> > automatic >> > > assumption that this is a heavy-duty device intended for use to >> > protect major >> > > servers and nodes in the critical information infrastructure. >> > (Whatever that >> > > means.) This kind of thing is built by professionals, for >> > professionals. Trained >> > > people. >> > > >> > > However, given the current computing environment, I think it would >> > be relatively >> > > easy to make a case that such a device is not going to do all that >> > much good. That >> > > a more accessible device, intended for "Grannyx" users, would >> > actually do more to >> > > protect the infrastructure. After all, it isn't major nodes on >> > the net that make up >> > > botnets, it's the little guys. Protect them, and you reduce the >> > threat. This is the >> > > "low hanging fruit" for the blackhats, so protecting that crop is >> > going to give us >> > > the greatest benefit for the commitment of resources. >> > > >> > > This makes a difference. Not, perhaps, in terms of questions >> > about multithreading >> > > analysis streams using graphics co-processors. But certainly in >> > most other areas. >> > > >> > > We've talked about extensibility. If we create a standard >> > template or format for >> > > signatures, the "who" makes a difference. Professionals need a >> > warning and a >> > > packet. Grannyx users need a warning, no packet, a clear >> > explanation of what and >> > > how important, and a recommended course of action. Makes a >> > difference to the >> > > template. >> > > >> > > In terms of my recommendation of a paran-o-meter, it makes a >> > difference. >> > > Actually, I see huge debates over initial settings: do we keep it >> > low to keep from >> > > crying wolf, or keep it high to keep people as safe as possible. >> > But one thing that >> > > should be done is make the paranoia settings not-quite-obvious up >> > front, so that >> > > somebody needs to know a little about the implications before they >> > start fiddling >> > > with settings. >> > > >> > > Heck, if it's a professional device, we don't need to worry about >> > the interface at >> > > all. If it's for Granny, we definitely do. >> > > >> > > It also makes a difference in terms of the technology to be >> > included. If it is for >> > > professionals, we can throw in everything. If for Granny, we need >> > to make a >> > > careful choice about maximum protection for minimum performance >> > drain. >> > > >> > > ====================== (quote inserted randomly by Pegasus >> > Mailer) >> > > rslade at vcn.bc.ca >> > slade at victoria.tc.ca >> > rslade at computercrime.org >> > > I'm getting so absent-minded that sometimes in the middle >> > of >> > > victoria.tc.ca/techrev/rms.htm >> > >> > blogs.securiteam.com/index.php/archives/author/p1/ >> > >> > > _______________________________________________ >> > > Discussion mailing list >> > > Discussion at openinfosecfoundation.org >> > >> > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > > >> > > >> > >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From mcholste at gmail.com Sun Oct 19 13:51:20 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 12:51:20 -0500 Subject: [Discussion] I think everyone is describing Bro In-Reply-To: <48FB51C9.5030709@jonkmans.com> References: <48FB51C9.5030709@jonkmans.com> Message-ID: Wow, I haven't looked at Bro in over a year, and the new additions to 1.4 are stunning. Nice work! I agree that Bro certainly has all of the back-end features that we have been discussing, down the HTTP sniffer I was alluding to earlier, the realtime loading of IP lists, and the scripts versus signatures design. Most importantly, it has a scalable, extremely modular design. I think what Bro has been missing is a community just like this to act as a catalyst for breaking out it of its niche in academia and becoming something that average organizations can use. I think that it's telling that Bro only recently got the ability to write anything to a database bundled with it natively, as that's a basic requirement for many people. As I look through the policy scripts, it's like I'm walking through a strange wonderland where anything is possible and great things are happening, but I have no idea what's going on. There is a rather steep learning curve for the Bro scripting language, and though well documented, I think that's rather off-putting for a lot of people. I actually had an easier time understanding what was going on with the pehunter source code written in C than I did trying to read the Bro scripting language straight-up. I'm not saying that it's not worth learning, I'm just saying that there's a pretty significant initial investment needed to get going with it. I think what this group could do would be to provide standards, tools, etc. for keeping track of all of these policies, writing new ones, and providing a comprehensive framework for managing it all, and a frontend for acting on the intelligence it provides. I see Bro as something of an uncut diamond. If this group adopted Bro as a platform of choice, I think we'd have a real shot at decreeing that "this group shall re-invent no wheels" and still achieve all of the goals we set out to do. The idea being that we're not here to resolve problems, but rather to apply and focus work already done into something that produces community-aware, actionable intelligence instead of log files. I will start a new thread with the beginnings of a proposal to the group to explore that idea further. On Sun, Oct 19, 2008 at 10:27 AM, Matt Jonkman wrote: > I definitely agree. There are a great number of features in bro that we > can learn from or steal. :) > > Matt > > Thorsten Holz wrote: > > On 19.10.2008, at 06:25, Seth Hall wrote: > > > >> Sorry, I just joined the list so I'm going to be doing some odd > >> quoting from the list archive :) I do want to point out too, that I'm > >> not writing this email to downplay OISFs goals but rather to hopefully > >> guide OISF toward improving an existing opensource project (Bro - > >> http://www.bro-ids.org/ > >> ) that already does much of what is being discussed on this list. > > > > I agree with Seth: why implement something completely new when you > > can extend an existing project that already contains many features > > that should be included in the resulting tool? Bro has some cool > > features and could be considered as a starting point. > > > > Cheers, > > Thorsten > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/47402660/attachment.html From mcholste at gmail.com Sun Oct 19 14:06:02 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 13:06:02 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: Yes, exactly, well put. What about chartering a CVE-inspired protocol for our blacklists? I think someone mentioned there was an IETF draft or something for that that didn't get off the ground. A specification could be as simple as a DTD for XML blacklists like:
x.x.x.x
 Or whatever--you get the idea. I think that the really cool part would be if the "reason" tag referred to an actual event uploaded in a snortsam-like way to a community event database. Providing a standard blacklist definition would make it much easier for orgs to write their own filters for what blacklist items they want to block on, which would lead to higher adoption rates. To further accelerate adoption, the community could even parse other publicly available lists at the start and republish them in the new standard until those sources natively publish in the standard. --Martin On Sun, Oct 19, 2008 at 12:50 PM, David Glosser wrote: > How about if a subset of the data (IP blacklists, domain blacklists, > etc) is made available to the googles of the world, Granny's isps, > browser-plug-in writers of the world, and desktop-based software. Then > granny can be protected on the desktop, her ISP, etc. > > > > > On Sun, Oct 19, 2008 at 1:18 PM, Martin Holste wrote: > > I vehemently agree with Matt about the necessity of staying out of the > > client-side arena. I think that if we're going to actually produce > anything > > of value, we need to narrow the scope to start with. My vote is that we > > consider notifying Granny that there's something bad on her PC out of > scope > > until we figure out how to notify Granny's ISP's network security admin > that > > there's something bad. If we can figure out how to reliably find > successful > > compromises and get that information output as actionable intelligence to > > the netsec community, the rest will come in time. > > > > For instance, if we can get to the point where we're %99.99 sure that a > > given URL is bad, then we could work on setting up an alliance with a > place > > like Google, which already has an infrastructure for end-user > notification > > of bad sites (as in, the "this site may harm your computer" displayed > under > > links to sql injected pages). I'm sure Neils Provos would be interested > in > > hints from this group. I would say that blackhat SEO is 80% of the > malware > > infection vector right now, so that would be huge. > > > > Someday even farther in the future, we could use a platform like OSSEC > HIDS > > to deliver our blacklists, or a browser plugin, but that is a carrot on a > > stick for another day. > > > > The point is that we need to start by doing one thing and doing it well: > > provide the most accurate method of network-based detection and > prevention > > of compromises to date by leveraging the collective wisdom of the > community. > > > > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman > wrote: > >> > >> Martin Holste wrote: > >> > I'm guessing that the intent is to be the "detection" component of the > >> > Trusted Internet Connection (TIC) > >> > > >> > ( > http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html > ), > >> > in which the mandate is to protect all federal assets, especially the > >> > federal desktop environment. > >> > >> Certainly a sound approach to tackle such a problem. Glad I don't have > >> to solve that one, it's huge among gov't agencies. Especially in the US, > >> but surely similar everywhere. > >> > >> > >> Even if DHS has different goals, I think > >> > that this group could do a great service by providing the TIC's > >> > protection geared toward the client-side, and I think for most > sectors, > >> > that's where the most imminent threats lie. Again, I'm certainly not > >> > saying that defending servers is out of scope, just that IDS has > >> > historically been about defending servers as the primary goal, with > >> > clients secondary, and I think that needs to be reversed now. > >> > >> I agree. At emerging threats we've been lately heavy into post-infection > >> signatures in our research. There are many vulnerabilities out there in > >> client apps, but I'd venture to say that a very high percentage of the > >> infections out there aren't the result of a remotely exploitable > >> vulnerability. Most come from users clicking on email attachments, > >> installing fake software/codecs, and visiting websites with hostile > code. > >> > >> We've tried many times to write effective signatures to detect hostile > >> html/java/gifs, etc. It's just not feasible as is. Code is too flexible > >> for signature-based approaches. You can say the same thing a hundred > >> ways, especially in html. > >> > >> So how can we go after client side? > >> > >> I really REALLY am not excited about trying to make a windows client. > >> Not only does that open up a huge responsibility in support and the > >> inevitable bluescreens, but I have had a difficult time over the years > >> believing that any process on ANY os (especially windows) could be > >> trusted and independant enough to watch itself. Take into account how > >> easy it is for trojans and rootkits to shut down antivirus, or blind it. > >> And these are products with hundreds of the most skilled coders around > >> working on them. > >> > >> I know we're sharp as a community, but I don't think that's a battle we > >> want to get in to. So how can we do it at the network layer? > >> > >> Sandboxing? > >> > >> Virtual emulators? > >> > >> Or do we continue the thinking that for any infection to be of any use > >> to anyone it has to generate traffic? It has to call home, send spam, > >> report stolen information, something. So we concentrate on detecting and > >> stopping the post infection? > >> > >> What does everyone think about that? > >> > >> Matt > >> > >> > >> > > >> > --Martin > >> > > >> > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig >> > > wrote: > >> > > >> > I doubt the intent of the DHS is to simply do good, they are most > >> > likely > >> > much more focused on producing technology that allows them to > >> > detect/mitigate/prevent attacks against critical components. This > >> > of > >> > course means detecting attacks that fly below the threshold of > >> > detection > >> > for todays technology. If it comes to "doing good" or detecting > >> > state > >> > sponsored attacks against critical components (think custom > attacks > >> > against unknown vulns), i'm going to go out on a limb and say they > >> > would > >> > rather protect the critical component vs the enduser. > >> > > >> > What you are discussing still has value and merit but im not so > sure > >> > it > >> > is what should be focused on, but of course I am not the person to > >> > decide such things. > >> > > >> > Andre > >> > > >> > > >> > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > >> > > Question: what are we making? Oh, yeah, I read the blurb: "The > >> > OISF has been > >> > > chartered and funded to build a next-generation intrusion > >> > detection and prevention > >> > > engine. This project will consider every new and existing > >> > technology, concept and > >> > > idea to build a completely open source licensed engine." > >> > > > >> > > OK, we're making an IDS. But I think we need to be more > specific. > >> > In particular, > >> > > we need to answer the question of "who." > >> > > > >> > > Since the DHS has provided money, I suspect there would be an > >> > automatic > >> > > assumption that this is a heavy-duty device intended for use to > >> > protect major > >> > > servers and nodes in the critical information infrastructure. > >> > (Whatever that > >> > > means.) This kind of thing is built by professionals, for > >> > professionals. Trained > >> > > people. > >> > > > >> > > However, given the current computing environment, I think it > would > >> > be relatively > >> > > easy to make a case that such a device is not going to do all > that > >> > much good. That > >> > > a more accessible device, intended for "Grannyx" users, would > >> > actually do more to > >> > > protect the infrastructure. After all, it isn't major nodes on > >> > the net that make up > >> > > botnets, it's the little guys. Protect them, and you reduce the > >> > threat. This is the > >> > > "low hanging fruit" for the blackhats, so protecting that crop > is > >> > going to give us > >> > > the greatest benefit for the commitment of resources. > >> > > > >> > > This makes a difference. Not, perhaps, in terms of questions > >> > about multithreading > >> > > analysis streams using graphics co-processors. But certainly in > >> > most other areas. > >> > > > >> > > We've talked about extensibility. If we create a standard > >> > template or format for > >> > > signatures, the "who" makes a difference. Professionals need a > >> > warning and a > >> > > packet. Grannyx users need a warning, no packet, a clear > >> > explanation of what and > >> > > how important, and a recommended course of action. Makes a > >> > difference to the > >> > > template. > >> > > > >> > > In terms of my recommendation of a paran-o-meter, it makes a > >> > difference. > >> > > Actually, I see huge debates over initial settings: do we keep > it > >> > low to keep from > >> > > crying wolf, or keep it high to keep people as safe as possible. > >> > But one thing that > >> > > should be done is make the paranoia settings not-quite-obvious > up > >> > front, so that > >> > > somebody needs to know a little about the implications before > they > >> > start fiddling > >> > > with settings. > >> > > > >> > > Heck, if it's a professional device, we don't need to worry > about > >> > the interface at > >> > > all. If it's for Granny, we definitely do. > >> > > > >> > > It also makes a difference in terms of the technology to be > >> > included. If it is for > >> > > professionals, we can throw in everything. If for Granny, we > need > >> > to make a > >> > > careful choice about maximum protection for minimum performance > >> > drain. > >> > > > >> > > ====================== (quote inserted randomly by Pegasus > >> > Mailer) > >> > > rslade at vcn.bc.ca > >> > slade at victoria.tc.ca > >> > rslade at computercrime.org > >> > > I'm getting so absent-minded that sometimes in the > middle > >> > of > >> > > victoria.tc.ca/techrev/rms.htm > >> > > >> > blogs.securiteam.com/index.php/archives/author/p1/ > >> > > >> > > _______________________________________________ > >> > > Discussion mailing list > >> > > Discussion at openinfosecfoundation.org > >> > > >> > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > > > >> > > > >> > > >> > _______________________________________________ > >> > Discussion mailing list > >> > Discussion at openinfosecfoundation.org > >> > > >> > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > > >> > > >> > > >> > > ------------------------------------------------------------------------ > >> > > >> > _______________________________________________ > >> > Discussion mailing list > >> > Discussion at openinfosecfoundation.org > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > >> -- > >> -------------------------------------------- > >> Matthew Jonkman > >> Emerging Threats > >> Phone 765-429-0398 > >> Fax 312-264-0205 > >> http://www.emergingthreats.net > >> -------------------------------------------- > >> > >> PGP: http://www.jonkmans.com/mattjonkman.asc > >> > >> > > > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/c6f87feb/attachment-0001.html From david.glosser at gmail.com Sun Oct 19 14:15:21 2008 From: david.glosser at gmail.com (David Glosser) Date: Sun, 19 Oct 2008 14:15:21 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: Getting the information out could be in multiple formats - static stuff like XML, null routes, dns blocklists, even the dreaded host-file... More dynamic stuff could be BGP, DNS (or partner with opendns), etc. On Sun, Oct 19, 2008 at 2:06 PM, Martin Holste wrote: > Yes, exactly, well put. What about chartering a CVE-inspired protocol for > our blacklists? I think someone mentioned there was an IETF draft or > something for that that didn't get off the ground. A specification could be > as simple as a DTD for XML blacklists like: > > > > > > > >
x.x.x.x
> > > > > > > > Or whatever--you get the idea. I think that the really cool part would be > if the "reason" tag referred to an actual event uploaded in a snortsam-like > way to a community event database. > > Providing a standard blacklist definition would make it much easier for orgs > to write their own filters for what blacklist items they want to block on, > which would lead to higher adoption rates. To further accelerate adoption, > the community could even parse other publicly available lists at the start > and republish them in the new standard until those sources natively publish > in the standard. > > --Martin > > On Sun, Oct 19, 2008 at 12:50 PM, David Glosser > wrote: >> >> How about if a subset of the data (IP blacklists, domain blacklists, >> etc) is made available to the googles of the world, Granny's isps, >> browser-plug-in writers of the world, and desktop-based software. Then >> granny can be protected on the desktop, her ISP, etc. >> >> >> >> >> On Sun, Oct 19, 2008 at 1:18 PM, Martin Holste wrote: >> > I vehemently agree with Matt about the necessity of staying out of the >> > client-side arena. I think that if we're going to actually produce >> > anything >> > of value, we need to narrow the scope to start with. My vote is that we >> > consider notifying Granny that there's something bad on her PC out of >> > scope >> > until we figure out how to notify Granny's ISP's network security admin >> > that >> > there's something bad. If we can figure out how to reliably find >> > successful >> > compromises and get that information output as actionable intelligence >> > to >> > the netsec community, the rest will come in time. >> > >> > For instance, if we can get to the point where we're %99.99 sure that a >> > given URL is bad, then we could work on setting up an alliance with a >> > place >> > like Google, which already has an infrastructure for end-user >> > notification >> > of bad sites (as in, the "this site may harm your computer" displayed >> > under >> > links to sql injected pages). I'm sure Neils Provos would be interested >> > in >> > hints from this group. I would say that blackhat SEO is 80% of the >> > malware >> > infection vector right now, so that would be huge. >> > >> > Someday even farther in the future, we could use a platform like OSSEC >> > HIDS >> > to deliver our blacklists, or a browser plugin, but that is a carrot on >> > a >> > stick for another day. >> > >> > The point is that we need to start by doing one thing and doing it well: >> > provide the most accurate method of network-based detection and >> > prevention >> > of compromises to date by leveraging the collective wisdom of the >> > community. >> > >> > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman >> > wrote: >> >> >> >> Martin Holste wrote: >> >> > I'm guessing that the intent is to be the "detection" component of >> >> > the >> >> > Trusted Internet Connection (TIC) >> >> > >> >> > >> >> > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), >> >> > in which the mandate is to protect all federal assets, especially the >> >> > federal desktop environment. >> >> >> >> Certainly a sound approach to tackle such a problem. Glad I don't have >> >> to solve that one, it's huge among gov't agencies. Especially in the >> >> US, >> >> but surely similar everywhere. >> >> >> >> >> >> Even if DHS has different goals, I think >> >> > that this group could do a great service by providing the TIC's >> >> > protection geared toward the client-side, and I think for most >> >> > sectors, >> >> > that's where the most imminent threats lie. Again, I'm certainly not >> >> > saying that defending servers is out of scope, just that IDS has >> >> > historically been about defending servers as the primary goal, with >> >> > clients secondary, and I think that needs to be reversed now. >> >> >> >> I agree. At emerging threats we've been lately heavy into >> >> post-infection >> >> signatures in our research. There are many vulnerabilities out there in >> >> client apps, but I'd venture to say that a very high percentage of the >> >> infections out there aren't the result of a remotely exploitable >> >> vulnerability. Most come from users clicking on email attachments, >> >> installing fake software/codecs, and visiting websites with hostile >> >> code. >> >> >> >> We've tried many times to write effective signatures to detect hostile >> >> html/java/gifs, etc. It's just not feasible as is. Code is too flexible >> >> for signature-based approaches. You can say the same thing a hundred >> >> ways, especially in html. >> >> >> >> So how can we go after client side? >> >> >> >> I really REALLY am not excited about trying to make a windows client. >> >> Not only does that open up a huge responsibility in support and the >> >> inevitable bluescreens, but I have had a difficult time over the years >> >> believing that any process on ANY os (especially windows) could be >> >> trusted and independant enough to watch itself. Take into account how >> >> easy it is for trojans and rootkits to shut down antivirus, or blind >> >> it. >> >> And these are products with hundreds of the most skilled coders around >> >> working on them. >> >> >> >> I know we're sharp as a community, but I don't think that's a battle we >> >> want to get in to. So how can we do it at the network layer? >> >> >> >> Sandboxing? >> >> >> >> Virtual emulators? >> >> >> >> Or do we continue the thinking that for any infection to be of any use >> >> to anyone it has to generate traffic? It has to call home, send spam, >> >> report stolen information, something. So we concentrate on detecting >> >> and >> >> stopping the post infection? >> >> >> >> What does everyone think about that? >> >> >> >> Matt >> >> >> >> >> >> > >> >> > --Martin >> >> > >> >> > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > >> > > wrote: >> >> > >> >> > I doubt the intent of the DHS is to simply do good, they are most >> >> > likely >> >> > much more focused on producing technology that allows them to >> >> > detect/mitigate/prevent attacks against critical components. >> >> > This >> >> > of >> >> > course means detecting attacks that fly below the threshold of >> >> > detection >> >> > for todays technology. If it comes to "doing good" or detecting >> >> > state >> >> > sponsored attacks against critical components (think custom >> >> > attacks >> >> > against unknown vulns), i'm going to go out on a limb and say >> >> > they >> >> > would >> >> > rather protect the critical component vs the enduser. >> >> > >> >> > What you are discussing still has value and merit but im not so >> >> > sure >> >> > it >> >> > is what should be focused on, but of course I am not the person >> >> > to >> >> > decide such things. >> >> > >> >> > Andre >> >> > >> >> > >> >> > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: >> >> > > Question: what are we making? Oh, yeah, I read the blurb: "The >> >> > OISF has been >> >> > > chartered and funded to build a next-generation intrusion >> >> > detection and prevention >> >> > > engine. This project will consider every new and existing >> >> > technology, concept and >> >> > > idea to build a completely open source licensed engine." >> >> > > >> >> > > OK, we're making an IDS. But I think we need to be more >> >> > specific. >> >> > In particular, >> >> > > we need to answer the question of "who." >> >> > > >> >> > > Since the DHS has provided money, I suspect there would be an >> >> > automatic >> >> > > assumption that this is a heavy-duty device intended for use to >> >> > protect major >> >> > > servers and nodes in the critical information infrastructure. >> >> > (Whatever that >> >> > > means.) This kind of thing is built by professionals, for >> >> > professionals. Trained >> >> > > people. >> >> > > >> >> > > However, given the current computing environment, I think it >> >> > would >> >> > be relatively >> >> > > easy to make a case that such a device is not going to do all >> >> > that >> >> > much good. That >> >> > > a more accessible device, intended for "Grannyx" users, would >> >> > actually do more to >> >> > > protect the infrastructure. After all, it isn't major nodes on >> >> > the net that make up >> >> > > botnets, it's the little guys. Protect them, and you reduce >> >> > the >> >> > threat. This is the >> >> > > "low hanging fruit" for the blackhats, so protecting that crop >> >> > is >> >> > going to give us >> >> > > the greatest benefit for the commitment of resources. >> >> > > >> >> > > This makes a difference. Not, perhaps, in terms of questions >> >> > about multithreading >> >> > > analysis streams using graphics co-processors. But certainly >> >> > in >> >> > most other areas. >> >> > > >> >> > > We've talked about extensibility. If we create a standard >> >> > template or format for >> >> > > signatures, the "who" makes a difference. Professionals need a >> >> > warning and a >> >> > > packet. Grannyx users need a warning, no packet, a clear >> >> > explanation of what and >> >> > > how important, and a recommended course of action. Makes a >> >> > difference to the >> >> > > template. >> >> > > >> >> > > In terms of my recommendation of a paran-o-meter, it makes a >> >> > difference. >> >> > > Actually, I see huge debates over initial settings: do we keep >> >> > it >> >> > low to keep from >> >> > > crying wolf, or keep it high to keep people as safe as >> >> > possible. >> >> > But one thing that >> >> > > should be done is make the paranoia settings not-quite-obvious >> >> > up >> >> > front, so that >> >> > > somebody needs to know a little about the implications before >> >> > they >> >> > start fiddling >> >> > > with settings. >> >> > > >> >> > > Heck, if it's a professional device, we don't need to worry >> >> > about >> >> > the interface at >> >> > > all. If it's for Granny, we definitely do. >> >> > > >> >> > > It also makes a difference in terms of the technology to be >> >> > included. If it is for >> >> > > professionals, we can throw in everything. If for Granny, we >> >> > need >> >> > to make a >> >> > > careful choice about maximum protection for minimum performance >> >> > drain. >> >> > > >> >> > > ====================== (quote inserted randomly by Pegasus >> >> > Mailer) >> >> > > rslade at vcn.bc.ca >> >> > slade at victoria.tc.ca >> >> > rslade at computercrime.org >> >> > > I'm getting so absent-minded that sometimes in the >> >> > middle >> >> > of >> >> > > victoria.tc.ca/techrev/rms.htm >> >> > >> >> > blogs.securiteam.com/index.php/archives/author/p1/ >> >> > >> >> > > _______________________________________________ >> >> > > Discussion mailing list >> >> > > Discussion at openinfosecfoundation.org >> >> > >> >> > > >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > > >> >> > > >> >> > >> >> > _______________________________________________ >> >> > Discussion mailing list >> >> > Discussion at openinfosecfoundation.org >> >> > >> >> > >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------ >> >> > >> >> > _______________________________________________ >> >> > Discussion mailing list >> >> > Discussion at openinfosecfoundation.org >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> >> -- >> >> -------------------------------------------- >> >> Matthew Jonkman >> >> Emerging Threats >> >> Phone 765-429-0398 >> >> Fax 312-264-0205 >> >> http://www.emergingthreats.net >> >> -------------------------------------------- >> >> >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> >> >> > >> > >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > >> > > > From mcholste at gmail.com Sun Oct 19 15:30:58 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 14:30:58 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: Right, but I envision the XML to be the source that scripts would parse into whatever is needed, like router config, dns blocklists, host files, search engine blacklists, etc. The key would be to create a standard capable of being specific enough to feed the lowest common demoninator. --Martin On Sun, Oct 19, 2008 at 1:15 PM, David Glosser wrote: > Getting the information out could be in multiple formats - static > stuff like XML, null routes, dns blocklists, even the dreaded > host-file... > > More dynamic stuff could be BGP, DNS (or partner with opendns), etc. > > > > > > On Sun, Oct 19, 2008 at 2:06 PM, Martin Holste wrote: > > Yes, exactly, well put. What about chartering a CVE-inspired protocol > for > > our blacklists? I think someone mentioned there was an IETF draft or > > something for that that didn't get off the ground. A specification could > be > > as simple as a DTD for XML blacklists like: > > > > > > > > > > > > > > > >
x.x.x.x
> > > > > > > > > > > > > > > > Or whatever--you get the idea. I think that the really cool part would > be > > if the "reason" tag referred to an actual event uploaded in a > snortsam-like > > way to a community event database. > > > > Providing a standard blacklist definition would make it much easier for > orgs > > to write their own filters for what blacklist items they want to block > on, > > which would lead to higher adoption rates. To further accelerate > adoption, > > the community could even parse other publicly available lists at the > start > > and republish them in the new standard until those sources natively > publish > > in the standard. > > > > --Martin > > > > On Sun, Oct 19, 2008 at 12:50 PM, David Glosser > > > wrote: > >> > >> How about if a subset of the data (IP blacklists, domain blacklists, > >> etc) is made available to the googles of the world, Granny's isps, > >> browser-plug-in writers of the world, and desktop-based software. Then > >> granny can be protected on the desktop, her ISP, etc. > >> > >> > >> > >> > >> On Sun, Oct 19, 2008 at 1:18 PM, Martin Holste > wrote: > >> > I vehemently agree with Matt about the necessity of staying out of the > >> > client-side arena. I think that if we're going to actually produce > >> > anything > >> > of value, we need to narrow the scope to start with. My vote is that > we > >> > consider notifying Granny that there's something bad on her PC out of > >> > scope > >> > until we figure out how to notify Granny's ISP's network security > admin > >> > that > >> > there's something bad. If we can figure out how to reliably find > >> > successful > >> > compromises and get that information output as actionable intelligence > >> > to > >> > the netsec community, the rest will come in time. > >> > > >> > For instance, if we can get to the point where we're %99.99 sure that > a > >> > given URL is bad, then we could work on setting up an alliance with a > >> > place > >> > like Google, which already has an infrastructure for end-user > >> > notification > >> > of bad sites (as in, the "this site may harm your computer" displayed > >> > under > >> > links to sql injected pages). I'm sure Neils Provos would be > interested > >> > in > >> > hints from this group. I would say that blackhat SEO is 80% of the > >> > malware > >> > infection vector right now, so that would be huge. > >> > > >> > Someday even farther in the future, we could use a platform like OSSEC > >> > HIDS > >> > to deliver our blacklists, or a browser plugin, but that is a carrot > on > >> > a > >> > stick for another day. > >> > > >> > The point is that we need to start by doing one thing and doing it > well: > >> > provide the most accurate method of network-based detection and > >> > prevention > >> > of compromises to date by leveraging the collective wisdom of the > >> > community. > >> > > >> > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman > >> > wrote: > >> >> > >> >> Martin Holste wrote: > >> >> > I'm guessing that the intent is to be the "detection" component of > >> >> > the > >> >> > Trusted Internet Connection (TIC) > >> >> > > >> >> > > >> >> > ( > http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html > ), > >> >> > in which the mandate is to protect all federal assets, especially > the > >> >> > federal desktop environment. > >> >> > >> >> Certainly a sound approach to tackle such a problem. Glad I don't > have > >> >> to solve that one, it's huge among gov't agencies. Especially in the > >> >> US, > >> >> but surely similar everywhere. > >> >> > >> >> > >> >> Even if DHS has different goals, I think > >> >> > that this group could do a great service by providing the TIC's > >> >> > protection geared toward the client-side, and I think for most > >> >> > sectors, > >> >> > that's where the most imminent threats lie. Again, I'm certainly > not > >> >> > saying that defending servers is out of scope, just that IDS has > >> >> > historically been about defending servers as the primary goal, with > >> >> > clients secondary, and I think that needs to be reversed now. > >> >> > >> >> I agree. At emerging threats we've been lately heavy into > >> >> post-infection > >> >> signatures in our research. There are many vulnerabilities out there > in > >> >> client apps, but I'd venture to say that a very high percentage of > the > >> >> infections out there aren't the result of a remotely exploitable > >> >> vulnerability. Most come from users clicking on email attachments, > >> >> installing fake software/codecs, and visiting websites with hostile > >> >> code. > >> >> > >> >> We've tried many times to write effective signatures to detect > hostile > >> >> html/java/gifs, etc. It's just not feasible as is. Code is too > flexible > >> >> for signature-based approaches. You can say the same thing a hundred > >> >> ways, especially in html. > >> >> > >> >> So how can we go after client side? > >> >> > >> >> I really REALLY am not excited about trying to make a windows client. > >> >> Not only does that open up a huge responsibility in support and the > >> >> inevitable bluescreens, but I have had a difficult time over the > years > >> >> believing that any process on ANY os (especially windows) could be > >> >> trusted and independant enough to watch itself. Take into account how > >> >> easy it is for trojans and rootkits to shut down antivirus, or blind > >> >> it. > >> >> And these are products with hundreds of the most skilled coders > around > >> >> working on them. > >> >> > >> >> I know we're sharp as a community, but I don't think that's a battle > we > >> >> want to get in to. So how can we do it at the network layer? > >> >> > >> >> Sandboxing? > >> >> > >> >> Virtual emulators? > >> >> > >> >> Or do we continue the thinking that for any infection to be of any > use > >> >> to anyone it has to generate traffic? It has to call home, send spam, > >> >> report stolen information, something. So we concentrate on detecting > >> >> and > >> >> stopping the post infection? > >> >> > >> >> What does everyone think about that? > >> >> > >> >> Matt > >> >> > >> >> > >> >> > > >> >> > --Martin > >> >> > > >> >> > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig < > aludwig at packetspy.com > >> >> > > wrote: > >> >> > > >> >> > I doubt the intent of the DHS is to simply do good, they are > most > >> >> > likely > >> >> > much more focused on producing technology that allows them to > >> >> > detect/mitigate/prevent attacks against critical components. > >> >> > This > >> >> > of > >> >> > course means detecting attacks that fly below the threshold of > >> >> > detection > >> >> > for todays technology. If it comes to "doing good" or > detecting > >> >> > state > >> >> > sponsored attacks against critical components (think custom > >> >> > attacks > >> >> > against unknown vulns), i'm going to go out on a limb and say > >> >> > they > >> >> > would > >> >> > rather protect the critical component vs the enduser. > >> >> > > >> >> > What you are discussing still has value and merit but im not so > >> >> > sure > >> >> > it > >> >> > is what should be focused on, but of course I am not the person > >> >> > to > >> >> > decide such things. > >> >> > > >> >> > Andre > >> >> > > >> >> > > >> >> > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > >> >> > > Question: what are we making? Oh, yeah, I read the blurb: > "The > >> >> > OISF has been > >> >> > > chartered and funded to build a next-generation intrusion > >> >> > detection and prevention > >> >> > > engine. This project will consider every new and existing > >> >> > technology, concept and > >> >> > > idea to build a completely open source licensed engine." > >> >> > > > >> >> > > OK, we're making an IDS. But I think we need to be more > >> >> > specific. > >> >> > In particular, > >> >> > > we need to answer the question of "who." > >> >> > > > >> >> > > Since the DHS has provided money, I suspect there would be an > >> >> > automatic > >> >> > > assumption that this is a heavy-duty device intended for use > to > >> >> > protect major > >> >> > > servers and nodes in the critical information infrastructure. > >> >> > (Whatever that > >> >> > > means.) This kind of thing is built by professionals, for > >> >> > professionals. Trained > >> >> > > people. > >> >> > > > >> >> > > However, given the current computing environment, I think it > >> >> > would > >> >> > be relatively > >> >> > > easy to make a case that such a device is not going to do all > >> >> > that > >> >> > much good. That > >> >> > > a more accessible device, intended for "Grannyx" users, would > >> >> > actually do more to > >> >> > > protect the infrastructure. After all, it isn't major nodes > on > >> >> > the net that make up > >> >> > > botnets, it's the little guys. Protect them, and you reduce > >> >> > the > >> >> > threat. This is the > >> >> > > "low hanging fruit" for the blackhats, so protecting that > crop > >> >> > is > >> >> > going to give us > >> >> > > the greatest benefit for the commitment of resources. > >> >> > > > >> >> > > This makes a difference. Not, perhaps, in terms of questions > >> >> > about multithreading > >> >> > > analysis streams using graphics co-processors. But certainly > >> >> > in > >> >> > most other areas. > >> >> > > > >> >> > > We've talked about extensibility. If we create a standard > >> >> > template or format for > >> >> > > signatures, the "who" makes a difference. Professionals need > a > >> >> > warning and a > >> >> > > packet. Grannyx users need a warning, no packet, a clear > >> >> > explanation of what and > >> >> > > how important, and a recommended course of action. Makes a > >> >> > difference to the > >> >> > > template. > >> >> > > > >> >> > > In terms of my recommendation of a paran-o-meter, it makes a > >> >> > difference. > >> >> > > Actually, I see huge debates over initial settings: do we > keep > >> >> > it > >> >> > low to keep from > >> >> > > crying wolf, or keep it high to keep people as safe as > >> >> > possible. > >> >> > But one thing that > >> >> > > should be done is make the paranoia settings > not-quite-obvious > >> >> > up > >> >> > front, so that > >> >> > > somebody needs to know a little about the implications before > >> >> > they > >> >> > start fiddling > >> >> > > with settings. > >> >> > > > >> >> > > Heck, if it's a professional device, we don't need to worry > >> >> > about > >> >> > the interface at > >> >> > > all. If it's for Granny, we definitely do. > >> >> > > > >> >> > > It also makes a difference in terms of the technology to be > >> >> > included. If it is for > >> >> > > professionals, we can throw in everything. If for Granny, we > >> >> > need > >> >> > to make a > >> >> > > careful choice about maximum protection for minimum > performance > >> >> > drain. > >> >> > > > >> >> > > ====================== (quote inserted randomly by Pegasus > >> >> > Mailer) > >> >> > > rslade at vcn.bc.ca > >> >> > slade at victoria.tc.ca > >> >> > rslade at computercrime.org > >> >> > > I'm getting so absent-minded that sometimes in the > >> >> > middle > >> >> > of > >> >> > > victoria.tc.ca/techrev/rms.htm > >> >> > > >> >> > blogs.securiteam.com/index.php/archives/author/p1/ > >> >> > > >> >> > > _______________________________________________ > >> >> > > Discussion mailing list > >> >> > > Discussion at openinfosecfoundation.org > >> >> > > >> >> > > > >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> >> > > > >> >> > > > >> >> > > >> >> > _______________________________________________ > >> >> > Discussion mailing list > >> >> > Discussion at openinfosecfoundation.org > >> >> > > >> >> > > >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > ------------------------------------------------------------------------ > >> >> > > >> >> > _______________________________________________ > >> >> > Discussion mailing list > >> >> > Discussion at openinfosecfoundation.org > >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> >> > >> >> -- > >> >> -------------------------------------------- > >> >> Matthew Jonkman > >> >> Emerging Threats > >> >> Phone 765-429-0398 > >> >> Fax 312-264-0205 > >> >> http://www.emergingthreats.net > >> >> -------------------------------------------- > >> >> > >> >> PGP: http://www.jonkmans.com/mattjonkman.asc > >> >> > >> >> > >> > > >> > > >> > _______________________________________________ > >> > Discussion mailing list > >> > Discussion at openinfosecfoundation.org > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/10ce89dc/attachment-0001.html From david.glosser at gmail.com Sun Oct 19 18:24:44 2008 From: david.glosser at gmail.com (David Glosser) Date: Sun, 19 Oct 2008 18:24:44 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: no argument from me ;) just want to make the and granny and granny's IAP's happy... Also, would like to note it would be great if web hosting services (and website owners) are alerted when their site is hosting a bot, contains sql injection code, etc. I'm wondering about a rule set or something tailored to this reverse direction, from their OWN internal networks outbound. Maybe an IDS set tuned for the ISPs and web hosters of the world? Like what somebody said earlier (sorry, I don't remember who) said, you can't and shouldn't assume that all threats come from the outside.... On Sun, Oct 19, 2008 at 3:30 PM, Martin Holste wrote: > Right, but I envision the XML to be the source that scripts would parse into > whatever is needed, like router config, dns blocklists, host files, search > engine blacklists, etc. The key would be to create a standard capable of > being specific enough to feed the lowest common demoninator. > > --Martin > > On Sun, Oct 19, 2008 at 1:15 PM, David Glosser > wrote: >> >> Getting the information out could be in multiple formats - static >> stuff like XML, null routes, dns blocklists, even the dreaded >> host-file... >> >> More dynamic stuff could be BGP, DNS (or partner with opendns), etc. >> >> >> >> >> >> On Sun, Oct 19, 2008 at 2:06 PM, Martin Holste wrote: >> > Yes, exactly, well put. What about chartering a CVE-inspired protocol >> > for >> > our blacklists? I think someone mentioned there was an IETF draft or >> > something for that that didn't get off the ground. A specification >> > could be >> > as simple as a DTD for XML blacklists like: >> > >> > >> > >> > >> > >> > >> > >> >
x.x.x.x
>> > >> > >> > >> > >> > >> > >> > >> > Or whatever--you get the idea. I think that the really cool part would >> > be >> > if the "reason" tag referred to an actual event uploaded in a >> > snortsam-like >> > way to a community event database. >> > >> > Providing a standard blacklist definition would make it much easier for >> > orgs >> > to write their own filters for what blacklist items they want to block >> > on, >> > which would lead to higher adoption rates. To further accelerate >> > adoption, >> > the community could even parse other publicly available lists at the >> > start >> > and republish them in the new standard until those sources natively >> > publish >> > in the standard. >> > >> > --Martin >> > >> > On Sun, Oct 19, 2008 at 12:50 PM, David Glosser >> > >> > wrote: >> >> >> >> How about if a subset of the data (IP blacklists, domain blacklists, >> >> etc) is made available to the googles of the world, Granny's isps, >> >> browser-plug-in writers of the world, and desktop-based software. Then >> >> granny can be protected on the desktop, her ISP, etc. >> >> >> >> >> >> >> >> >> >> On Sun, Oct 19, 2008 at 1:18 PM, Martin Holste >> >> wrote: >> >> > I vehemently agree with Matt about the necessity of staying out of >> >> > the >> >> > client-side arena. I think that if we're going to actually produce >> >> > anything >> >> > of value, we need to narrow the scope to start with. My vote is that >> >> > we >> >> > consider notifying Granny that there's something bad on her PC out of >> >> > scope >> >> > until we figure out how to notify Granny's ISP's network security >> >> > admin >> >> > that >> >> > there's something bad. If we can figure out how to reliably find >> >> > successful >> >> > compromises and get that information output as actionable >> >> > intelligence >> >> > to >> >> > the netsec community, the rest will come in time. >> >> > >> >> > For instance, if we can get to the point where we're %99.99 sure that >> >> > a >> >> > given URL is bad, then we could work on setting up an alliance with a >> >> > place >> >> > like Google, which already has an infrastructure for end-user >> >> > notification >> >> > of bad sites (as in, the "this site may harm your computer" displayed >> >> > under >> >> > links to sql injected pages). I'm sure Neils Provos would be >> >> > interested >> >> > in >> >> > hints from this group. I would say that blackhat SEO is 80% of the >> >> > malware >> >> > infection vector right now, so that would be huge. >> >> > >> >> > Someday even farther in the future, we could use a platform like >> >> > OSSEC >> >> > HIDS >> >> > to deliver our blacklists, or a browser plugin, but that is a carrot >> >> > on >> >> > a >> >> > stick for another day. >> >> > >> >> > The point is that we need to start by doing one thing and doing it >> >> > well: >> >> > provide the most accurate method of network-based detection and >> >> > prevention >> >> > of compromises to date by leveraging the collective wisdom of the >> >> > community. >> >> > >> >> > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman >> >> > wrote: >> >> >> >> >> >> Martin Holste wrote: >> >> >> > I'm guessing that the intent is to be the "detection" component of >> >> >> > the >> >> >> > Trusted Internet Connection (TIC) >> >> >> > >> >> >> > >> >> >> > >> >> >> > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), >> >> >> > in which the mandate is to protect all federal assets, especially >> >> >> > the >> >> >> > federal desktop environment. >> >> >> >> >> >> Certainly a sound approach to tackle such a problem. Glad I don't >> >> >> have >> >> >> to solve that one, it's huge among gov't agencies. Especially in the >> >> >> US, >> >> >> but surely similar everywhere. >> >> >> >> >> >> >> >> >> Even if DHS has different goals, I think >> >> >> > that this group could do a great service by providing the TIC's >> >> >> > protection geared toward the client-side, and I think for most >> >> >> > sectors, >> >> >> > that's where the most imminent threats lie. Again, I'm certainly >> >> >> > not >> >> >> > saying that defending servers is out of scope, just that IDS has >> >> >> > historically been about defending servers as the primary goal, >> >> >> > with >> >> >> > clients secondary, and I think that needs to be reversed now. >> >> >> >> >> >> I agree. At emerging threats we've been lately heavy into >> >> >> post-infection >> >> >> signatures in our research. There are many vulnerabilities out there >> >> >> in >> >> >> client apps, but I'd venture to say that a very high percentage of >> >> >> the >> >> >> infections out there aren't the result of a remotely exploitable >> >> >> vulnerability. Most come from users clicking on email attachments, >> >> >> installing fake software/codecs, and visiting websites with hostile >> >> >> code. >> >> >> >> >> >> We've tried many times to write effective signatures to detect >> >> >> hostile >> >> >> html/java/gifs, etc. It's just not feasible as is. Code is too >> >> >> flexible >> >> >> for signature-based approaches. You can say the same thing a hundred >> >> >> ways, especially in html. >> >> >> >> >> >> So how can we go after client side? >> >> >> >> >> >> I really REALLY am not excited about trying to make a windows >> >> >> client. >> >> >> Not only does that open up a huge responsibility in support and the >> >> >> inevitable bluescreens, but I have had a difficult time over the >> >> >> years >> >> >> believing that any process on ANY os (especially windows) could be >> >> >> trusted and independant enough to watch itself. Take into account >> >> >> how >> >> >> easy it is for trojans and rootkits to shut down antivirus, or blind >> >> >> it. >> >> >> And these are products with hundreds of the most skilled coders >> >> >> around >> >> >> working on them. >> >> >> >> >> >> I know we're sharp as a community, but I don't think that's a battle >> >> >> we >> >> >> want to get in to. So how can we do it at the network layer? >> >> >> >> >> >> Sandboxing? >> >> >> >> >> >> Virtual emulators? >> >> >> >> >> >> Or do we continue the thinking that for any infection to be of any >> >> >> use >> >> >> to anyone it has to generate traffic? It has to call home, send >> >> >> spam, >> >> >> report stolen information, something. So we concentrate on detecting >> >> >> and >> >> >> stopping the post infection? >> >> >> >> >> >> What does everyone think about that? >> >> >> >> >> >> Matt >> >> >> >> >> >> >> >> >> > >> >> >> > --Martin >> >> >> > >> >> >> > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig >> >> >> > > >> >> > > wrote: >> >> >> > >> >> >> > I doubt the intent of the DHS is to simply do good, they are >> >> >> > most >> >> >> > likely >> >> >> > much more focused on producing technology that allows them to >> >> >> > detect/mitigate/prevent attacks against critical components. >> >> >> > This >> >> >> > of >> >> >> > course means detecting attacks that fly below the threshold of >> >> >> > detection >> >> >> > for todays technology. If it comes to "doing good" or >> >> >> > detecting >> >> >> > state >> >> >> > sponsored attacks against critical components (think custom >> >> >> > attacks >> >> >> > against unknown vulns), i'm going to go out on a limb and say >> >> >> > they >> >> >> > would >> >> >> > rather protect the critical component vs the enduser. >> >> >> > >> >> >> > What you are discussing still has value and merit but im not >> >> >> > so >> >> >> > sure >> >> >> > it >> >> >> > is what should be focused on, but of course I am not the >> >> >> > person >> >> >> > to >> >> >> > decide such things. >> >> >> > >> >> >> > Andre >> >> >> > >> >> >> > >> >> >> > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: >> >> >> > > Question: what are we making? Oh, yeah, I read the blurb: >> >> >> > "The >> >> >> > OISF has been >> >> >> > > chartered and funded to build a next-generation intrusion >> >> >> > detection and prevention >> >> >> > > engine. This project will consider every new and existing >> >> >> > technology, concept and >> >> >> > > idea to build a completely open source licensed engine." >> >> >> > > >> >> >> > > OK, we're making an IDS. But I think we need to be more >> >> >> > specific. >> >> >> > In particular, >> >> >> > > we need to answer the question of "who." >> >> >> > > >> >> >> > > Since the DHS has provided money, I suspect there would be >> >> >> > an >> >> >> > automatic >> >> >> > > assumption that this is a heavy-duty device intended for use >> >> >> > to >> >> >> > protect major >> >> >> > > servers and nodes in the critical information >> >> >> > infrastructure. >> >> >> > (Whatever that >> >> >> > > means.) This kind of thing is built by professionals, for >> >> >> > professionals. Trained >> >> >> > > people. >> >> >> > > >> >> >> > > However, given the current computing environment, I think it >> >> >> > would >> >> >> > be relatively >> >> >> > > easy to make a case that such a device is not going to do >> >> >> > all >> >> >> > that >> >> >> > much good. That >> >> >> > > a more accessible device, intended for "Grannyx" users, >> >> >> > would >> >> >> > actually do more to >> >> >> > > protect the infrastructure. After all, it isn't major nodes >> >> >> > on >> >> >> > the net that make up >> >> >> > > botnets, it's the little guys. Protect them, and you reduce >> >> >> > the >> >> >> > threat. This is the >> >> >> > > "low hanging fruit" for the blackhats, so protecting that >> >> >> > crop >> >> >> > is >> >> >> > going to give us >> >> >> > > the greatest benefit for the commitment of resources. >> >> >> > > >> >> >> > > This makes a difference. Not, perhaps, in terms of >> >> >> > questions >> >> >> > about multithreading >> >> >> > > analysis streams using graphics co-processors. But >> >> >> > certainly >> >> >> > in >> >> >> > most other areas. >> >> >> > > >> >> >> > > We've talked about extensibility. If we create a standard >> >> >> > template or format for >> >> >> > > signatures, the "who" makes a difference. Professionals >> >> >> > need a >> >> >> > warning and a >> >> >> > > packet. Grannyx users need a warning, no packet, a clear >> >> >> > explanation of what and >> >> >> > > how important, and a recommended course of action. Makes a >> >> >> > difference to the >> >> >> > > template. >> >> >> > > >> >> >> > > In terms of my recommendation of a paran-o-meter, it makes a >> >> >> > difference. >> >> >> > > Actually, I see huge debates over initial settings: do we >> >> >> > keep >> >> >> > it >> >> >> > low to keep from >> >> >> > > crying wolf, or keep it high to keep people as safe as >> >> >> > possible. >> >> >> > But one thing that >> >> >> > > should be done is make the paranoia settings >> >> >> > not-quite-obvious >> >> >> > up >> >> >> > front, so that >> >> >> > > somebody needs to know a little about the implications >> >> >> > before >> >> >> > they >> >> >> > start fiddling >> >> >> > > with settings. >> >> >> > > >> >> >> > > Heck, if it's a professional device, we don't need to worry >> >> >> > about >> >> >> > the interface at >> >> >> > > all. If it's for Granny, we definitely do. >> >> >> > > >> >> >> > > It also makes a difference in terms of the technology to be >> >> >> > included. If it is for >> >> >> > > professionals, we can throw in everything. If for Granny, >> >> >> > we >> >> >> > need >> >> >> > to make a >> >> >> > > careful choice about maximum protection for minimum >> >> >> > performance >> >> >> > drain. >> >> >> > > >> >> >> > > ====================== (quote inserted randomly by Pegasus >> >> >> > Mailer) >> >> >> > > rslade at vcn.bc.ca >> >> >> > slade at victoria.tc.ca >> >> >> > rslade at computercrime.org >> >> >> > > I'm getting so absent-minded that sometimes in the >> >> >> > middle >> >> >> > of >> >> >> > > victoria.tc.ca/techrev/rms.htm >> >> >> > >> >> >> > blogs.securiteam.com/index.php/archives/author/p1/ >> >> >> > >> >> >> > > _______________________________________________ >> >> >> > > Discussion mailing list >> >> >> > > Discussion at openinfosecfoundation.org >> >> >> > >> >> >> > > >> >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > > >> >> >> > > >> >> >> > >> >> >> > _______________________________________________ >> >> >> > Discussion mailing list >> >> >> > Discussion at openinfosecfoundation.org >> >> >> > >> >> >> > >> >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------------ >> >> >> > >> >> >> > _______________________________________________ >> >> >> > Discussion mailing list >> >> >> > Discussion at openinfosecfoundation.org >> >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> >> >> >> -- >> >> >> -------------------------------------------- >> >> >> Matthew Jonkman >> >> >> Emerging Threats >> >> >> Phone 765-429-0398 >> >> >> Fax 312-264-0205 >> >> >> http://www.emergingthreats.net >> >> >> -------------------------------------------- >> >> >> >> >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> >> >> >> >> >> > >> >> > >> >> > _______________________________________________ >> >> > Discussion mailing list >> >> > Discussion at openinfosecfoundation.org >> >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > >> >> > >> > >> > > > From mcholste at gmail.com Sun Oct 19 19:37:53 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 18:37:53 -0500 Subject: [Discussion] RFC: Proposal for Analysis Framework Message-ID: This is my first draft for describing the analysis framework that I think we need to combat current and future threats. Identifying the flaws in this first draft will allow us to more clearly state what we do and do not want to accomplish and what will and will not work to that end. Please comment liberally as to what you agree and disagree with. I will begin with the overall goals, and then describe an outline for a general technical architecture which aims to complete those goals. Then I will propose a specific technical implementation of the general technical architecture with an example of how the implementation would execute, followed by a high-level list of the sub-projects required to construct the framework. Goal One: Identify Contexts of Network Traffic Determining behavior means determining action within a context. Therefore the prerequisite to identifying behavior means establishing a context for all actions. This means converting packets into streams, streams into applications, and applications into contexts. Contexts can be simple like "an HTTP download has occurred" or more specific like "a Google search" or "an update from Microsoft." Goal Two: Analyze Contexts to Classify Behavior The identified contexts are passed to a higher level framework for classifying the behavior of traffic based on its context. This requires applying information available from a variety of sources to given contexts including performing heuristic-based analysis possibly using sandboxing technology, applying reputations from external sources, applying long-term contextual analysis with historical information, and consulting data describing the current environment. If the resulting classification of the context is considered malicious, it is forwarded on to the result management portion of the framework. Goal Three: Manage and Communicate Results Once traffic is classified as malicious, the information needs to be communicated in a way that can be acted upon. This means storing the result and delivering the result to the framework's administrators running the local implementation. This also includes automatically formatting the results and delivering them to a community repository. The mechanism for doing this must be extremely flexible and extensible as it will require many organization-specific customizations. The framework must allow for configuration of the components, monitoring of the components' performance, archival of results, an extensive query interface, communication of results in a sanitized way, and extensive reporting capabilities. This needs to be performed in a scalable way which achieves as close to realtime performance as possible. Goal Four: Act on Results Managing results includes defining which results require further action. The framework must provide actions which can be taken on the results. This includes the act of preventing intrusions by blocking network traffic and initializing incident handling procedures. The implementations of these goals will vary so greatly between organizations that the framework must provide a plugin architecture for interfacing with it. Example plugins providing the most common actions should be included with the framework, but the actual creation of plugins should be considered out of scope of the project proper and instead be considered a separate (but important) side-project. General Architecture To establish network contexts and trigger on specific traffic, the basic component will be a network inspection system capable of full-content capture, session capture, and event generation from predefined criteria. The inspection system will use these predefined criteria to parse and contextualize traffic according to its layer 7 attributes. Predefined triggers will forward noteworthy contexts to the analysis framework where correlation and high level analysis will take place. This framework will be built from high level, modular code which focuses on moving information asynchronously so that any given step in the analysis will not block the analysis of other contexts. This framework will act as the clutch between the low-level, packet-bound inspection system and the high-level user interaction. User interaction will occur primarily via a web console extensive enough to provide a way to configure, monitor, and manage the system as it runs, as well as a way of viewing, querying, and acting upon those results. The analysis framework will be the bridge between the web console and the inspection system, updating the web console in realtime, and allowing the administrator to issue actions for results. Result actions will include active response packet dropping by sending reset packets to the offending connections, routing table changes to employ blackhole route filtering, configurable email notifications, and list submission mechanisms. Specific Implementation Network Inspection: Full content capture: Time Machine < http://www.net.t-labs.tu-berlin.de/research/tm/TM_HOWTO_tm-20080814-0> Inspection: Bro Flow data: Netflow (via routers or something like nProbe) < http://www.ntop.org/nprobe.html> Communications: Broccoli plus a newly created Perl plugin similar to the existing python plugin Context Analysis and Result Management: I propose a framework for content analysis based on Perl using the POE framework. Perl is the right "weight" of language in that it can perform low-level tasks like sending raw network packets while providing access to a rich code base of high-level tasks like analyzing pcaps, checking client patch levels and asset information in a CMDB, and performing web queries. The POE event framework for Perl is a robust, mature, formal framework for asynchronous event handling. It specializes in non-blocking input/output operations with a built-in framework for communicating natively between nodes either on the same host or across a network. It has an extensive code base created by a large user community and is extremely scalable and flexible. Proof of the compatibility of its design goals can be found in the fact that the project's creator, Rocco Caputo, made a module specifically designed to tail Snort log files and take actions on events. POE also has the great strength of being the perfect server-side partner for AJAX requests from a web browser. This would allow for a thin layer of separation between what occurs on the backend traffic inspection nodes and what is communicated to the end user via a web console, unifying the two technologies. This advantage is critical because it reduces the amount of overall code and complexity of the project and encourages a modular framework in which functionality for inspection and functionality for management can be created in using the same technology and with a minimal amount of separation. Platform: Perl + POE Blacklist sharing: centralized XML submission to Emerging Threats Result Actions: Forged packet active response via sending RST/ICMP prohibited packets via perl Routing table alteration via active router configuration via perl or on-board routing daemon like Zebra Email templates for notification/incident creation Example Execution Path Here is an example of how the proposed technical implementation would detect zero-day malware from an unknown IP: 1. A user inside the org is clickjacked and downloads new malware from a previously unknown IP. 2. This matches predefined criteria instructing Bro to begin extracting files from from the network stream of any HTTP download with an executable header and to forward the extracted file and related netflow to the event analysis framework via the Perl-Broccoli connection. 3. The Perl event analysis framework receives this new event complete with flows and files extracted from HTTP. 4. The extracted files are submitted to CWSandbox via a framework plugin for CWSandbox, which in turn automatically submits to VirusTotal. 5. CWSandbox emails the results back to the framework, which parses the email and stores the results. The framework checks the CWSandbox network activity results and observes that an outbound connection to a separate IP was reported. 6. The CWSandbox output and VirusTotal output show that the download was malicious and detected by several AV vendors. Both IP addresses are marked as malicious. 7. The framework issues a query to Time Machine via the Time Machine remote query console for all packets to and from the second malicious IP address. 8. The framework uses the Net::Analysis Perl module to analyze the pcap from Time Machine and determines that there were packets present and that they were not all unanswered SYN packets, indicating that this host was successfully compromised and that there was possible data leakage. 9. The framework creates a report of the event and issues a high-priority notification to the organization's network security admin staff. It then submits the malware sample to Symantec's web submit page, which will create rapid response definitions to remove the malware. 10. The framework updates the organization's blackhole router to include both malicious IP's, preventing any further data leakage and shielding the other clients. 11. The framework downloads and installs the new rapid response definitions, mounts the infected client's C$ share via samba, initiates a scan of the network share, and either cleans the malware or sends an email to the help desk system requesting a re-imaging of the client. 12. The framework submits both of the malicious IP's to Emerging Threats' blacklist. 13. ET adds the IP's to the malware blacklist. 14. Another organization's IDS framework checks in and updates its lists of bad IP's from ET. 15. The other org's framework notices the new IP's which are added and updates its blackhole routes accordingly, protecting its org. 16. Fifteen minutes later, the network security staff at the original org return from lunch and read the report of all that occurred while they were at Chotchkies. Tasks: The tasks for this implementation can be broken down into sub-projects. - Create a Perl interface for Bro's Broccoli - Create an ET repository for Bro scripts - Create the Perl analysis framework - Create analysis framework plugins for various tasks - Create a standard format for blacklists - Create a web console frontend for the framework Some of these are some huge tasks, but there is such a large amount of code already available for Perl on CPAN, that the framework really consists of knitting together different modules to make a coherent one. Also, a lot of it can be done in parallel by separate working groups. A note about other technologies: there are lots of ways to implement what I've discussed, what I listed as technologies for implementation were ones which already do what I've discussed or which I have working prototypes for. I hope this proposal starts a discussion on both the design and implementation choices I've laid out here. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/c2b3f581/attachment-0001.html From jeremy at sudosecure.net Sun Oct 19 20:46:14 2008 From: jeremy at sudosecure.net (Jeremy) Date: Sun, 19 Oct 2008 19:46:14 -0500 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> Message-ID: <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> Thought of another feature I would like to see.... [not sure of the number] I would like to see a single capture engine have the ability to perform the analysis of the data (however that may be), and also this same engine should be able to capture session statistics (flow data), and full packet captures. What I mean by this is that currently I run 3 applications on all of my IDS' to get this data. I use SANCP for session data, Tcpdump for full packet capture and snort for IDS functionality. I believe it would improve overall system performance to only have one application read the packets from the capture library such as libpcap instead of several applications having to read this data. I know that I ran into a wall with the 32 bit architecture in that I could only allocate small amounts of Low memory space to each application, as the kernel can only be allocated 1 GB. This 1 GB has to be divided up between with these three applications and well all other applications. I recently was told you could actually allocate 2 GB low memory on a 32 bit architecture, but as of today I have not figured out how. If you over run the low memory space reserved by the kernel you end up with oom kills which can be really frustrating as it will kill off applications based off some really odd/cryptic algorithm. Anyways getting back to one application performing all three tasks, the packet only needs to be copied once and therefore I could allocate a substantial amount of the low memory reserved by the kernel to this application. Now I know not everyone will want to capture flow data, and full packet data on their IDS boxes, but if this was a configurable option it may help some of us. I also noticed someone else posted about SANCP indexing abilities, this would be nice to have in the new engine as well to make correlating Session data (flow data) to full packet captures very fast and easy. Having the full packet captures on my IDS boxes has really helped in identifying false positives and signature tweaking. The session data (flow data) has proven to be an outstanding addition to our anomaly detection capabilities. --jeremy On Sat, Oct 18, 2008 at 9:17 PM, th3 m0nq wrote: > > As Robert and Martin eluded in an other > thread, any sort of correlation and IP analysis, scoring, > severity-bumping, etc, is best done in the analysis or portal engine. > > > I agree here, but does next generation mean an advanced version of > snort? Does next generation mean an evolution in terms of thinking > about IDS/IPS? Do we leave things as they are and say that there > exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system > for my network? I would like to see an evolution of such technologies > be more biological in nature. > > > I don't think any existing detection engine of an IDS is the proper place. > But perhaps a parallel engine that collects network stream data and does > matching on hostile IP's, and then checks (perhaps by packet/session > timestamp) if the IDS did produce an alert in a recent time window > (since the session-IP-eval-engine will surely lag behind) > > > I am with you here as well. Why is the parallel engine not part of > the IDS? Why should next generation IDS mean a better snort? Why > can't it mean a better way of looking at network security? > http://en.wikipedia.org/wiki/Immune_system#Layered_defense What about > an all encompassing IDS/IPS _system_? > > I am not saying I have the answers, or something similar to an expert > system is the answer. I am saying I think the issue should be looked > at as we dump the current way we do things in a temporary holding > cell, and think in evolutionary terms. This makes 4 cents ;-). > > > Adrian > > > > > > > > > On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe wrote: >> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: >>> [..] I think there is >>> room, at minimum, to have packets that do not necessarily trigger a >>> rule "flagged" in some way in an IDS. This could be implemented as a >>> plug-in component of the system. This may be way outside of the scope >>> of what you guys are looking at, but that's my 2 cents. >> >> Receiving alerts (with session content) on IP's that don't trip existing >> signatures, but are present in a hostile-IP list, is a good idea (for >> example, to detect new evasion techniques of existing badware). >> >> However, I don't think the sensor is the proper place for that due to >> the immense IP-matching workload that would have to occur on a >> per-packet or per-session basis (remember, we're talking at least tens >> of thousands hostile IP's). As Robert and Martin eluded in an other >> thread, any sort of correlation and IP analysis, scoring, >> severity-bumping, etc, is best done in the analysis or portal engine. >> (We do those things in our shop as well, with back-end scripts and our >> portal). >> >> While you can make decisions on the IP "badness value" for analysis, you >> would of course miss the content of the IP not receiving an alert. I >> don't think any existing detection engine of an IDS is the proper place. >> But perhaps a parallel engine that collects network stream data and does >> matching on hostile IP's, and then checks (perhaps by packet/session >> timestamp) if the IDS did produce an alert in a recent time window >> (since the session-IP-eval-engine will surely lag behind). That check >> would probably be best done against the IDS/portal database. A simple >> bpf filter with hostile 20,000+ IP's to capture content will probably >> not cut it :) >> >> -Frank >> >> >> -- >> It is said that the Internet is a public utility. As such, it is best >> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >> against your ports. >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From frank at knobbe.us Sun Oct 19 21:36:11 2008 From: frank at knobbe.us (Frank Knobbe) Date: Sun, 19 Oct 2008 20:36:11 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: <1224466571.20986.14.camel@localhost> On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: > Right, but I envision the XML to be the source that scripts would > parse into whatever is needed, like router config, dns blocklists, > host files, search engine blacklists, etc. The key would be to create > a standard capable of being specific enough to feed the lowest common > demoninator. Just be aware that there are lots and lots of hostile IP's. I'm not sure XML is the proper format to deliver those since that data file would balloon quite drastically :) -Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/ae81b7d2/attachment.bin From jeremy at sudosecure.net Sun Oct 19 21:46:29 2008 From: jeremy at sudosecure.net (Jeremy) Date: Sun, 19 Oct 2008 20:46:29 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <1224466571.20986.14.camel@localhost> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> Message-ID: <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> I would have to agree with this and would venture to say a custom indexed binary format would be best for this, and not a plain text xml file. Much like Maxminds GeoIP and ASN database files. --jeremy On Sun, Oct 19, 2008 at 8:36 PM, Frank Knobbe wrote: > On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: >> Right, but I envision the XML to be the source that scripts would >> parse into whatever is needed, like router config, dns blocklists, >> host files, search engine blacklists, etc. The key would be to create >> a standard capable of being specific enough to feed the lowest common >> demoninator. > > Just be aware that there are lots and lots of hostile IP's. I'm not sure > XML is the proper format to deliver those since that data file would > balloon quite drastically :) > > -Frank > > > > > -- > It is said that the Internet is a public utility. As such, it is best > compared to a sewer. A big, fat pipe with a bunch of crap sloshing > against your ports. > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From david.glosser at gmail.com Sun Oct 19 21:48:57 2008 From: david.glosser at gmail.com (David Glosser) Date: Sun, 19 Oct 2008 21:48:57 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <1224466571.20986.14.camel@localhost> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> Message-ID: Based on other conversations with Matt concerning malwaredomains & the dns-bh list, there could be several BadNess lists/rules, hostility within the previous 24 hours, previous 48 hrs, previous 96 hrs, 1 week, etc. Then data file would only be super large for the full "BadNess" list and a few of the long ones. Of course, "Enumerating Badness" is still reactive. Matt, I know you've done things with predictive blacklisting..... http://www.emergingthreats.net/content/view/88/1/). Could this research be leveraged as well? On Sun, Oct 19, 2008 at 9:36 PM, Frank Knobbe wrote: > On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: >> Right, but I envision the XML to be the source that scripts would >> parse into whatever is needed, like router config, dns blocklists, >> host files, search engine blacklists, etc. The key would be to create >> a standard capable of being specific enough to feed the lowest common >> demoninator. > > Just be aware that there are lots and lots of hostile IP's. I'm not sure > XML is the proper format to deliver those since that data file would > balloon quite drastically :) > > -Frank > > > > > -- > It is said that the Internet is a public utility. As such, it is best > compared to a sewer. A big, fat pipe with a bunch of crap sloshing > against your ports. > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From jlewis at packetnexus.com Sun Oct 19 21:49:48 2008 From: jlewis at packetnexus.com (Jason Lewis) Date: Sun, 19 Oct 2008 21:49:48 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> Message-ID: <48FBE3BC.4000305@packetnexus.com> I think you have to do both. XML for data description and communications....and the binary for passing the actual data. Jeremy wrote: > I would have to agree with this and would venture to say a custom > indexed binary format would be best for this, and not a plain text xml > file. Much like Maxminds GeoIP and ASN database files. > > --jeremy > > On Sun, Oct 19, 2008 at 8:36 PM, Frank Knobbe wrote: > >> On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: >> >>> Right, but I envision the XML to be the source that scripts would >>> parse into whatever is needed, like router config, dns blocklists, >>> host files, search engine blacklists, etc. The key would be to create >>> a standard capable of being specific enough to feed the lowest common >>> demoninator. >>> >> Just be aware that there are lots and lots of hostile IP's. I'm not sure >> XML is the proper format to deliver those since that data file would >> balloon quite drastically :) >> >> -Frank >> >> >> >> >> -- >> It is said that the Internet is a public utility. As such, it is best >> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >> against your ports. >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From jlewis at packetnexus.com Sun Oct 19 22:56:52 2008 From: jlewis at packetnexus.com (Jason Lewis) Date: Sun, 19 Oct 2008 22:56:52 -0400 Subject: [Discussion] RFC: Proposal for Analysis Framework In-Reply-To: References: Message-ID: <48FBF374.3010409@packetnexus.com> It might help to compile a list of data that should be processed and exported. I'll try to get the ball rolling with potential data sets. For example, the system would process: network traffic malware data correlation data from other systems and export: blacklists (IP, ASN,etc) alerts reports From mcholste at gmail.com Sun Oct 19 23:54:38 2008 From: mcholste at gmail.com (Martin Holste) Date: Sun, 19 Oct 2008 22:54:38 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FBE3BC.4000305@packetnexus.com> References: <48FA1839.14684.23E77B8@localhost> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> Message-ID: Right, you could do database-driven lookups based on addresses contained in the binary format which would return a descriptive result. As a regular user of their database, I agree with Jeremy that the MaxMind model is a good one for this kind of thing. On Sun, Oct 19, 2008 at 8:49 PM, Jason Lewis wrote: > I think you have to do both. XML for data description and > communications....and the binary for passing the actual data. > > Jeremy wrote: > > I would have to agree with this and would venture to say a custom > > indexed binary format would be best for this, and not a plain text xml > > file. Much like Maxminds GeoIP and ASN database files. > > > > --jeremy > > > > On Sun, Oct 19, 2008 at 8:36 PM, Frank Knobbe wrote: > > > >> On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: > >> > >>> Right, but I envision the XML to be the source that scripts would > >>> parse into whatever is needed, like router config, dns blocklists, > >>> host files, search engine blacklists, etc. The key would be to create > >>> a standard capable of being specific enough to feed the lowest common > >>> demoninator. > >>> > >> Just be aware that there are lots and lots of hostile IP's. I'm not sure > >> XML is the proper format to deliver those since that data file would > >> balloon quite drastically :) > >> > >> -Frank > >> > >> > >> > >> > >> -- > >> It is said that the Internet is a public utility. As such, it is best > >> compared to a sewer. A big, fat pipe with a bunch of crap sloshing > >> against your ports. > >> > >> > >> _______________________________________________ > >> Discussion mailing list > >> Discussion at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > >> > >> > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081019/ab0636fa/attachment.html From pepperjack at afferentsecurity.com Mon Oct 20 09:31:38 2008 From: pepperjack at afferentsecurity.com (Jack Pepper) Date: Mon, 20 Oct 2008 08:31:38 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FB64C1.4000803@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: <20081020083138.tncynmtz4wg84wsw@mail.afferentsecurity.com> Quoting Matt Jonkman : > I really REALLY am not excited about trying to make a windows client. > Not only does that open up a huge responsibility in support and the > inevitable bluescreens, but I have had a difficult time over the years > believing that any process on ANY os (especially windows) could be > trusted and independant enough to watch itself. Take into account how > easy it is for trojans and rootkits to shut down antivirus, or blind it. > And these are products with hundreds of the most skilled coders around > working on them. The problem with client side technical controls is not a technical problem. Several people have solved it several ways. But paying customers won't buy it. Home users won't use it. Techies admire the novelty, but they are not the "at risk" community. Client side is simply not possible due to political and religious issues. > I know we're sharp as a community, but I don't think that's a battle we > want to get in to. So how can we do it at the network layer? I beat my head against that wall for years before I finally gave up on end user security. Screw 'em. The only viable approach is to make the network safe, in spite of user misconduct. You should assume Granny has an infected workstation and spends all her idle time trying to hack the IDS and compromise every other machine on the internet. (I will spare the list from a rant on the subject of NAC technology) jp -- Framework? I don't need no stinking framework! ---------------------------------------------------------------- @fferent Security Labs: Isolate/Insulate/Innovate http://www.afferentsecurity.com From jonkman at jonkmans.com Mon Oct 20 13:42:34 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 20 Oct 2008 13:42:34 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> Message-ID: <48FCC30A.3010401@jonkmans.com> David Glosser wrote: > Based on other conversations with Matt concerning malwaredomains & the > dns-bh list, there could be several BadNess lists/rules, hostility > within the previous 24 hours, previous 48 hrs, previous 96 hrs, 1 > week, etc. > > Then data file would only be super large for the full "BadNess" list > and a few of the long ones. > > Of course, "Enumerating Badness" is still reactive. > > Matt, I know you've done things with predictive blacklisting..... > http://www.emergingthreats.net/content/view/88/1/). Could this > research be leveraged as well? Just updated our website backend so the link you were probably looking for is: http://www.emergingthreats.net/index.php/component/content/article/18-research/88-hpb.html This was work Dshield is publishing, which is really good stuff. I hope we can contribute to and benefit from it in the engine. But yes, ip reputation even is still reactive, but if published quickly enough I think it's still very effective. matt > > > > > On Sun, Oct 19, 2008 at 9:36 PM, Frank Knobbe wrote: >> On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: >>> Right, but I envision the XML to be the source that scripts would >>> parse into whatever is needed, like router config, dns blocklists, >>> host files, search engine blacklists, etc. The key would be to create >>> a standard capable of being specific enough to feed the lowest common >>> demoninator. >> Just be aware that there are lots and lots of hostile IP's. I'm not sure >> XML is the proper format to deliver those since that data file would >> balloon quite drastically :) >> >> -Frank >> >> >> >> >> -- >> It is said that the Internet is a public utility. As such, it is best >> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >> against your ports. >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Mon Oct 20 13:44:45 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 20 Oct 2008 13:44:45 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FBE3BC.4000305@packetnexus.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> Message-ID: <48FCC38D.3040003@jonkmans.com> I think it less important (at least at this point) that we identify the exact format than to decide the kind of data we'll deal in as mentioned in another post. How it's stored in the engine in ways that it's quickly usable is a separate issue. The push mechanism I don't see how we can move that much data without it being a binary stream. The overhead of xml and others is way to high for moving thousands of blocks/reputation changes. But we can easily have that data available to be pushed to outside tools similar to unified_output into whatever format the end tool requires. Matt Jason Lewis wrote: > I think you have to do both. XML for data description and > communications....and the binary for passing the actual data. > > Jeremy wrote: >> I would have to agree with this and would venture to say a custom >> indexed binary format would be best for this, and not a plain text xml >> file. Much like Maxminds GeoIP and ASN database files. >> >> --jeremy >> >> On Sun, Oct 19, 2008 at 8:36 PM, Frank Knobbe wrote: >> >>> On Sun, 2008-10-19 at 14:30 -0500, Martin Holste wrote: >>> >>>> Right, but I envision the XML to be the source that scripts would >>>> parse into whatever is needed, like router config, dns blocklists, >>>> host files, search engine blacklists, etc. The key would be to create >>>> a standard capable of being specific enough to feed the lowest common >>>> demoninator. >>>> >>> Just be aware that there are lots and lots of hostile IP's. I'm not sure >>> XML is the proper format to deliver those since that data file would >>> balloon quite drastically :) >>> >>> -Frank >>> >>> >>> >>> >>> -- >>> It is said that the Internet is a public utility. As such, it is best >>> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >>> against your ports. >>> >>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Mon Oct 20 13:48:52 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 20 Oct 2008 13:48:52 -0400 Subject: [Discussion] Features In-Reply-To: <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> Message-ID: <48FCC484.9080308@jonkmans.com> I agree. I know a lot of installs that use rotating tcpdump files which allows the analyst to go back and see everything that happened related to an incident. And as long as you have good disks it's a very feasible thing to do. If you're running inline you can even do the detection from these files to avoid packet loss at peak traffic flows. Definitely adding this to the list! Matt Jeremy wrote: > Thought of another feature I would like to see.... > > [not sure of the number] > I would like to see a single capture engine have the ability to > perform the analysis of the data (however that may be), and also this > same engine should be able to capture session statistics (flow data), > and full packet captures. What I mean by this is that currently I run > 3 applications on all of my IDS' to get this data. I use SANCP for > session data, Tcpdump for full packet capture and snort for IDS > functionality. I believe it would improve overall system performance > to only have one application read the packets from the capture library > such as libpcap instead of several applications having to read this > data. I know that I ran into a wall with the 32 bit architecture in > that I could only allocate small amounts of Low memory space to each > application, as the kernel can only be allocated 1 GB. This 1 GB has > to be divided up between with these three applications and well all > other applications. I recently was told you could actually allocate 2 > GB low memory on a 32 bit architecture, but as of today I have not > figured out how. If you over run the low memory space reserved by the > kernel you end up with oom kills which can be really frustrating as it > will kill off applications based off some really odd/cryptic > algorithm. Anyways getting back to one application performing all > three tasks, the packet only needs to be copied once and therefore I > could allocate a substantial amount of the low memory reserved by the > kernel to this application. Now I know not everyone will want to > capture flow data, and full packet data on their IDS boxes, but if > this was a configurable option it may help some of us. I also noticed > someone else posted about SANCP indexing abilities, this would be nice > to have in the new engine as well to make correlating Session data > (flow data) to full packet captures very fast and easy. Having the > full packet captures on my IDS boxes has really helped in identifying > false positives and signature tweaking. The session data (flow data) > has proven to be an outstanding addition to our anomaly detection > capabilities. > > > --jeremy > > On Sat, Oct 18, 2008 at 9:17 PM, th3 m0nq wrote: >> >> As Robert and Martin eluded in an other >> thread, any sort of correlation and IP analysis, scoring, >> severity-bumping, etc, is best done in the analysis or portal engine. >> >> >> I agree here, but does next generation mean an advanced version of >> snort? Does next generation mean an evolution in terms of thinking >> about IDS/IPS? Do we leave things as they are and say that there >> exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system >> for my network? I would like to see an evolution of such technologies >> be more biological in nature. >> >> >> I don't think any existing detection engine of an IDS is the proper place. >> But perhaps a parallel engine that collects network stream data and does >> matching on hostile IP's, and then checks (perhaps by packet/session >> timestamp) if the IDS did produce an alert in a recent time window >> (since the session-IP-eval-engine will surely lag behind) >> >> >> I am with you here as well. Why is the parallel engine not part of >> the IDS? Why should next generation IDS mean a better snort? Why >> can't it mean a better way of looking at network security? >> http://en.wikipedia.org/wiki/Immune_system#Layered_defense What about >> an all encompassing IDS/IPS _system_? >> >> I am not saying I have the answers, or something similar to an expert >> system is the answer. I am saying I think the issue should be looked >> at as we dump the current way we do things in a temporary holding >> cell, and think in evolutionary terms. This makes 4 cents ;-). >> >> >> Adrian >> >> >> >> >> >> >> >> >> On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe wrote: >>> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: >>>> [..] I think there is >>>> room, at minimum, to have packets that do not necessarily trigger a >>>> rule "flagged" in some way in an IDS. This could be implemented as a >>>> plug-in component of the system. This may be way outside of the scope >>>> of what you guys are looking at, but that's my 2 cents. >>> Receiving alerts (with session content) on IP's that don't trip existing >>> signatures, but are present in a hostile-IP list, is a good idea (for >>> example, to detect new evasion techniques of existing badware). >>> >>> However, I don't think the sensor is the proper place for that due to >>> the immense IP-matching workload that would have to occur on a >>> per-packet or per-session basis (remember, we're talking at least tens >>> of thousands hostile IP's). As Robert and Martin eluded in an other >>> thread, any sort of correlation and IP analysis, scoring, >>> severity-bumping, etc, is best done in the analysis or portal engine. >>> (We do those things in our shop as well, with back-end scripts and our >>> portal). >>> >>> While you can make decisions on the IP "badness value" for analysis, you >>> would of course miss the content of the IP not receiving an alert. I >>> don't think any existing detection engine of an IDS is the proper place. >>> But perhaps a parallel engine that collects network stream data and does >>> matching on hostile IP's, and then checks (perhaps by packet/session >>> timestamp) if the IDS did produce an alert in a recent time window >>> (since the session-IP-eval-engine will surely lag behind). That check >>> would probably be best done against the IDS/portal database. A simple >>> bpf filter with hostile 20,000+ IP's to capture content will probably >>> not cut it :) >>> >>> -Frank >>> >>> >>> -- >>> It is said that the Internet is a public utility. As such, it is best >>> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >>> against your ports. >>> >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Mon Oct 20 14:05:16 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 20 Oct 2008 14:05:16 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: <48FCC85C.1080307@jonkmans.com> As for scope: here's my plan. I definitely want us to push every idea and sub-project we can here. We'll get it all on the list and consider it all as a whole. Down the road, closer to the time we'll start major coding, we can take the entire list and as a community (maybe voting-style, who knows) and whittle down to what we want in the initial release, what down the road, what needs to be a separate project (and spawn those subprojects), and what is just not feasible. So keep the discussion going, lets not limit based on what might seem to large a scope. We may end up moving our focus to be right in the middle of feature X, we just don't know. We have time, lets explore everything for a while and see where we end up. Matt Martin Holste wrote: > I vehemently agree with Matt about the necessity of staying out of the > client-side arena. I think that if we're going to actually produce > anything of value, we need to narrow the scope to start with. My vote > is that we consider notifying Granny that there's something bad on her > PC out of scope until we figure out how to notify Granny's ISP's network > security admin that there's something bad. If we can figure out how to > reliably find successful compromises and get that information output as > actionable intelligence to the netsec community, the rest will come in time. > > For instance, if we can get to the point where we're %99.99 sure that a > given URL is bad, then we could work on setting up an alliance with a > place like Google, which already has an infrastructure for end-user > notification of bad sites (as in, the "this site may harm your computer" > displayed under links to sql injected pages). I'm sure Neils Provos > would be interested in hints from this group. I would say that blackhat > SEO is 80% of the malware infection vector right now, so that would be huge. > > Someday even farther in the future, we could use a platform like OSSEC > HIDS to deliver our blacklists, or a browser plugin, but that is a > carrot on a stick for another day. > > The point is that we need to start by doing one thing and doing it well: > provide the most accurate method of network-based detection and > prevention of compromises to date by leveraging the collective wisdom of > the community. > > On Sun, Oct 19, 2008 at 11:48 AM, Matt Jonkman > wrote: > > Martin Holste wrote: > > I'm guessing that the intent is to be the "detection" component of the > > Trusted Internet Connection (TIC) > > > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monitor.html), > > in which the mandate is to protect all federal assets, especially the > > federal desktop environment. > > Certainly a sound approach to tackle such a problem. Glad I don't have > to solve that one, it's huge among gov't agencies. Especially in the US, > but surely similar everywhere. > > > Even if DHS has different goals, I think > > that this group could do a great service by providing the TIC's > > protection geared toward the client-side, and I think for most > sectors, > > that's where the most imminent threats lie. Again, I'm certainly not > > saying that defending servers is out of scope, just that IDS has > > historically been about defending servers as the primary goal, with > > clients secondary, and I think that needs to be reversed now. > > I agree. At emerging threats we've been lately heavy into post-infection > signatures in our research. There are many vulnerabilities out there in > client apps, but I'd venture to say that a very high percentage of the > infections out there aren't the result of a remotely exploitable > vulnerability. Most come from users clicking on email attachments, > installing fake software/codecs, and visiting websites with hostile > code. > > We've tried many times to write effective signatures to detect hostile > html/java/gifs, etc. It's just not feasible as is. Code is too flexible > for signature-based approaches. You can say the same thing a hundred > ways, especially in html. > > So how can we go after client side? > > I really REALLY am not excited about trying to make a windows client. > Not only does that open up a huge responsibility in support and the > inevitable bluescreens, but I have had a difficult time over the years > believing that any process on ANY os (especially windows) could be > trusted and independant enough to watch itself. Take into account how > easy it is for trojans and rootkits to shut down antivirus, or blind it. > And these are products with hundreds of the most skilled coders around > working on them. > > I know we're sharp as a community, but I don't think that's a battle we > want to get in to. So how can we do it at the network layer? > > Sandboxing? > > Virtual emulators? > > Or do we continue the thinking that for any infection to be of any use > to anyone it has to generate traffic? It has to call home, send spam, > report stolen information, something. So we concentrate on detecting and > stopping the post infection? > > What does everyone think about that? > > Matt > > > > > > --Martin > > > > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > > > >> wrote: > > > > I doubt the intent of the DHS is to simply do good, they are > most likely > > much more focused on producing technology that allows them to > > detect/mitigate/prevent attacks against critical components. > This of > > course means detecting attacks that fly below the threshold of > detection > > for todays technology. If it comes to "doing good" or > detecting state > > sponsored attacks against critical components (think custom > attacks > > against unknown vulns), i'm going to go out on a limb and say > they would > > rather protect the critical component vs the enduser. > > > > What you are discussing still has value and merit but im not > so sure it > > is what should be focused on, but of course I am not the person to > > decide such things. > > > > Andre > > > > > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > > > Question: what are we making? Oh, yeah, I read the blurb: "The > > OISF has been > > > chartered and funded to build a next-generation intrusion > > detection and prevention > > > engine. This project will consider every new and existing > > technology, concept and > > > idea to build a completely open source licensed engine." > > > > > > OK, we're making an IDS. But I think we need to be more > specific. > > In particular, > > > we need to answer the question of "who." > > > > > > Since the DHS has provided money, I suspect there would be an > > automatic > > > assumption that this is a heavy-duty device intended for use to > > protect major > > > servers and nodes in the critical information infrastructure. > > (Whatever that > > > means.) This kind of thing is built by professionals, for > > professionals. Trained > > > people. > > > > > > However, given the current computing environment, I think it > would > > be relatively > > > easy to make a case that such a device is not going to do > all that > > much good. That > > > a more accessible device, intended for "Grannyx" users, would > > actually do more to > > > protect the infrastructure. After all, it isn't major nodes on > > the net that make up > > > botnets, it's the little guys. Protect them, and you reduce the > > threat. This is the > > > "low hanging fruit" for the blackhats, so protecting that > crop is > > going to give us > > > the greatest benefit for the commitment of resources. > > > > > > This makes a difference. Not, perhaps, in terms of questions > > about multithreading > > > analysis streams using graphics co-processors. But certainly in > > most other areas. > > > > > > We've talked about extensibility. If we create a standard > > template or format for > > > signatures, the "who" makes a difference. Professionals need a > > warning and a > > > packet. Grannyx users need a warning, no packet, a clear > > explanation of what and > > > how important, and a recommended course of action. Makes a > > difference to the > > > template. > > > > > > In terms of my recommendation of a paran-o-meter, it makes a > > difference. > > > Actually, I see huge debates over initial settings: do we > keep it > > low to keep from > > > crying wolf, or keep it high to keep people as safe as possible. > > But one thing that > > > should be done is make the paranoia settings > not-quite-obvious up > > front, so that > > > somebody needs to know a little about the implications > before they > > start fiddling > > > with settings. > > > > > > Heck, if it's a professional device, we don't need to worry > about > > the interface at > > > all. If it's for Granny, we definitely do. > > > > > > It also makes a difference in terms of the technology to be > > included. If it is for > > > professionals, we can throw in everything. If for Granny, > we need > > to make a > > > careful choice about maximum protection for minimum > performance drain. > > > > > > ====================== (quote inserted randomly by Pegasus > Mailer) > > > rslade at vcn.bc.ca > > > > slade at victoria.tc.ca > > > > rslade at computercrime.org > > > > > I'm getting so absent-minded that sometimes in the > middle of > > > victoria.tc.ca/techrev/rms.htm > > > > > blogs.securiteam.com/index.php/archives/author/p1/ > > > > > > _______________________________________________ > > > Discussion mailing list > > > Discussion at openinfosecfoundation.org > > > > > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From lists at inliniac.net Mon Oct 20 18:52:39 2008 From: lists at inliniac.net (Victor Julien) Date: Tue, 21 Oct 2008 00:52:39 +0200 Subject: [Discussion] IPS features: traffic normalization Message-ID: <48FD0BB7.3030008@inliniac.net> Since I became interested in security I've always been more interested in the prevention side than 'just' the detection side. Which explains my involvement in the Snort_inline project. I also have another project that is a firewall manager for Linux/iptables (it's called Vuurmuur, www.vuurmuur.org). One thing I'm very much interested in is advanced traffic normalization. In an inline setup, we control everything that is passed so we can rewrite it or replace it. In Snort_inline we do this in two ways: (1) some TCP packet rewriting, mostly to adjust the TCP window and (2) by the replace action, that allows you to replace a rule-specified string in the payload by a rule-specified string of the same length. It's all pretty limited, and I think extending this would be very useful. I think we should be able to have the engine normalize all packets we want it to. For example if we don't want to support TCP SACK, remove it from the packet. Same for TCP wscaling, etc. This is actually quite simple to implement. More interesting will be to have advanced data layer normalization. I think something like a regex search and replace would be great to have. One use case could be working around proxies leaking credentials (like Will wrote about here: http://node5.blogspot.com/2007/12/proxies-behaving-badly.html). We could have rules detecting and removing such leakage from a stream. Another even more advanced example would be to force randomization upon the stream, for example in the Kaminsky DNS issues forcing randomization upon the transaction ID space. Thoughts? Cheers, Victor From jim.mcquaid at gmail.com Tue Oct 21 07:25:34 2008 From: jim.mcquaid at gmail.com (James McQuaid) Date: Tue, 21 Oct 2008 07:25:34 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: References: Message-ID: Concur. Pushing reputational data to home users (with no interaction required by the home user) offers opportunities. WOT and the paid version of SiteAdvisor are attempting to do this in the browser, but require some user interaction. There are hundreds of thousands of misconfigured, compromised and ineffective home routers out there. If we could work with the manufacturers, it might be possible to return these devices to productive use. > Or do we go with just pushing reputational data to the home user? What I > mean is if we build this engine to generate and act upon IP reputation > data could we know enough about the Internet collectively to simply push > a blacklist to the home user's router/firewall? > > On the more sophisticated devices where software could be installed > maybe it does run a stripped down detection engine and help feed IP data > back to the group. But overall it's still primarily benefiting only from > the blacklisting and whitelisting of the whole? > > How many false positives would we encounter that might actually affect a > home user? We can take steps to ensure that this is not a big issue. > I think it'd be a very interesting day if we were to have essentially a > Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even > more IP mobile than they are now. > > Take atrivo/intercare/mccolo for example. Infested with crap, and have > been for years. But since they can't really be blocked on the backbone > home users still hit the same scam AV sites, give their credit card > info, and get screwed. We know the sites are there, the registrars won't > take them down, the ISP is colluding with the bad guys so they'll stay > online. What can we do? (besides scream to our representatives for more > effective laws) > > We can block those bad IPs at the home user's level. That'll make them > start moving of course, just like bots being used to spam until they're > listed. So we have to be able to immediately move quickly with the. Shut them out in real time with multiple daily updates; most of the data would not change, so the diff would usually be a very small file. > What does everyone think there? The basic idea being to use a normal > engine model by most security pro's to feed IP reputation into a global > database, and then the home user gets some sort of very basic tool or > button they can click on to benefit from that data? Maybe even feed back > to us. > > Matt > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > -- James McQuaid http://www.jamesmcquaid.com From jonkman at jonkmans.com Tue Oct 21 10:40:20 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Tue, 21 Oct 2008 10:40:20 -0400 Subject: [Discussion] First Brainstorming Session Scheduled Message-ID: <48FDE9D4.9000305@jonkmans.com> Our first in-person, up-close and personal Brainstorming Session is scheduled. I'll be at Deepsec in Vienna November 11-14. The first two days are training in which Victor Julien and I are doing a Protocol Analysis for Writing Snort Signatures class. (2 days, in depth, you should come!). Then we've got space in the general conference on the 13th and 14th for the Brainstorming Session. More on the schedule here: https://deepsec.net/schedule/ As you can see there are some great sessions there. Jose Nazario, Joe Stewart, Johnny Long, all the greats. This is definitely a conference to hit, and you can't beat Vienna for some good drinking with your history. Again, this is just the first of many sessions we're going to set up. If you're going to a conference you think we ought to be at please let us know. There are so many coming up we need to make sure we hit the right ones. Hope to see you there! -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Tue Oct 21 14:05:31 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Tue, 21 Oct 2008 14:05:31 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: References: Message-ID: <48FE19EB.7030501@jonkmans.com> So who are the major vendors we try to talk to? What are the OS's we'd need to hit? These things still running stuff like vxworks? Or do we figure out a dns lookup kind of thing? Anyone have contacts within the industry? Matt James McQuaid wrote: > Concur. Pushing reputational data to home users (with no interaction > required by the home user) offers opportunities. WOT and the paid > version of SiteAdvisor are attempting to do this in the browser, but > require some user interaction. > > There are hundreds of thousands of misconfigured, compromised and > ineffective home routers out there. If we could work with the > manufacturers, it might be possible to return these devices to > productive use. > > >> Or do we go with just pushing reputational data to the home user? What I >> mean is if we build this engine to generate and act upon IP reputation >> data could we know enough about the Internet collectively to simply push >> a blacklist to the home user's router/firewall? >> >> On the more sophisticated devices where software could be installed >> maybe it does run a stripped down detection engine and help feed IP data >> back to the group. But overall it's still primarily benefiting only from >> the blacklisting and whitelisting of the whole? >> >> How many false positives would we encounter that might actually affect a >> home user? > > We can take steps to ensure that this is not a big issue. > > > >> I think it'd be a very interesting day if we were to have essentially a >> Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even >> more IP mobile than they are now. >> >> Take atrivo/intercare/mccolo for example. Infested with crap, and have >> been for years. But since they can't really be blocked on the backbone >> home users still hit the same scam AV sites, give their credit card >> info, and get screwed. We know the sites are there, the registrars won't >> take them down, the ISP is colluding with the bad guys so they'll stay >> online. What can we do? (besides scream to our representatives for more >> effective laws) >> >> We can block those bad IPs at the home user's level. That'll make them >> start moving of course, just like bots being used to spam until they're >> listed. So we have to be able to immediately move quickly with the. > > Shut them out in real time with multiple daily updates; most of the > data would not change, so the diff would usually be a very small file. > > >> What does everyone think there? The basic idea being to use a normal >> engine model by most security pro's to feed IP reputation into a global >> database, and then the home user gets some sort of very basic tool or >> button they can click on to benefit from that data? Maybe even feed back >> to us. >> >> Matt >> > >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From aludwig at packetspy.com Tue Oct 21 14:09:25 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Tue, 21 Oct 2008 14:09:25 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> Message-ID: <48FE1AD5.9000804@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 In my humble honest opinion we should stay out of the client side game, there are plenty of products that serve those needs. If you want to dabble in that client side may I suggest research into behavior based detection/mitigation of malware. (its the hot new freshness) That being said there is tons of movement in that arena in place, and in all honesty the weakest link at this point in time IS NOT client side technology (the issue there is ADOPTION of what is currently available IMHO). What is absolutely horrible in its current state is IDS/IPS, as it simply can not keep up with application layer attacks. Granted this is easier to handle on the client side, but who exactly wants easy anyways? If you can create the right technology that scales properly and put it in the "cloud" you will be light years ahead of the rest of the industry as far as capability. As this group seems to be more of a "here is some grant money, come up with something interesting and creative", I dont think the value is really there to produce carbon copies of existing technology for the client side. Not to say that there wouldn't be value in a open source alternative to current commercial offerings for endpoint security, I just think this opportunity should not be spent chasing the known. We should look at this as an opportunity to prove a few crucial points. 1. This community can work together in an effective collaborative approach to produce something of value. 2. An open and transparent approach can produce meaningful results. (in reference to point 1) 3. We can in fact "build a next-generation intrusion detection and prevention engine" Lets try to focus on what works, what doesnt work, and what doesnt exist yet. (again just my two cents) An example of how horrible IDS/IPS is in its current state can be seen by using the technique used in eescan from Matt Richards. Pay particularly close attention to the "random chunks" of exploit feature. No IDS I know of can properly handle such a technique of obfuscation, and this illustrates the massive weakness of NIDS. www.eescan.net/downloads/DC15.pdf http://code.google.com/p/eescan/ Andre Ludwig -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP4a18jAfVnRK9hXAQjKoQf/V4XDUOYJgHyY9xn5s+3xbwcBSyY990/c 6j1XGSkS8oqyVNGHhSbCorHS9Gy0S2f4LYtMXQqftQ/P/aKkzmha9b807QpF3Pu8 yL9ofUnBUQR7x1moTJ2Ep04xYv/uRzIA8FrxY6anIIgwjRQbDoyksoaSODm8sNkS m7sKEgXu6anLRbZWLlQsDOW+n0/pSPWgfQfUm6G2L06oSZ7PChU1II5PdOeJA+QB HL3yJs0hbF9gIQz8OmtrSKlfCJjA8xrCWk3dHNEQ2TyD4qbasnePbNVIJ7XqFOp6 gXc6uQKVSABgjeSzGlH/LtruImh1ma5foOTw2+LVkJHALLDv4Kcdvg== =73dN -----END PGP SIGNATURE----- From david.glosser at gmail.com Tue Oct 21 14:42:57 2008 From: david.glosser at gmail.com (David Glosser) Date: Tue, 21 Oct 2008 14:42:57 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: <48FE19EB.7030501@jonkmans.com> References: <48FE19EB.7030501@jonkmans.com> Message-ID: >From this article (http://www.networkworld.com/net.worker/columnists/2004/0329wolf.html) back in 2004, granny's home router is: Linksys - 33% Netgear 12% D-Link 12% then there's Belkin, Buffalo, and everyone else. And dozens of models, with probably as many firmware versions out there. So, for Granny (home user), reputational data could be pushed (pulled?) at the following chokepoints: Browser -- OS -- home router -- ISP -----> Internet Some of these locations already have products (such as Browser-based WOT & siteadvisor, OS antivirus, etc, which work with varying degrees of effectiveness) and some don't... On Tue, Oct 21, 2008 at 2:05 PM, Matt Jonkman wrote: > So who are the major vendors we try to talk to? What are the OS's we'd > need to hit? > > These things still running stuff like vxworks? > > Or do we figure out a dns lookup kind of thing? > > Anyone have contacts within the industry? > > Matt > > James McQuaid wrote: >> Concur. Pushing reputational data to home users (with no interaction >> required by the home user) offers opportunities. WOT and the paid >> version of SiteAdvisor are attempting to do this in the browser, but >> require some user interaction. >> >> There are hundreds of thousands of misconfigured, compromised and >> ineffective home routers out there. If we could work with the >> manufacturers, it might be possible to return these devices to >> productive use. >> >> >>> Or do we go with just pushing reputational data to the home user? What I >>> mean is if we build this engine to generate and act upon IP reputation >>> data could we know enough about the Internet collectively to simply push >>> a blacklist to the home user's router/firewall? >>> >>> On the more sophisticated devices where software could be installed >>> maybe it does run a stripped down detection engine and help feed IP data >>> back to the group. But overall it's still primarily benefiting only from >>> the blacklisting and whitelisting of the whole? >>> >>> How many false positives would we encounter that might actually affect a >>> home user? >> >> We can take steps to ensure that this is not a big issue. >> >> >> >>> I think it'd be a very interesting day if we were to have essentially a >>> Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even >>> more IP mobile than they are now. >>> >>> Take atrivo/intercare/mccolo for example. Infested with crap, and have >>> been for years. But since they can't really be blocked on the backbone >>> home users still hit the same scam AV sites, give their credit card >>> info, and get screwed. We know the sites are there, the registrars won't >>> take them down, the ISP is colluding with the bad guys so they'll stay >>> online. What can we do? (besides scream to our representatives for more >>> effective laws) >>> >>> We can block those bad IPs at the home user's level. That'll make them >>> start moving of course, just like bots being used to spam until they're >>> listed. So we have to be able to immediately move quickly with the. >> >> Shut them out in real time with multiple daily updates; most of the >> data would not change, so the diff would usually be a very small file. >> >> >>> What does everyone think there? The basic idea being to use a normal >>> engine model by most security pro's to feed IP reputation into a global >>> database, and then the home user gets some sort of very basic tool or >>> button they can click on to benefit from that data? Maybe even feed back >>> to us. >>> >>> Matt >>> >> >>> -- >>> -------------------------------------------- >>> Matthew Jonkman >>> Emerging Threats >>> Phone 765-429-0398 >>> Fax 312-264-0205 >>> http://www.emergingthreats.net >>> -------------------------------------------- >>> >>> PGP: http://www.jonkmans.com/mattjonkman.asc >>> >>> >> > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From jeremy at sudosecure.net Tue Oct 21 14:51:45 2008 From: jeremy at sudosecure.net (Jeremy) Date: Tue, 21 Oct 2008 13:51:45 -0500 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: References: <48FE19EB.7030501@jonkmans.com> Message-ID: <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> Do we really have to build and worry about the vendors and work out their issues for them? What I could see happening is if this project truly builds the next intrusion detection engine with all of the intelligent data sharing capabilities as discussed on this list, that vendors will be stumbling over themselves to tap into this "cloud" of information and will work their own issues out. If you build it they will come, is my take on this. As long as you provide some sort of open API and/or control structure into your "cloud" then I think this project will have done their part. --jeremy On Tue, Oct 21, 2008 at 1:42 PM, David Glosser wrote: > >From this article > (http://www.networkworld.com/net.worker/columnists/2004/0329wolf.html) > back in 2004, > > granny's home router is: > Linksys - 33% > Netgear 12% > D-Link 12% > > then there's Belkin, Buffalo, and everyone else. And dozens of > models, with probably as many firmware versions out there. > > > So, for Granny (home user), reputational data could be pushed > (pulled?) at the following chokepoints: > > Browser -- OS -- home router -- ISP -----> Internet > > Some of these locations already have products (such as Browser-based > WOT & siteadvisor, OS antivirus, etc, which work with varying degrees > of effectiveness) and some don't... > > > > > > On Tue, Oct 21, 2008 at 2:05 PM, Matt Jonkman wrote: >> So who are the major vendors we try to talk to? What are the OS's we'd >> need to hit? >> >> These things still running stuff like vxworks? >> >> Or do we figure out a dns lookup kind of thing? >> >> Anyone have contacts within the industry? >> >> Matt >> >> James McQuaid wrote: >>> Concur. Pushing reputational data to home users (with no interaction >>> required by the home user) offers opportunities. WOT and the paid >>> version of SiteAdvisor are attempting to do this in the browser, but >>> require some user interaction. >>> >>> There are hundreds of thousands of misconfigured, compromised and >>> ineffective home routers out there. If we could work with the >>> manufacturers, it might be possible to return these devices to >>> productive use. >>> >>> >>>> Or do we go with just pushing reputational data to the home user? What I >>>> mean is if we build this engine to generate and act upon IP reputation >>>> data could we know enough about the Internet collectively to simply push >>>> a blacklist to the home user's router/firewall? >>>> >>>> On the more sophisticated devices where software could be installed >>>> maybe it does run a stripped down detection engine and help feed IP data >>>> back to the group. But overall it's still primarily benefiting only from >>>> the blacklisting and whitelisting of the whole? >>>> >>>> How many false positives would we encounter that might actually affect a >>>> home user? >>> >>> We can take steps to ensure that this is not a big issue. >>> >>> >>> >>>> I think it'd be a very interesting day if we were to have essentially a >>>> Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even >>>> more IP mobile than they are now. >>>> >>>> Take atrivo/intercare/mccolo for example. Infested with crap, and have >>>> been for years. But since they can't really be blocked on the backbone >>>> home users still hit the same scam AV sites, give their credit card >>>> info, and get screwed. We know the sites are there, the registrars won't >>>> take them down, the ISP is colluding with the bad guys so they'll stay >>>> online. What can we do? (besides scream to our representatives for more >>>> effective laws) >>>> >>>> We can block those bad IPs at the home user's level. That'll make them >>>> start moving of course, just like bots being used to spam until they're >>>> listed. So we have to be able to immediately move quickly with the. >>> >>> Shut them out in real time with multiple daily updates; most of the >>> data would not change, so the diff would usually be a very small file. >>> >>> >>>> What does everyone think there? The basic idea being to use a normal >>>> engine model by most security pro's to feed IP reputation into a global >>>> database, and then the home user gets some sort of very basic tool or >>>> button they can click on to benefit from that data? Maybe even feed back >>>> to us. >>>> >>>> Matt >>>> >>> >>>> -- >>>> -------------------------------------------- >>>> Matthew Jonkman >>>> Emerging Threats >>>> Phone 765-429-0398 >>>> Fax 312-264-0205 >>>> http://www.emergingthreats.net >>>> -------------------------------------------- >>>> >>>> PGP: http://www.jonkmans.com/mattjonkman.asc >>>> >>>> >>> >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From aludwig at packetspy.com Tue Oct 21 15:06:35 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Tue, 21 Oct 2008 15:06:35 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> References: <48FE19EB.7030501@jonkmans.com> <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> Message-ID: <48FE283B.90802@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I dont think we should worry about vendors, their stupidity, and their lack of competence. What we should worry about is focusing on producing technology that works, given the vast sum of experience and ideas in this group. We need less save the world, and more change the world. I would hate to see this group get sidetracked with "save the world" projects that produce little to nothing of an end result. This is our opportunity as a group to move an entire industry in a direction we deem "best", lets not squander it by attempting to coordinate with vendors on end user configuration issues or other worthwhile but less disruptive practices. (This is just my take, everyone is entitled to their own views of course. I am NOT trying to detract from any of the ideas that have been presented to date (or in the future). All of them have their own merit and worth of which I may or may not be qualified to judge. ) Andre Ludwig Jeremy wrote: > Do we really have to build and worry about the vendors and work out > their issues for them? What I could see happening is if this project > truly builds the next intrusion detection engine with all of the > intelligent data sharing capabilities as discussed on this list, that > vendors will be stumbling over themselves to tap into this "cloud" of > information and will work their own issues out. If you build it they > will come, is my take on this. As long as you provide some sort of > open API and/or control structure into your "cloud" then I think this > project will have done their part. > > --jeremy > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP4oPMjAfVnRK9hXAQhdVggAydfP08jv6tAlc3bYjxb1JHYXX+m5Ynj6 caj5a/YbCkm3g78gUWSyPb3pvpeM4qRQBG+lptSIVfuImvzOPOYDg3zXPm5QzFB/ I0imjCYx8644apF/29v1ls9i017KVtnVpyFz2mbf4wbr/wiUS0Ed0LW/PYPOmMrl eU46s+Tr32na1rxVkEO6eBUbZlDotFb+u83mrV3tTj0l/mCVuOFr/7/9eorYxQln S2+HSUiToO23T4wy6Y8vmSZ9XM/Ec/FC6CmLd24fi80WIcSyZD1R6spGQ785PO32 D6RkPJ3gFbbBwpgVYxGagIRzRPW05tfl1SraDNMETQfXGvmGxrs3Ww== =UPKn -----END PGP SIGNATURE----- From jonkman at jonkmans.com Tue Oct 21 15:14:48 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Tue, 21 Oct 2008 15:14:48 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: <48FE283B.90802@packetspy.com> References: <48FE19EB.7030501@jonkmans.com> <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> <48FE283B.90802@packetspy.com> Message-ID: <48FE2A28.4080204@jonkmans.com> Ya, agree here, and with Jeremy's comments that if we make the data available we can go from there. We will definitely get a good deal of ip reputation info from what it appears we're heading toward. Once we have that then we can consider other places to use it, as well as other places to source from. Matt Andre Ludwig wrote: > I dont think we should worry about vendors, their stupidity, and their > lack of competence. > > What we should worry about is focusing on producing technology that > works, given the vast sum of experience and ideas in this group. > > We need less save the world, and more change the world. I would hate to > see this group get sidetracked with "save the world" projects that > produce little to nothing of an end result. This is our opportunity as > a group to move an entire industry in a direction we deem "best", lets > not squander it by attempting to coordinate with vendors on end user > configuration issues or other worthwhile but less disruptive practices. > > (This is just my take, everyone is entitled to their own views of > course. I am NOT trying to detract from any of the ideas that have been > presented to date (or in the future). All of them have their own merit > and worth of which I may or may not be qualified to judge. ) > > Andre Ludwig > > Jeremy wrote: >> Do we really have to build and worry about the vendors and work out >> their issues for them? What I could see happening is if this project >> truly builds the next intrusion detection engine with all of the >> intelligent data sharing capabilities as discussed on this list, that >> vendors will be stumbling over themselves to tap into this "cloud" of >> information and will work their own issues out. If you build it they >> will come, is my take on this. As long as you provide some sort of >> open API and/or control structure into your "cloud" then I think this >> project will have done their part. > >> --jeremy > > > _______________________________________________ Discussion mailing list Discussion at openinfosecfoundation.org http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From david.glosser at gmail.com Tue Oct 21 16:23:26 2008 From: david.glosser at gmail.com (David Glosser) Date: Tue, 21 Oct 2008 16:23:26 -0400 Subject: [Discussion] Discussion Digest, Vol 1, Issue 14 In-Reply-To: <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> References: <48FE19EB.7030501@jonkmans.com> <1a0f92ae0810211151o2dd0d4aaw299bb3e8b99decda@mail.gmail.com> Message-ID: I think an open API is the way to go, and personally I can't see Granny wanting to pay an extra $10 or whatever. I don't think telling her "but this new router has a next intrusion detection engine with intelligent data sharing capabilities" will impress her. As it is, most users think "popup blocker" and protection are the same... "I already have a popup blocker, why do I need to buy more software?" On Tue, Oct 21, 2008 at 2:51 PM, Jeremy wrote: > Do we really have to build and worry about the vendors and work out > their issues for them? What I could see happening is if this project > truly builds the next intrusion detection engine with all of the > intelligent data sharing capabilities as discussed on this list, that > vendors will be stumbling over themselves to tap into this "cloud" of > information and will work their own issues out. If you build it they > will come, is my take on this. As long as you provide some sort of > open API and/or control structure into your "cloud" then I think this > project will have done their part. > > --jeremy > > On Tue, Oct 21, 2008 at 1:42 PM, David Glosser wrote: >> >From this article >> (http://www.networkworld.com/net.worker/columnists/2004/0329wolf.html) >> back in 2004, >> >> granny's home router is: >> Linksys - 33% >> Netgear 12% >> D-Link 12% >> >> then there's Belkin, Buffalo, and everyone else. And dozens of >> models, with probably as many firmware versions out there. >> >> >> So, for Granny (home user), reputational data could be pushed >> (pulled?) at the following chokepoints: >> >> Browser -- OS -- home router -- ISP -----> Internet >> >> Some of these locations already have products (such as Browser-based >> WOT & siteadvisor, OS antivirus, etc, which work with varying degrees >> of effectiveness) and some don't... >> >> >> >> >> >> On Tue, Oct 21, 2008 at 2:05 PM, Matt Jonkman wrote: >>> So who are the major vendors we try to talk to? What are the OS's we'd >>> need to hit? >>> >>> These things still running stuff like vxworks? >>> >>> Or do we figure out a dns lookup kind of thing? >>> >>> Anyone have contacts within the industry? >>> >>> Matt >>> >>> James McQuaid wrote: >>>> Concur. Pushing reputational data to home users (with no interaction >>>> required by the home user) offers opportunities. WOT and the paid >>>> version of SiteAdvisor are attempting to do this in the browser, but >>>> require some user interaction. >>>> >>>> There are hundreds of thousands of misconfigured, compromised and >>>> ineffective home routers out there. If we could work with the >>>> manufacturers, it might be possible to return these devices to >>>> productive use. >>>> >>>> >>>>> Or do we go with just pushing reputational data to the home user? What I >>>>> mean is if we build this engine to generate and act upon IP reputation >>>>> data could we know enough about the Internet collectively to simply push >>>>> a blacklist to the home user's router/firewall? >>>>> >>>>> On the more sophisticated devices where software could be installed >>>>> maybe it does run a stripped down detection engine and help feed IP data >>>>> back to the group. But overall it's still primarily benefiting only from >>>>> the blacklisting and whitelisting of the whole? >>>>> >>>>> How many false positives would we encounter that might actually affect a >>>>> home user? >>>> >>>> We can take steps to ensure that this is not a big issue. >>>> >>>> >>>> >>>>> I think it'd be a very interesting day if we were to have essentially a >>>>> Spamhaus/SURBL for IPs, thus pushing the bad guys to have to be even >>>>> more IP mobile than they are now. >>>>> >>>>> Take atrivo/intercare/mccolo for example. Infested with crap, and have >>>>> been for years. But since they can't really be blocked on the backbone >>>>> home users still hit the same scam AV sites, give their credit card >>>>> info, and get screwed. We know the sites are there, the registrars won't >>>>> take them down, the ISP is colluding with the bad guys so they'll stay >>>>> online. What can we do? (besides scream to our representatives for more >>>>> effective laws) >>>>> >>>>> We can block those bad IPs at the home user's level. That'll make them >>>>> start moving of course, just like bots being used to spam until they're >>>>> listed. So we have to be able to immediately move quickly with the. >>>> >>>> Shut them out in real time with multiple daily updates; most of the >>>> data would not change, so the diff would usually be a very small file. >>>> >>>> >>>>> What does everyone think there? The basic idea being to use a normal >>>>> engine model by most security pro's to feed IP reputation into a global >>>>> database, and then the home user gets some sort of very basic tool or >>>>> button they can click on to benefit from that data? Maybe even feed back >>>>> to us. >>>>> >>>>> Matt >>>>> >>>> >>>>> -- >>>>> -------------------------------------------- >>>>> Matthew Jonkman >>>>> Emerging Threats >>>>> Phone 765-429-0398 >>>>> Fax 312-264-0205 >>>>> http://www.emergingthreats.net >>>>> -------------------------------------------- >>>>> >>>>> PGP: http://www.jonkmans.com/mattjonkman.asc >>>>> >>>>> >>>> >>> >>> -- >>> -------------------------------------------- >>> Matthew Jonkman >>> Emerging Threats >>> Phone 765-429-0398 >>> Fax 312-264-0205 >>> http://www.emergingthreats.net >>> -------------------------------------------- >>> >>> PGP: http://www.jonkmans.com/mattjonkman.asc >>> >>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From robert.jamison at bt.com Tue Oct 21 18:44:17 2008 From: robert.jamison at bt.com (robert.jamison@bt.com) Date: Tue, 21 Oct 2008 23:44:17 +0100 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FB64C1.4000803@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> Message-ID: <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> It seems we're a split camp with: [Keynesian CAMP] Client Side Product/Service with ability to protect/detect compromise on grannyx home user *scope most thoroughly represented by Martin's " RFC: Proposal for Analysis Framework" [Supply Side CAMP] Focus on server side protection for net critical assets *Andre/Jack "What is absolutely horrible in its current state is IDS/IPS" / "Client side is simply not possible due to political and religious issues." Additional notes gathered (I've just caught up on my reading;-) (a) Consideration for re-write defanging capability as inline protection (b) Efficiency in stream storage--essentially normalize data inspection so it doesn't have to be redone by multiple engines (c) XML vs. Binary distribution of verbose alerts vs. instruction inferred datapoints (d) Consideration for extending existing project Bro Anything I'm missing? Rob -----Original Message----- From: discussion-bounces at openinfosecfoundation.org [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt Jonkman Sent: Sunday, October 19, 2008 12:48 PM To: Martin Holste Cc: discussion at openinfosecfoundation.org; rMslade at shaw.ca Subject: Re: [Discussion] What are we making? -- CLIENT Side Martin Holste wrote: > I'm guessing that the intent is to be the "detection" component of the > Trusted Internet Connection (TIC) > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monito r.html), > in which the mandate is to protect all federal assets, especially the > federal desktop environment. Certainly a sound approach to tackle such a problem. Glad I don't have to solve that one, it's huge among gov't agencies. Especially in the US, but surely similar everywhere. Even if DHS has different goals, I think > that this group could do a great service by providing the TIC's > protection geared toward the client-side, and I think for most sectors, > that's where the most imminent threats lie. Again, I'm certainly not > saying that defending servers is out of scope, just that IDS has > historically been about defending servers as the primary goal, with > clients secondary, and I think that needs to be reversed now. I agree. At emerging threats we've been lately heavy into post-infection signatures in our research. There are many vulnerabilities out there in client apps, but I'd venture to say that a very high percentage of the infections out there aren't the result of a remotely exploitable vulnerability. Most come from users clicking on email attachments, installing fake software/codecs, and visiting websites with hostile code. We've tried many times to write effective signatures to detect hostile html/java/gifs, etc. It's just not feasible as is. Code is too flexible for signature-based approaches. You can say the same thing a hundred ways, especially in html. So how can we go after client side? I really REALLY am not excited about trying to make a windows client. Not only does that open up a huge responsibility in support and the inevitable bluescreens, but I have had a difficult time over the years believing that any process on ANY os (especially windows) could be trusted and independant enough to watch itself. Take into account how easy it is for trojans and rootkits to shut down antivirus, or blind it. And these are products with hundreds of the most skilled coders around working on them. I know we're sharp as a community, but I don't think that's a battle we want to get in to. So how can we do it at the network layer? Sandboxing? Virtual emulators? Or do we continue the thinking that for any infection to be of any use to anyone it has to generate traffic? It has to call home, send spam, report stolen information, something. So we concentrate on detecting and stopping the post infection? What does everyone think about that? Matt > > --Martin > > On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > wrote: > > I doubt the intent of the DHS is to simply do good, they are most likely > much more focused on producing technology that allows them to > detect/mitigate/prevent attacks against critical components. This of > course means detecting attacks that fly below the threshold of detection > for todays technology. If it comes to "doing good" or detecting state > sponsored attacks against critical components (think custom attacks > against unknown vulns), i'm going to go out on a limb and say they would > rather protect the critical component vs the enduser. > > What you are discussing still has value and merit but im not so sure it > is what should be focused on, but of course I am not the person to > decide such things. > > Andre > > > Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > > Question: what are we making? Oh, yeah, I read the blurb: "The > OISF has been > > chartered and funded to build a next-generation intrusion > detection and prevention > > engine. This project will consider every new and existing > technology, concept and > > idea to build a completely open source licensed engine." > > > > OK, we're making an IDS. But I think we need to be more specific. > In particular, > > we need to answer the question of "who." > > > > Since the DHS has provided money, I suspect there would be an > automatic > > assumption that this is a heavy-duty device intended for use to > protect major > > servers and nodes in the critical information infrastructure. > (Whatever that > > means.) This kind of thing is built by professionals, for > professionals. Trained > > people. > > > > However, given the current computing environment, I think it would > be relatively > > easy to make a case that such a device is not going to do all that > much good. That > > a more accessible device, intended for "Grannyx" users, would > actually do more to > > protect the infrastructure. After all, it isn't major nodes on > the net that make up > > botnets, it's the little guys. Protect them, and you reduce the > threat. This is the > > "low hanging fruit" for the blackhats, so protecting that crop is > going to give us > > the greatest benefit for the commitment of resources. > > > > This makes a difference. Not, perhaps, in terms of questions > about multithreading > > analysis streams using graphics co-processors. But certainly in > most other areas. > > > > We've talked about extensibility. If we create a standard > template or format for > > signatures, the "who" makes a difference. Professionals need a > warning and a > > packet. Grannyx users need a warning, no packet, a clear > explanation of what and > > how important, and a recommended course of action. Makes a > difference to the > > template. > > > > In terms of my recommendation of a paran-o-meter, it makes a > difference. > > Actually, I see huge debates over initial settings: do we keep it > low to keep from > > crying wolf, or keep it high to keep people as safe as possible. > But one thing that > > should be done is make the paranoia settings not-quite-obvious up > front, so that > > somebody needs to know a little about the implications before they > start fiddling > > with settings. > > > > Heck, if it's a professional device, we don't need to worry about > the interface at > > all. If it's for Granny, we definitely do. > > > > It also makes a difference in terms of the technology to be > included. If it is for > > professionals, we can throw in everything. If for Granny, we need > to make a > > careful choice about maximum protection for minimum performance drain. > > > > ====================== (quote inserted randomly by Pegasus Mailer) > > rslade at vcn.bc.ca > slade at victoria.tc.ca > rslade at computercrime.org > > I'm getting so absent-minded that sometimes in the middle of > > victoria.tc.ca/techrev/rms.htm > > blogs.securiteam.com/index.php/archives/author/p1/ > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc _______________________________________________ Discussion mailing list Discussion at openinfosecfoundation.org http://lists.openinfosecfoundation.org/mailman/listinfo/discussion From jlewis at packetnexus.com Tue Oct 21 19:17:37 2008 From: jlewis at packetnexus.com (Jason Lewis) Date: Tue, 21 Oct 2008 19:17:37 -0400 Subject: [Discussion] Interesting project In-Reply-To: <48FBE3BC.4000305@packetnexus.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> Message-ID: <48FE6311.8070804@packetnexus.com> Argonne adopted IDMEF for their communication. https://www.anl.gov/it/Cyber_Security/Federations_for_Cyber_Defense/index.html From david.glosser at gmail.com Tue Oct 21 19:21:59 2008 From: david.glosser at gmail.com (David Glosser) Date: Tue, 21 Oct 2008 19:21:59 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> Message-ID: Where does granny's ISP fall into this and the Server-supply ISP? Is it more of a continuum between granny and the server and back? Basically, following the bouncing malware packet back and forth... Just brainstorming and attempting to identify ALL possible and potential places to inject this new project... On Tue, Oct 21, 2008 at 6:44 PM, wrote: > It seems we're a split camp with: > > [Keynesian CAMP] > Client Side Product/Service with ability to protect/detect compromise on > grannyx home user > *scope most thoroughly represented by Martin's " RFC: Proposal for > Analysis Framework" > > [Supply Side CAMP] > Focus on server side protection for net critical assets > *Andre/Jack "What is absolutely horrible in its current state is > IDS/IPS" / "Client side is simply not possible due to political and > religious issues." > > Additional notes gathered (I've just caught up on my reading;-) > > (a) Consideration for re-write defanging capability as inline protection > (b) Efficiency in stream storage--essentially normalize data inspection > so it doesn't have to be redone by multiple engines > (c) XML vs. Binary distribution of verbose alerts vs. instruction > inferred datapoints > (d) Consideration for extending existing project Bro > > Anything I'm missing? > > Rob > > -----Original Message----- > From: discussion-bounces at openinfosecfoundation.org > [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt > Jonkman > Sent: Sunday, October 19, 2008 12:48 PM > To: Martin Holste > Cc: discussion at openinfosecfoundation.org; rMslade at shaw.ca > Subject: Re: [Discussion] What are we making? -- CLIENT Side > > Martin Holste wrote: >> I'm guessing that the intent is to be the "detection" component of the >> Trusted Internet Connection (TIC) >> > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monito > r.html), >> in which the mandate is to protect all federal assets, especially the >> federal desktop environment. > > Certainly a sound approach to tackle such a problem. Glad I don't have > to solve that one, it's huge among gov't agencies. Especially in the US, > but surely similar everywhere. > > > Even if DHS has different goals, I think >> that this group could do a great service by providing the TIC's >> protection geared toward the client-side, and I think for most > sectors, >> that's where the most imminent threats lie. Again, I'm certainly not >> saying that defending servers is out of scope, just that IDS has >> historically been about defending servers as the primary goal, with >> clients secondary, and I think that needs to be reversed now. > > I agree. At emerging threats we've been lately heavy into post-infection > signatures in our research. There are many vulnerabilities out there in > client apps, but I'd venture to say that a very high percentage of the > infections out there aren't the result of a remotely exploitable > vulnerability. Most come from users clicking on email attachments, > installing fake software/codecs, and visiting websites with hostile > code. > > We've tried many times to write effective signatures to detect hostile > html/java/gifs, etc. It's just not feasible as is. Code is too flexible > for signature-based approaches. You can say the same thing a hundred > ways, especially in html. > > So how can we go after client side? > > I really REALLY am not excited about trying to make a windows client. > Not only does that open up a huge responsibility in support and the > inevitable bluescreens, but I have had a difficult time over the years > believing that any process on ANY os (especially windows) could be > trusted and independant enough to watch itself. Take into account how > easy it is for trojans and rootkits to shut down antivirus, or blind it. > And these are products with hundreds of the most skilled coders around > working on them. > > I know we're sharp as a community, but I don't think that's a battle we > want to get in to. So how can we do it at the network layer? > > Sandboxing? > > Virtual emulators? > > Or do we continue the thinking that for any infection to be of any use > to anyone it has to generate traffic? It has to call home, send spam, > report stolen information, something. So we concentrate on detecting and > stopping the post infection? > > What does everyone think about that? > > Matt > > >> >> --Martin >> >> On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig > > wrote: >> >> I doubt the intent of the DHS is to simply do good, they are most > likely >> much more focused on producing technology that allows them to >> detect/mitigate/prevent attacks against critical components. This > of >> course means detecting attacks that fly below the threshold of > detection >> for todays technology. If it comes to "doing good" or detecting > state >> sponsored attacks against critical components (think custom > attacks >> against unknown vulns), i'm going to go out on a limb and say they > would >> rather protect the critical component vs the enduser. >> >> What you are discussing still has value and merit but im not so > sure it >> is what should be focused on, but of course I am not the person to >> decide such things. >> >> Andre >> >> >> Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: >> > Question: what are we making? Oh, yeah, I read the blurb: "The >> OISF has been >> > chartered and funded to build a next-generation intrusion >> detection and prevention >> > engine. This project will consider every new and existing >> technology, concept and >> > idea to build a completely open source licensed engine." >> > >> > OK, we're making an IDS. But I think we need to be more > specific. >> In particular, >> > we need to answer the question of "who." >> > >> > Since the DHS has provided money, I suspect there would be an >> automatic >> > assumption that this is a heavy-duty device intended for use to >> protect major >> > servers and nodes in the critical information infrastructure. >> (Whatever that >> > means.) This kind of thing is built by professionals, for >> professionals. Trained >> > people. >> > >> > However, given the current computing environment, I think it > would >> be relatively >> > easy to make a case that such a device is not going to do all > that >> much good. That >> > a more accessible device, intended for "Grannyx" users, would >> actually do more to >> > protect the infrastructure. After all, it isn't major nodes on >> the net that make up >> > botnets, it's the little guys. Protect them, and you reduce the >> threat. This is the >> > "low hanging fruit" for the blackhats, so protecting that crop > is >> going to give us >> > the greatest benefit for the commitment of resources. >> > >> > This makes a difference. Not, perhaps, in terms of questions >> about multithreading >> > analysis streams using graphics co-processors. But certainly in >> most other areas. >> > >> > We've talked about extensibility. If we create a standard >> template or format for >> > signatures, the "who" makes a difference. Professionals need a >> warning and a >> > packet. Grannyx users need a warning, no packet, a clear >> explanation of what and >> > how important, and a recommended course of action. Makes a >> difference to the >> > template. >> > >> > In terms of my recommendation of a paran-o-meter, it makes a >> difference. >> > Actually, I see huge debates over initial settings: do we keep > it >> low to keep from >> > crying wolf, or keep it high to keep people as safe as possible. >> But one thing that >> > should be done is make the paranoia settings not-quite-obvious > up >> front, so that >> > somebody needs to know a little about the implications before > they >> start fiddling >> > with settings. >> > >> > Heck, if it's a professional device, we don't need to worry > about >> the interface at >> > all. If it's for Granny, we definitely do. >> > >> > It also makes a difference in terms of the technology to be >> included. If it is for >> > professionals, we can throw in everything. If for Granny, we > need >> to make a >> > careful choice about maximum protection for minimum performance > drain. >> > >> > ====================== (quote inserted randomly by Pegasus > Mailer) >> > rslade at vcn.bc.ca >> slade at victoria.tc.ca >> rslade at computercrime.org >> > I'm getting so absent-minded that sometimes in the > middle of >> > victoria.tc.ca/techrev/rms.htm >> >> blogs.securiteam.com/index.php/archives/author/p1/ >> >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> >> > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > >> > >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> >> > ------------------------------------------------------------------------ >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From aludwig at packetspy.com Tue Oct 21 19:25:39 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Tue, 21 Oct 2008 19:25:39 -0400 Subject: [Discussion] Interesting project In-Reply-To: <48FE6311.8070804@packetnexus.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> <48FE6311.8070804@packetnexus.com> Message-ID: <48FE64F3.7060207@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 My limited experience with IDMEF lead me to think of it as a 5000lb elephant attempting to tight rope walk across a large gorge on dental floss. (in case you were wondering I thought it was a large lumbering beast of a RFC that no one honestly cared about and felt it should be left for dead, which is apparently the case as there hasnt been much work done on it since 2005) Andre Ludwig Jason Lewis wrote: > Argonne adopted IDMEF for their communication. > > https://www.anl.gov/it/Cyber_Security/Federations_for_Cyber_Defense/index.html > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP5k9cjAfVnRK9hXAQhdCwf9F9Kf/6fxEVAmirCztQlbv/TGixSdi5i3 WHsLftImP395ggRO47eQx6thHt/PYx0s2w0CzFrY7KxBeLjNmXlIBnMlVO8jkV32 bBzdS5isFCRx+P7fPU5g8Z3MsCG/2qdVVA/RqXoRgRPZsCi3i7nELcv4qp3JU4D5 kXLni4GAJMpEtsx21HqI7l+4l4Y6XERm8bvEdrJPoUmlq63LUWk4BrriQbETandg JURk/1imOsVRsdloeNSYI3kZinDF036aluwgDnMx8J7uaPMYG01Yn1c4Jv8xMV9+ r18m+prz944uj+Cr+Q4JN4P6STtL/yTlt5ePpNqS7igw6EspcxVRRA== =HJF1 -----END PGP SIGNATURE----- From jlewis at packetnexus.com Tue Oct 21 21:44:14 2008 From: jlewis at packetnexus.com (Jason Lewis) Date: Tue, 21 Oct 2008 21:44:14 -0400 Subject: [Discussion] Interesting project In-Reply-To: <48FE64F3.7060207@packetspy.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <1224466571.20986.14.camel@localhost> <1a0f92ae0810191846n39555dc3xca7c563463443b80@mail.gmail.com> <48FBE3BC.4000305@packetnexus.com> <48FE6311.8070804@packetnexus.com> <48FE64F3.7060207@packetspy.com> Message-ID: <48FE856E.5010404@packetnexus.com> I agree that some of these xml standards are bloated and probably overkill. My experience has been that all of these network devices use their own format for data sharing and you end up having to write parsers for each one. The best solution is probably an open engine that would allow someone to write input and output services. My only concern is yet another IDS (or whatever) that ignores the benefit of a standard that anyone could use. For example, CVE has made vulnerability reporting easier to process. jas Andre Ludwig wrote: > My limited experience with IDMEF lead me to think of it as a 5000lb > elephant attempting to tight rope walk across a large gorge on dental > floss. > (in case you were wondering I thought it was a large lumbering beast of > a RFC that no one honestly cared about and felt it should be left for > dead, which is apparently the case as there hasnt been much work done on > it since 2005) > > Andre Ludwig > > Jason Lewis wrote: > > Argonne adopted IDMEF for their communication. > > > > https://www.anl.gov/it/Cyber_Security/Federations_for_Cyber_Defense/index.html > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > From mcholste at gmail.com Tue Oct 21 23:23:34 2008 From: mcholste at gmail.com (Martin Holste) Date: Tue, 21 Oct 2008 22:23:34 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> Message-ID: I think the middle part of the continuum that I see the most value in is honing IDS into something which is used to build solid blacklists based on client compromises which can be used as an API by other technologies. So, I would definitely say that I'm in the server-side camp in that I don't want anything to do with client-side code other than providing a list that somebody else's client-side code could use. I hope my "Proposal" didn't insinuate otherwise. I meant it as a place to start discussing specific technology we would like to use, but it was apparently a bit premature. --Martin On Tue, Oct 21, 2008 at 6:21 PM, David Glosser wrote: > Where does granny's ISP fall into this and the Server-supply ISP? > Is it more of a continuum between granny and the server and back? > > Basically, following the bouncing malware packet back and forth... > Just brainstorming and attempting to identify ALL possible and > potential places to inject this new project... > > > > On Tue, Oct 21, 2008 at 6:44 PM, wrote: > > It seems we're a split camp with: > > > > [Keynesian CAMP] > > Client Side Product/Service with ability to protect/detect compromise on > > grannyx home user > > *scope most thoroughly represented by Martin's " RFC: Proposal for > > Analysis Framework" > > > > [Supply Side CAMP] > > Focus on server side protection for net critical assets > > *Andre/Jack "What is absolutely horrible in its current state is > > IDS/IPS" / "Client side is simply not possible due to political and > > religious issues." > > > > Additional notes gathered (I've just caught up on my reading;-) > > > > (a) Consideration for re-write defanging capability as inline protection > > (b) Efficiency in stream storage--essentially normalize data inspection > > so it doesn't have to be redone by multiple engines > > (c) XML vs. Binary distribution of verbose alerts vs. instruction > > inferred datapoints > > (d) Consideration for extending existing project Bro > > > > Anything I'm missing? > > > > Rob > > > > -----Original Message----- > > From: discussion-bounces at openinfosecfoundation.org > > [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt > > Jonkman > > Sent: Sunday, October 19, 2008 12:48 PM > > To: Martin Holste > > Cc: discussion at openinfosecfoundation.org; rMslade at shaw.ca > > Subject: Re: [Discussion] What are we making? -- CLIENT Side > > > > Martin Holste wrote: > >> I'm guessing that the intent is to be the "detection" component of the > >> Trusted Internet Connection (TIC) > >> > > (http://taosecurity.blogspot.com/2007/12/feds-plan-to-reduce-then-monito > > r.html), > >> in which the mandate is to protect all federal assets, especially the > >> federal desktop environment. > > > > Certainly a sound approach to tackle such a problem. Glad I don't have > > to solve that one, it's huge among gov't agencies. Especially in the US, > > but surely similar everywhere. > > > > > > Even if DHS has different goals, I think > >> that this group could do a great service by providing the TIC's > >> protection geared toward the client-side, and I think for most > > sectors, > >> that's where the most imminent threats lie. Again, I'm certainly not > >> saying that defending servers is out of scope, just that IDS has > >> historically been about defending servers as the primary goal, with > >> clients secondary, and I think that needs to be reversed now. > > > > I agree. At emerging threats we've been lately heavy into post-infection > > signatures in our research. There are many vulnerabilities out there in > > client apps, but I'd venture to say that a very high percentage of the > > infections out there aren't the result of a remotely exploitable > > vulnerability. Most come from users clicking on email attachments, > > installing fake software/codecs, and visiting websites with hostile > > code. > > > > We've tried many times to write effective signatures to detect hostile > > html/java/gifs, etc. It's just not feasible as is. Code is too flexible > > for signature-based approaches. You can say the same thing a hundred > > ways, especially in html. > > > > So how can we go after client side? > > > > I really REALLY am not excited about trying to make a windows client. > > Not only does that open up a huge responsibility in support and the > > inevitable bluescreens, but I have had a difficult time over the years > > believing that any process on ANY os (especially windows) could be > > trusted and independant enough to watch itself. Take into account how > > easy it is for trojans and rootkits to shut down antivirus, or blind it. > > And these are products with hundreds of the most skilled coders around > > working on them. > > > > I know we're sharp as a community, but I don't think that's a battle we > > want to get in to. So how can we do it at the network layer? > > > > Sandboxing? > > > > Virtual emulators? > > > > Or do we continue the thinking that for any infection to be of any use > > to anyone it has to generate traffic? It has to call home, send spam, > > report stolen information, something. So we concentrate on detecting and > > stopping the post infection? > > > > What does everyone think about that? > > > > Matt > > > > > >> > >> --Martin > >> > >> On Sat, Oct 18, 2008 at 9:31 PM, Andre Ludwig >> > wrote: > >> > >> I doubt the intent of the DHS is to simply do good, they are most > > likely > >> much more focused on producing technology that allows them to > >> detect/mitigate/prevent attacks against critical components. This > > of > >> course means detecting attacks that fly below the threshold of > > detection > >> for todays technology. If it comes to "doing good" or detecting > > state > >> sponsored attacks against critical components (think custom > > attacks > >> against unknown vulns), i'm going to go out on a limb and say they > > would > >> rather protect the critical component vs the enduser. > >> > >> What you are discussing still has value and merit but im not so > > sure it > >> is what should be focused on, but of course I am not the person to > >> decide such things. > >> > >> Andre > >> > >> > >> Rob, grandpa of Ryan, Trevor, Devon & Hannah wrote: > >> > Question: what are we making? Oh, yeah, I read the blurb: "The > >> OISF has been > >> > chartered and funded to build a next-generation intrusion > >> detection and prevention > >> > engine. This project will consider every new and existing > >> technology, concept and > >> > idea to build a completely open source licensed engine." > >> > > >> > OK, we're making an IDS. But I think we need to be more > > specific. > >> In particular, > >> > we need to answer the question of "who." > >> > > >> > Since the DHS has provided money, I suspect there would be an > >> automatic > >> > assumption that this is a heavy-duty device intended for use to > >> protect major > >> > servers and nodes in the critical information infrastructure. > >> (Whatever that > >> > means.) This kind of thing is built by professionals, for > >> professionals. Trained > >> > people. > >> > > >> > However, given the current computing environment, I think it > > would > >> be relatively > >> > easy to make a case that such a device is not going to do all > > that > >> much good. That > >> > a more accessible device, intended for "Grannyx" users, would > >> actually do more to > >> > protect the infrastructure. After all, it isn't major nodes on > >> the net that make up > >> > botnets, it's the little guys. Protect them, and you reduce the > >> threat. This is the > >> > "low hanging fruit" for the blackhats, so protecting that crop > > is > >> going to give us > >> > the greatest benefit for the commitment of resources. > >> > > >> > This makes a difference. Not, perhaps, in terms of questions > >> about multithreading > >> > analysis streams using graphics co-processors. But certainly in > >> most other areas. > >> > > >> > We've talked about extensibility. If we create a standard > >> template or format for > >> > signatures, the "who" makes a difference. Professionals need a > >> warning and a > >> > packet. Grannyx users need a warning, no packet, a clear > >> explanation of what and > >> > how important, and a recommended course of action. Makes a > >> difference to the > >> > template. > >> > > >> > In terms of my recommendation of a paran-o-meter, it makes a > >> difference. > >> > Actually, I see huge debates over initial settings: do we keep > > it > >> low to keep from > >> > crying wolf, or keep it high to keep people as safe as possible. > >> But one thing that > >> > should be done is make the paranoia settings not-quite-obvious > > up > >> front, so that > >> > somebody needs to know a little about the implications before > > they > >> start fiddling > >> > with settings. > >> > > >> > Heck, if it's a professional device, we don't need to worry > > about > >> the interface at > >> > all. If it's for Granny, we definitely do. > >> > > >> > It also makes a difference in terms of the technology to be > >> included. If it is for > >> > professionals, we can throw in everything. If for Granny, we > > need > >> to make a > >> > careful choice about maximum protection for minimum performance > > drain. > >> > > >> > ====================== (quote inserted randomly by Pegasus > > Mailer) > >> > rslade at vcn.bc.ca > >> slade at victoria.tc.ca > >> rslade at computercrime.org > >> > I'm getting so absent-minded that sometimes in the > > middle of > >> > victoria.tc.ca/techrev/rms.htm > >> > >> blogs.securiteam.com/index.php/archives/author/p1/ > >> > >> > _______________________________________________ > >> > Discussion mailing list > >> > Discussion at openinfosecfoundation.org > >> > >> > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > > >> > > >> > >> _______________________________________________ > >> Discussion mailing list > >> Discussion at openinfosecfoundation.org > >> > >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > >> > >> > >> > > ------------------------------------------------------------------------ > >> > >> _______________________________________________ > >> Discussion mailing list > >> Discussion at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > -- > > -------------------------------------------- > > Matthew Jonkman > > Emerging Threats > > Phone 765-429-0398 > > Fax 312-264-0205 > > http://www.emergingthreats.net > > -------------------------------------------- > > > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081021/95cfffa9/attachment-0001.html From david.glosser at gmail.com Wed Oct 22 07:37:18 2008 From: david.glosser at gmail.com (David Glosser) Date: Wed, 22 Oct 2008 07:37:18 -0400 Subject: [Discussion] new thread: biggest threats Message-ID: What are the biggest threats out there (and tomorrow?) today that this new project may be of benefit? I'm voting for: asprox/sql injection - website owners having their sites infected, which means, for granny, it's no longer possible just to tell granny to only go to safe sites... And When adobe's site is infected (1) , it's a corporate issue as well fake security sites - so many domains, fast flux, double-fast flux, etc. very low initial detection, sigs are always playing catchup future - continuing infection of web sites running unpatched software, dns or bgp-related attacks/exploits As this is brainstorming, if you don't think it's a good thread, don't criticize, just don't respond ;) (1)http://blogs.zdnet.com/security/?p=2039 From mcholste at gmail.com Wed Oct 22 09:37:44 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Oct 2008 08:37:44 -0500 Subject: [Discussion] new thread: biggest threats In-Reply-To: References: Message-ID: I would agree that for the server arena, SQL injection is probably the biggest current threat for most as far as potential damage to their organization. For client side, I think that malicious Javascript has got to be near the top. I was picking apart an attack last week in which the attackers had gotten an ad banner on a major ad syndicate which was iframing to a particularly nasty bit of Javascript. This script created two Java classes by binary packing the entire object as a Javascript string, then referring to that object in the same Javascript. The next thing the client did was to make a malware download with "Java 1.5" in the user agent. While browser plugin and client-side app vulnerabilities rotate, the attack vectors and payload delivery framework usually rely on Javascript. Brainstorm: Create an IP/domain blacklist that the NoScript guys can have their plugin point at? --Martin On Wed, Oct 22, 2008 at 6:37 AM, David Glosser wrote: > What are the biggest threats out there (and tomorrow?) today that > this new project may be of benefit? > > I'm voting for: > asprox/sql injection - website owners having their sites infected, > which means, for granny, it's no longer possible just to tell granny > to only go to safe sites... And When adobe's site is infected (1) , > it's a corporate issue as well > fake security sites - so many domains, fast flux, double-fast flux, > etc. very low initial detection, sigs are always playing catchup > future - continuing infection of web sites running unpatched software, > dns or bgp-related attacks/exploits > > As this is brainstorming, if you don't think it's a good thread, > don't criticize, just don't respond ;) > > (1)http://blogs.zdnet.com/security/?p=2039 > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/4ba943b7/attachment.html From aludwig at packetspy.com Wed Oct 22 09:59:21 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Wed, 22 Oct 2008 09:59:21 -0400 Subject: [Discussion] new thread: biggest threats In-Reply-To: References: Message-ID: <48FF31B9.2070602@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 JS is a means, not an end. Andre Martin Holste wrote: > I would agree that for the server arena, SQL injection is probably the > biggest current threat for most as far as potential damage to their > organization. > > For client side, I think that malicious Javascript has got to be near > the top. I was picking apart an attack last week in which the > attackers had gotten an ad banner on a major ad syndicate which was > iframing to a particularly nasty bit of Javascript. This script > created two Java classes by binary packing the entire object as a > Javascript string, then referring to that object in the same > Javascript. The next thing the client did was to make a malware > download with "Java 1.5" in the user agent. While browser plugin and > client-side app vulnerabilities rotate, the attack vectors and payload > delivery framework usually rely on Javascript. > > Brainstorm: Create an IP/domain blacklist that the NoScript guys can > have their plugin point at? > > --Martin > > On Wed, Oct 22, 2008 at 6:37 AM, David Glosser > > wrote: > > What are the biggest threats out there (and tomorrow?) today that > this new project may be of benefit? > > I'm voting for: > asprox/sql injection - website owners having their sites infected, > which means, for granny, it's no longer possible just to tell granny > to only go to safe sites... And When adobe's site is infected (1) , > it's a corporate issue as well > fake security sites - so many domains, fast flux, double-fast flux, > etc. very low initial detection, sigs are always playing catchup > future - continuing infection of web sites running unpatched software, > dns or bgp-related attacks/exploits > > As this is brainstorming, if you don't think it's a good thread, > don't criticize, just don't respond ;) > > (1)http://blogs.zdnet.com/security/?p=2039 > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP8xusjAfVnRK9hXAQjwswf/aN0WNBJYYAgrKv9q2gHSpKT/N4ittxIY 2/iImQHxftwNfgic1YY6GWKIe1mNz66JjPSAqVQqAo0Yf0D5gE3jNHuVPMG4AxGw mGtvvjQFFTXNiY3QTuaRiWFAGnTaGTI50hApqOLs5kmvVRodSGqlNgdc96RqLF3R lEbU8AUcMQXn4TWQWK8hSkDNYOdcXhqg9FlXb2U0xwadrsSbS1zjcJ6rdbtsQLPk V1vgw/f3Eu2ZNeWGu4Q5ZkIHjL+iHj8+kHFfT92fbWjhsaklkdKfT9owZZTGVl/Z etBMNvt18gi6IosqVWWDdniFRw/byjsBqYiUFnqejkzJkylQy/vn2A== =bJtL -----END PGP SIGNATURE----- From mcholste at gmail.com Wed Oct 22 10:11:01 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Oct 2008 09:11:01 -0500 Subject: [Discussion] new thread: biggest threats In-Reply-To: <48FF31B9.2070602@packetspy.com> References: <48FF31B9.2070602@packetspy.com> Message-ID: Right, just like a network is a means, not an end. You inspect the network because you know the threats have to traverse it, and I would argue that similarly, there is value in inspecting Javascript because like the network, it is ubiquitously involved in malicious activity. I'm suggesting a JIDS as a plugin to a NIDS. On Wed, Oct 22, 2008 at 8:59 AM, Andre Ludwig wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > JS is a means, not an end. > > Andre > > Martin Holste wrote: > > I would agree that for the server arena, SQL injection is probably the > > biggest current threat for most as far as potential damage to their > > organization. > > > > For client side, I think that malicious Javascript has got to be near > > the top. I was picking apart an attack last week in which the > > attackers had gotten an ad banner on a major ad syndicate which was > > iframing to a particularly nasty bit of Javascript. This script > > created two Java classes by binary packing the entire object as a > > Javascript string, then referring to that object in the same > > Javascript. The next thing the client did was to make a malware > > download with "Java 1.5" in the user agent. While browser plugin and > > client-side app vulnerabilities rotate, the attack vectors and payload > > delivery framework usually rely on Javascript. > > > > Brainstorm: Create an IP/domain blacklist that the NoScript guys can > > have their plugin point at? > > > > --Martin > > > > On Wed, Oct 22, 2008 at 6:37 AM, David Glosser > > > wrote: > > > > What are the biggest threats out there (and tomorrow?) today that > > this new project may be of benefit? > > > > I'm voting for: > > asprox/sql injection - website owners having their sites infected, > > which means, for granny, it's no longer possible just to tell granny > > to only go to safe sites... And When adobe's site is infected (1) , > > it's a corporate issue as well > > fake security sites - so many domains, fast flux, double-fast flux, > > etc. very low initial detection, sigs are always playing catchup > > future - continuing infection of web sites running unpatched > software, > > dns or bgp-related attacks/exploits > > > > As this is brainstorming, if you don't think it's a good thread, > > don't criticize, just don't respond ;) > > > > (1)http://blogs.zdnet.com/security/?p=2039 > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > -----BEGIN PGP SIGNATURE----- > Version: PGP Desktop 9.9.0 (Build 397) > Charset: ISO-8859-1 > > wsBVAwUBSP8xusjAfVnRK9hXAQjwswf/aN0WNBJYYAgrKv9q2gHSpKT/N4ittxIY > 2/iImQHxftwNfgic1YY6GWKIe1mNz66JjPSAqVQqAo0Yf0D5gE3jNHuVPMG4AxGw > mGtvvjQFFTXNiY3QTuaRiWFAGnTaGTI50hApqOLs5kmvVRodSGqlNgdc96RqLF3R > lEbU8AUcMQXn4TWQWK8hSkDNYOdcXhqg9FlXb2U0xwadrsSbS1zjcJ6rdbtsQLPk > V1vgw/f3Eu2ZNeWGu4Q5ZkIHjL+iHj8+kHFfT92fbWjhsaklkdKfT9owZZTGVl/Z > etBMNvt18gi6IosqVWWDdniFRw/byjsBqYiUFnqejkzJkylQy/vn2A== > =bJtL > -----END PGP SIGNATURE----- > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/716253d3/attachment.html From aludwig at packetspy.com Wed Oct 22 10:19:20 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Wed, 22 Oct 2008 10:19:20 -0400 Subject: [Discussion] new thread: biggest threats In-Reply-To: References: <48FF31B9.2070602@packetspy.com> Message-ID: <48FF3668.5070601@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Correct, but the problem with JS is that it is an extremely expressive language. One that in conjunction with the DOM provides a nearly infinite number of ways to shovel an exploit to vulnerable browser, kernel, or third party code. That is the fundamental problem with IDS when attempting to detect/block these sorts of exploits at this layer. While I agree that your suggestion would be helpful it IMHO has no place on the network for sheer performance reasons alone (can you imagine trying to emulate a full DOM/JS engine at wire speed). Browsers can barely keep up with the onslaught of horribly written JS code, much less purposefully obfuscated or obtuse JS that hides an exploit. This is a topic that I have dedicated some thought to over the last three years so I would love to hear everyones ideas on the subject. Andre Ludwig Martin Holste wrote: > Right, just like a network is a means, not an end. You inspect the network because you know the threats have to traverse it, and I would argue that similarly, there is value in inspecting Javascript because like the network, it is ubiquitously involved in malicious activity. I'm suggesting a JIDS as a plugin to a NIDS. > On Wed, Oct 22, 2008 at 8:59 AM, Andre Ludwig < aludwig at packetspy.com> wrote: > > * PGP Bad Signature, Signed by an unverified key: 10/22/08 at 09:59:22 > > JS is a means, not an end. > > Andre > > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP82asjAfVnRK9hXAQgJmwf9Gl+a9xhnH/SOwphSa/3wNqWMhQ+Od+3S crAMZpO9XIiKuo4UdC+L13T2wQ38H5flnrq0bwQcQ8//AcZVeU6tSiy1xWR5chOd 36I9T675FrzdQtUWBp+lKsLiotubAcjgWvWVyw65tB27gUaIojzISMqm6XHVlfW3 2IDZZoo/mzPSkOnX44tO70i0kCKY5hBaJ9JlQfAW4giW7Q1Y+MTHUYm25npYrD8i mWgsEbTGUlBwM6JgNnDTFgRM6riOHw1GOdcZjm6/THSTjYz5IaItUyMD6SwEzOgE igxaOzT2V77zJKs4b9P8Xj1jCTZztkl45hbSFaOdYhB3rVTk0us73Q== =UtkX -----END PGP SIGNATURE----- From mcholste at gmail.com Wed Oct 22 11:13:25 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Oct 2008 10:13:25 -0500 Subject: [Discussion] new thread: biggest threats In-Reply-To: <48FF3668.5070601@packetspy.com> References: <48FF31B9.2070602@packetspy.com> <48FF3668.5070601@packetspy.com> Message-ID: All good points, and conclusions I had arrived at as well. Then I saw the Bro XML validator plugin which showed that it is possible to do extremely intensive, app layer inspection if you can do it in a clustered, asynchronous way. The other thought I had was that you wouldn't have to have the horsepower to actually render to memory what the browser would see. You could get 80-90% of the value through JS profiling, as in, how many eval calls does the code make? What is the entropy like? If we could "score" JS, it would really open the door to a lot of possibilities, but we wouldn't necessarily have to create the entire DOM on the fly, just get a feel for the style. Since Mozilla already provides the spidermonkey command line JS engine/API, the heavy lifting is already done. Here's an example of how I see this working: A sig detects that a user agent on a host changes and makes a note of it in a global array. (The UA changing is obviously not malicious in its own right). Another sig tracks which HTTP referrers were sent via iframe and keeps a short (by time) list of those in a global array. A third sig tries to decode JS a bit to see how many evals are in the code. A final sig looks for any executable downloads. If all four hit on the same client in quick succession, you've got "probable cause." Sounds like a lot of work, but I don't think it's as bad as the first impression makes it seem because instead of writing thousands of signatures for very specific occurrences like we do currently, you write a few hundred sigs for generic events, and then a few more hundred sigs for the matching up the combination of generic events which equals "badness." This is where Bro would make a lot of this easy. For instance, Bro allows for truly global lists to be shared between nodes, as in, the list of clients which have recently switched their user agent. So every node is aware of this stuff. Have you guys seen how cheap quad cores have become? Run 4 processes on the same box and you get some easy load balancing. Then you have enough horsepower to do this app layer inspection without packet drops. I suspect you could even implement load shedding if they get in trouble. I admit that this all sounds rather complicated, but there don't seem to be any simple exploits out there anymore in today's commoditized malware. I don't think we can continue to depend on single-event signatures. --Martin On Wed, Oct 22, 2008 at 9:19 AM, Andre Ludwig wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Correct, but the problem with JS is that it is an extremely expressive > language. One that in conjunction with the DOM provides a nearly > infinite number of ways to shovel an exploit to vulnerable browser, > kernel, or third party code. That is the fundamental problem with IDS > when attempting to detect/block these sorts of exploits at this layer. > While I agree that your suggestion would be helpful it IMHO has no place > on the network for sheer performance reasons alone (can you imagine > trying to emulate a full DOM/JS engine at wire speed). Browsers can > barely keep up with the onslaught of horribly written JS code, much less > purposefully obfuscated or obtuse JS that hides an exploit. This is a > topic that I have dedicated some thought to over the last three years so > I would love to hear everyones ideas on the subject. > > Andre Ludwig > > Martin Holste wrote: > > Right, just like a network is a means, not an end. You inspect the > network because you know the threats have to traverse it, and I would argue > that similarly, there is value in inspecting Javascript because like the > network, it is ubiquitously involved in malicious activity. I'm suggesting > a JIDS as a plugin to a NIDS. > > On Wed, Oct 22, 2008 at 8:59 AM, Andre Ludwig < aludwig at packetspy.com> > wrote: > > > > * PGP Bad Signature, Signed by an unverified key: 10/22/08 at 09:59:22 > > > > JS is a means, not an end. > > > > Andre > > > > > > > -----BEGIN PGP SIGNATURE----- > Version: PGP Desktop 9.9.0 (Build 397) > Charset: ISO-8859-1 > > wsBVAwUBSP82asjAfVnRK9hXAQgJmwf9Gl+a9xhnH/SOwphSa/3wNqWMhQ+Od+3S > crAMZpO9XIiKuo4UdC+L13T2wQ38H5flnrq0bwQcQ8//AcZVeU6tSiy1xWR5chOd > 36I9T675FrzdQtUWBp+lKsLiotubAcjgWvWVyw65tB27gUaIojzISMqm6XHVlfW3 > 2IDZZoo/mzPSkOnX44tO70i0kCKY5hBaJ9JlQfAW4giW7Q1Y+MTHUYm25npYrD8i > mWgsEbTGUlBwM6JgNnDTFgRM6riOHw1GOdcZjm6/THSTjYz5IaItUyMD6SwEzOgE > igxaOzT2V77zJKs4b9P8Xj1jCTZztkl45hbSFaOdYhB3rVTk0us73Q== > =UtkX > -----END PGP SIGNATURE----- > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/7fbcf357/attachment.html From aludwig at packetspy.com Wed Oct 22 11:31:37 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Wed, 22 Oct 2008 11:31:37 -0400 Subject: [Discussion] new thread: biggest threats In-Reply-To: References: <48FF31B9.2070602@packetspy.com> <48FF3668.5070601@packetspy.com> Message-ID: <48FF4759.4070007@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Profiling works, but I doubt it can get you 80-90% accuracy no matter how well the system works. The problem again is that JS is an extremely expressive language, and who is to say that you are even going to use JS to actually interact with the dom? Why not use ActionScript, VBS, or next weeks latest and greatest RIA scripting language? I agree on the distributed approach 100%, it is the only real way to scale and easily provide a modular approach to features and capabilities. Andre Martin Holste wrote: > All good points, and conclusions I had arrived at as well. Then I saw > the Bro XML validator plugin > which showed that it > is possible to do extremely intensive, app layer inspection if you can > do it in a clustered, asynchronous way. The other thought I had was > that you wouldn't have to have the horsepower to actually render to > memory what the browser would see. You could get 80-90% of the value > through JS profiling, as in, how many eval calls does the code make? > What is the entropy like? If we could "score" JS, it would really > open the door to a lot of possibilities, but we wouldn't necessarily > have to create the entire DOM on the fly, just get a feel for the > style. Since Mozilla already provides the spidermonkey command line > JS engine/API, the heavy lifting is already done. > > Here's an example of how I see this working: A sig detects that a > user agent on a host changes and makes a note of it in a global > array. (The UA changing is obviously not malicious in its own > right). Another sig tracks which HTTP referrers were sent via iframe > and keeps a short (by time) list of those in a global array. A third > sig tries to decode JS a bit to see how many evals are in the code. A > final sig looks for any executable downloads. If all four hit on the > same client in quick succession, you've got "probable cause." > > Sounds like a lot of work, but I don't think it's as bad as the first > impression makes it seem because instead of writing thousands of > signatures for very specific occurrences like we do currently, you > write a few hundred sigs for generic events, and then a few more > hundred sigs for the matching up the combination of generic events > which equals "badness." > > This is where Bro would make a lot of this easy. For instance, Bro > allows for truly global lists to be shared between nodes, as in, the > list of clients which have recently switched their user agent. So > every node is aware of this stuff. Have you guys seen how cheap quad > cores have become? Run 4 processes on the same box and you get some > easy load balancing. Then you have enough horsepower to do this app > layer inspection without packet drops. I suspect you could even > implement load shedding if they get in trouble. > > I admit that this all sounds rather complicated, but there don't seem > to be any simple exploits out there anymore in today's commoditized > malware. I don't think we can continue to depend on single-event > signatures. > > --Martin > > On Wed, Oct 22, 2008 at 9:19 AM, Andre Ludwig > wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Correct, but the problem with JS is that it is an extremely expressive > language. One that in conjunction with the DOM provides a nearly > infinite number of ways to shovel an exploit to vulnerable browser, > kernel, or third party code. That is the fundamental problem with IDS > when attempting to detect/block these sorts of exploits at this layer. > While I agree that your suggestion would be helpful it IMHO has no > place > on the network for sheer performance reasons alone (can you imagine > trying to emulate a full DOM/JS engine at wire speed). Browsers can > barely keep up with the onslaught of horribly written JS code, > much less > purposefully obfuscated or obtuse JS that hides an exploit. This > is a > topic that I have dedicated some thought to over the last three > years so > I would love to hear everyones ideas on the subject. > > Andre Ludwig > > Martin Holste wrote: > > Right, just like a network is a means, not an end. You inspect > the network because you know the threats have to traverse it, and > I would argue that similarly, there is value in inspecting > Javascript because like the network, it is ubiquitously involved > in malicious activity. I'm suggesting a JIDS as a plugin to a NIDS. > > On Wed, Oct 22, 2008 at 8:59 AM, Andre Ludwig < > aludwig at packetspy.com > wrote: > > > > * PGP Bad Signature, Signed by an unverified key: 10/22/08 at > 09:59:22 > > > > JS is a means, not an end. > > > > Andre > > > > > > > -----BEGIN PGP SIGNATURE----- > Version: PGP Desktop 9.9.0 (Build 397) > Charset: ISO-8859-1 > > wsBVAwUBSP82asjAfVnRK9hXAQgJmwf9Gl+a9xhnH/SOwphSa/3wNqWMhQ+Od+3S > crAMZpO9XIiKuo4UdC+L13T2wQ38H5flnrq0bwQcQ8//AcZVeU6tSiy1xWR5chOd > 36I9T675FrzdQtUWBp+lKsLiotubAcjgWvWVyw65tB27gUaIojzISMqm6XHVlfW3 > 2IDZZoo/mzPSkOnX44tO70i0kCKY5hBaJ9JlQfAW4giW7Q1Y+MTHUYm25npYrD8i > mWgsEbTGUlBwM6JgNnDTFgRM6riOHw1GOdcZjm6/THSTjYz5IaItUyMD6SwEzOgE > igxaOzT2V77zJKs4b9P8Xj1jCTZztkl45hbSFaOdYhB3rVTk0us73Q== > =UtkX > -----END PGP SIGNATURE----- > > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSP9HXMjAfVnRK9hXAQgyawgA2ME0jiz4L9+cOk0xTX5GTCPRu2YoLC4E fCxcL+e+AoQ3GuPKZwpf6oDvFWeq41pfLOyC4gBHn6EWgqzK6/LQCl+52dfY9cPN qqm+RsarWv8ErwKCzapb5+PhxwYWF31JNOp4U3UK/dg6Ow6MihLkjLVRrgXYSQW4 aXp2lMji4kksb6jVC5OIih/ZWooC0Wkb8h412H8BE6rT9LEyg/T1Y65Kzl57aIH7 smyLeoOGhgRjsNp3JfCqpLn59BdBqnXQteno+VsRE3fa296cdiDHQR5xd4sPYAbf 3mEfuFrTxaX+qtMWNLmXrkzqT4vlbvGw9OZtjFp953K6UOHGA7w6QA== =wTIM -----END PGP SIGNATURE----- From joep.gommers at gmail.com Wed Oct 22 11:49:53 2008 From: joep.gommers at gmail.com (Joep Gommers) Date: Wed, 22 Oct 2008 17:49:53 +0200 Subject: [Discussion] new thread: biggest threats In-Reply-To: References: <48FF31B9.2070602@packetspy.com> <48FF3668.5070601@packetspy.com> Message-ID: <48FF4BA1.1010302@gmail.com> Excellent post. I've been using the concept of generic signature to supply weight for higher-lever logic for a while now, be it in research environments only, - including such features inside a more low-level engine would be incredible. By using a variant of a hidden markov model (HMM), one could calculate probabilities of a certain process (represented by that model), states within that process and state transitions occurring. I have no clue if such a HMM could be used to create advanced signatures in a Bro-like infrastructure, but the possibilities if it would are fantastic. In risk management, the probability of a certain thing occurring are as, if not more, valuable as knowing for certain it is. The 'problem' with the more binary-output (hit/no hit) signatures is that correlated over space and time, probability as compared to 'still no hit' could again create valuable intelligence. And with the possibility of probabilities of things occurring, the possibilities for more elaborate signature increase as well. It's a kind of never ending cycle too because the concept of reporting on probabilities involves the same kind of hit/no hit signature - being it on a higher level. The trick for us INFOSEC people is to ever increase our capability of dealing in probabilities rather then 1/0 hits as long as possible. Just some 0.3$. Joep Martin Holste wrote: > All good points, and conclusions I had arrived at as well. Then I saw > the Bro XML validator plugin > which showed that it is > possible to do extremely intensive, app layer inspection if you can do > it in a clustered, asynchronous way. The other thought I had was that > you wouldn't have to have the horsepower to actually render to memory > what the browser would see. You could get 80-90% of the value through > JS profiling, as in, how many eval calls does the code make? What is > the entropy like? If we could "score" JS, it would really open the door > to a lot of possibilities, but we wouldn't necessarily have to create > the entire DOM on the fly, just get a feel for the style. Since Mozilla > already provides the spidermonkey command line JS engine/API, the heavy > lifting is already done. > > Here's an example of how I see this working: A sig detects that a user > agent on a host changes and makes a note of it in a global array. (The > UA changing is obviously not malicious in its own right). Another sig > tracks which HTTP referrers were sent via iframe and keeps a short (by > time) list of those in a global array. A third sig tries to decode JS a > bit to see how many evals are in the code. A final sig looks for any > executable downloads. If all four hit on the same client in quick > succession, you've got "probable cause." > > Sounds like a lot of work, but I don't think it's as bad as the first > impression makes it seem because instead of writing thousands of > signatures for very specific occurrences like we do currently, you write > a few hundred sigs for generic events, and then a few more hundred sigs > for the matching up the combination of generic events which equals > "badness." > > This is where Bro would make a lot of this easy. For instance, Bro > allows for truly global lists to be shared between nodes, as in, the > list of clients which have recently switched their user agent. So every > node is aware of this stuff. Have you guys seen how cheap quad cores > have become? Run 4 processes on the same box and you get some easy load > balancing. Then you have enough horsepower to do this app layer > inspection without packet drops. I suspect you could even implement > load shedding if they get in trouble. > > I admit that this all sounds rather complicated, but there don't seem to > be any simple exploits out there anymore in today's commoditized > malware. I don't think we can continue to depend on single-event > signatures. > > --Martin > > On Wed, Oct 22, 2008 at 9:19 AM, Andre Ludwig > wrote: > > Correct, but the problem with JS is that it is an extremely expressive > language. One that in conjunction with the DOM provides a nearly > infinite number of ways to shovel an exploit to vulnerable browser, > kernel, or third party code. That is the fundamental problem with IDS > when attempting to detect/block these sorts of exploits at this layer. > While I agree that your suggestion would be helpful it IMHO has no place > on the network for sheer performance reasons alone (can you imagine > trying to emulate a full DOM/JS engine at wire speed). Browsers can > barely keep up with the onslaught of horribly written JS code, much less > purposefully obfuscated or obtuse JS that hides an exploit. This is a > topic that I have dedicated some thought to over the last three years so > I would love to hear everyones ideas on the subject. > > Andre Ludwig > > Martin Holste wrote: >> Right, just like a network is a means, not an end. You inspect > the network because you know the threats have to traverse it, and I > would argue that similarly, there is value in inspecting Javascript > because like the network, it is ubiquitously involved in malicious > activity. I'm suggesting a JIDS as a plugin to a NIDS. >> On Wed, Oct 22, 2008 at 8:59 AM, Andre Ludwig < > aludwig at packetspy.com > wrote: > >> * PGP Bad Signature, Signed by an unverified key: 10/22/08 at 09:59:22 > >> JS is a means, not an end. > >> Andre > > > > > ------------------------------------------------------------------------ > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- Best regards, Joep Gommers +1 (571) 451-2007 +1 (703) 879-1681 Skype: jgommers From urule99 at gmail.com Wed Oct 22 18:35:37 2008 From: urule99 at gmail.com (Blake Hartstein) Date: Wed, 22 Oct 2008 18:35:37 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> Message-ID: <48FFAAB9.7010801@gmail.com> What if we focus on developing and distributing a better language for communicating actionable events? The idea is to make all intelligence more valuable and immediate. If I see this input event, alert, network, ISP, javascript, URL, how does it impact me, and what do I do about it? Instead of just collecting and distributing, the goal is to direct the actions for (ISP takedown, firewall, admin action, more). This enhances all of the prior research we've already done. Blake robert.jamison at bt.com wrote: > It seems we're a split camp with: > > [Keynesian CAMP] > Client Side Product/Service with ability to protect/detect compromise on > grannyx home user > *scope most thoroughly represented by Martin's " RFC: Proposal for > Analysis Framework" > > [Supply Side CAMP] > Focus on server side protection for net critical assets > *Andre/Jack "What is absolutely horrible in its current state is > IDS/IPS" / "Client side is simply not possible due to political and > religious issues." > > Additional notes gathered (I've just caught up on my reading;-) > > (a) Consideration for re-write defanging capability as inline protection > (b) Efficiency in stream storage--essentially normalize data inspection > so it doesn't have to be redone by multiple engines > (c) XML vs. Binary distribution of verbose alerts vs. instruction > inferred datapoints > (d) Consideration for extending existing project Bro > > Anything I'm missing? > > Rob From mcholste at gmail.com Wed Oct 22 19:35:15 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Oct 2008 18:35:15 -0500 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <48FFAAB9.7010801@gmail.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> Message-ID: I like the idea, but are there really that many different actions to be taken, and aren't they going to be org specific? If I know that an IP is spamming, I don't just want to block them from emailing, I want to block all access from that IP since it is untrustworthy. But I do think there is a lot of value in developing and distributing better language for describing why the given IP/host is now on the list and other descriptions. I'm more for giving orgs the most information that we can, and leaving it to them to implement the actual blocking decisions. --Martin On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein wrote: > What if we focus on developing and distributing a better language for > communicating actionable events? > The idea is to make all intelligence more valuable and immediate. If I > see this input event, alert, network, ISP, javascript, URL, how does it > impact me, and what do I do about it? Instead of just collecting and > distributing, the goal is to direct the actions for (ISP takedown, > firewall, admin action, more). This enhances all of the prior research > we've already done. > > > Blake > > > > robert.jamison at bt.com wrote: > > It seems we're a split camp with: > > > > [Keynesian CAMP] > > Client Side Product/Service with ability to protect/detect compromise on > > grannyx home user > > *scope most thoroughly represented by Martin's " RFC: Proposal for > > Analysis Framework" > > > > [Supply Side CAMP] > > Focus on server side protection for net critical assets > > *Andre/Jack "What is absolutely horrible in its current state is > > IDS/IPS" / "Client side is simply not possible due to political and > > religious issues." > > > > Additional notes gathered (I've just caught up on my reading;-) > > > > (a) Consideration for re-write defanging capability as inline protection > > (b) Efficiency in stream storage--essentially normalize data inspection > > so it doesn't have to be redone by multiple engines > > (c) XML vs. Binary distribution of verbose alerts vs. instruction > > inferred datapoints > > (d) Consideration for extending existing project Bro > > > > Anything I'm missing? > > > > Rob > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/6af17dfb/attachment.html From david.glosser at gmail.com Wed Oct 22 19:53:03 2008 From: david.glosser at gmail.com (David Glosser) Date: Wed, 22 Oct 2008 19:53:03 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> Message-ID: I think listing all possible actions is a good exercise, then, after brainstorming is over, deciding what is best. For example, for granny, there's blocking/preventing at the browser level, blocking/preventing at the OS, at her home router, via DNS, via IP, etc. then there's blocking (or null routing, etc) at her ISP, bringing down the host, contacting the hosting provider, etc... On Wed, Oct 22, 2008 at 7:35 PM, Martin Holste wrote: > I like the idea, but are there really that many different actions to be > taken, and aren't they going to be org specific? If I know that an IP is > spamming, I don't just want to block them from emailing, I want to block all > access from that IP since it is untrustworthy. But I do think there is a > lot of value in developing and distributing better language for describing > why the given IP/host is now on the list and other descriptions. I'm more > for giving orgs the most information that we can, and leaving it to them to > implement the actual blocking decisions. > > --Martin > > > On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein wrote: > >> What if we focus on developing and distributing a better language for >> communicating actionable events? >> The idea is to make all intelligence more valuable and immediate. If I >> see this input event, alert, network, ISP, javascript, URL, how does it >> impact me, and what do I do about it? Instead of just collecting and >> distributing, the goal is to direct the actions for (ISP takedown, >> firewall, admin action, more). This enhances all of the prior research >> we've already done. >> >> >> Blake >> >> >> >> robert.jamison at bt.com wrote: >> > It seems we're a split camp with: >> > >> > [Keynesian CAMP] >> > Client Side Product/Service with ability to protect/detect compromise on >> > grannyx home user >> > *scope most thoroughly represented by Martin's " RFC: Proposal for >> > Analysis Framework" >> > >> > [Supply Side CAMP] >> > Focus on server side protection for net critical assets >> > *Andre/Jack "What is absolutely horrible in its current state is >> > IDS/IPS" / "Client side is simply not possible due to political and >> > religious issues." >> > >> > Additional notes gathered (I've just caught up on my reading;-) >> > >> > (a) Consideration for re-write defanging capability as inline protection >> > (b) Efficiency in stream storage--essentially normalize data inspection >> > so it doesn't have to be redone by multiple engines >> > (c) XML vs. Binary distribution of verbose alerts vs. instruction >> > inferred datapoints >> > (d) Consideration for extending existing project Bro >> > >> > Anything I'm missing? >> > >> > Rob >> >> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/1b61b98f/attachment.html From david.glosser at gmail.com Wed Oct 22 20:04:01 2008 From: david.glosser at gmail.com (David Glosser) Date: Wed, 22 Oct 2008 20:04:01 -0400 Subject: [Discussion] Features In-Reply-To: <48FCC484.9080308@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> <48FCC484.9080308@jonkmans.com> Message-ID: Back to the idea of Spam-assassin scoring: Once a bad host is identified, then I'm wondering if IP reputation could maybe using a "halo effect" whereby other IPs by the same provider are given lower scores. Say reputation is from 0-100 (with 100 being bad) So if an IP on hosting provider "Btrivo" contains malware, that IP gets a reputation score of 50. Adjacent IPs are given a reputation score of 30. All of the IPs in "Btrivo's" netblock are given a reputation score of 10. Two adjacent IP addresses each hosting malware each will get a score of 90 (50+30+10). If malware goes away, score decreases each day. For each day host is still up, score increases.... On Mon, Oct 20, 2008 at 1:48 PM, Matt Jonkman wrote: > I agree. I know a lot of installs that use rotating tcpdump files which > allows the analyst to go back and see everything that happened related > to an incident. And as long as you have good disks it's a very feasible > thing to do. If you're running inline you can even do the detection from > these files to avoid packet loss at peak traffic flows. > > Definitely adding this to the list! > > Matt > > Jeremy wrote: > > Thought of another feature I would like to see.... > > > > [not sure of the number] > > I would like to see a single capture engine have the ability to > > perform the analysis of the data (however that may be), and also this > > same engine should be able to capture session statistics (flow data), > > and full packet captures. What I mean by this is that currently I run > > 3 applications on all of my IDS' to get this data. I use SANCP for > > session data, Tcpdump for full packet capture and snort for IDS > > functionality. I believe it would improve overall system performance > > to only have one application read the packets from the capture library > > such as libpcap instead of several applications having to read this > > data. I know that I ran into a wall with the 32 bit architecture in > > that I could only allocate small amounts of Low memory space to each > > application, as the kernel can only be allocated 1 GB. This 1 GB has > > to be divided up between with these three applications and well all > > other applications. I recently was told you could actually allocate 2 > > GB low memory on a 32 bit architecture, but as of today I have not > > figured out how. If you over run the low memory space reserved by the > > kernel you end up with oom kills which can be really frustrating as it > > will kill off applications based off some really odd/cryptic > > algorithm. Anyways getting back to one application performing all > > three tasks, the packet only needs to be copied once and therefore I > > could allocate a substantial amount of the low memory reserved by the > > kernel to this application. Now I know not everyone will want to > > capture flow data, and full packet data on their IDS boxes, but if > > this was a configurable option it may help some of us. I also noticed > > someone else posted about SANCP indexing abilities, this would be nice > > to have in the new engine as well to make correlating Session data > > (flow data) to full packet captures very fast and easy. Having the > > full packet captures on my IDS boxes has really helped in identifying > > false positives and signature tweaking. The session data (flow data) > > has proven to be an outstanding addition to our anomaly detection > > capabilities. > > > > > > --jeremy > > > > On Sat, Oct 18, 2008 at 9:17 PM, th3 m0nq wrote: > >> > >> As Robert and Martin eluded in an other > >> thread, any sort of correlation and IP analysis, scoring, > >> severity-bumping, etc, is best done in the analysis or portal engine. > >> > >> > >> I agree here, but does next generation mean an advanced version of > >> snort? Does next generation mean an evolution in terms of thinking > >> about IDS/IPS? Do we leave things as they are and say that there > >> exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system > >> for my network? I would like to see an evolution of such technologies > >> be more biological in nature. > >> > >> > >> I don't think any existing detection engine of an IDS is the proper > place. > >> But perhaps a parallel engine that collects network stream data and does > >> matching on hostile IP's, and then checks (perhaps by packet/session > >> timestamp) if the IDS did produce an alert in a recent time window > >> (since the session-IP-eval-engine will surely lag behind) > >> > >> > >> I am with you here as well. Why is the parallel engine not part of > >> the IDS? Why should next generation IDS mean a better snort? Why > >> can't it mean a better way of looking at network security? > >> http://en.wikipedia.org/wiki/Immune_system#Layered_defense What about > >> an all encompassing IDS/IPS _system_? > >> > >> I am not saying I have the answers, or something similar to an expert > >> system is the answer. I am saying I think the issue should be looked > >> at as we dump the current way we do things in a temporary holding > >> cell, and think in evolutionary terms. This makes 4 cents ;-). > >> > >> > >> Adrian > >> > >> > >> > >> > >> > >> > >> > >> > >> On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe wrote: > >>> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: > >>>> [..] I think there is > >>>> room, at minimum, to have packets that do not necessarily trigger a > >>>> rule "flagged" in some way in an IDS. This could be implemented as a > >>>> plug-in component of the system. This may be way outside of the scope > >>>> of what you guys are looking at, but that's my 2 cents. > >>> Receiving alerts (with session content) on IP's that don't trip > existing > >>> signatures, but are present in a hostile-IP list, is a good idea (for > >>> example, to detect new evasion techniques of existing badware). > >>> > >>> However, I don't think the sensor is the proper place for that due to > >>> the immense IP-matching workload that would have to occur on a > >>> per-packet or per-session basis (remember, we're talking at least tens > >>> of thousands hostile IP's). As Robert and Martin eluded in an other > >>> thread, any sort of correlation and IP analysis, scoring, > >>> severity-bumping, etc, is best done in the analysis or portal engine. > >>> (We do those things in our shop as well, with back-end scripts and our > >>> portal). > >>> > >>> While you can make decisions on the IP "badness value" for analysis, > you > >>> would of course miss the content of the IP not receiving an alert. I > >>> don't think any existing detection engine of an IDS is the proper > place. > >>> But perhaps a parallel engine that collects network stream data and > does > >>> matching on hostile IP's, and then checks (perhaps by packet/session > >>> timestamp) if the IDS did produce an alert in a recent time window > >>> (since the session-IP-eval-engine will surely lag behind). That check > >>> would probably be best done against the IDS/portal database. A simple > >>> bpf filter with hostile 20,000+ IP's to capture content will probably > >>> not cut it :) > >>> > >>> -Frank > >>> > >>> > >>> -- > >>> It is said that the Internet is a public utility. As such, it is best > >>> compared to a sewer. A big, fat pipe with a bunch of crap sloshing > >>> against your ports. > >>> > >>> > >> _______________________________________________ > >> Discussion mailing list > >> Discussion at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > >> > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/a63b7500/attachment-0001.html From mcholste at gmail.com Wed Oct 22 20:19:46 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Oct 2008 19:19:46 -0500 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <1a0f92ae0810161933q4741dfedp55231cb4e4601ef7@mail.gmail.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> <48FCC484.9080308@jonkmans.com> Message-ID: I really like that idea. It won't directly lead to blocking innocent IP's, but will still give the good guys a simple and reliable predictive capability. On Wed, Oct 22, 2008 at 7:04 PM, David Glosser wrote: > Back to the idea of Spam-assassin scoring: > > Once a bad host is identified, then I'm wondering if IP reputation could > maybe using a "halo effect" whereby other IPs by the same provider are given > lower scores. > > Say reputation is from 0-100 (with 100 being bad) > > So if an IP on hosting provider "Btrivo" contains malware, that IP gets a > reputation score of 50. > Adjacent IPs are given a reputation score of 30. > All of the IPs in "Btrivo's" netblock are given a reputation score of 10. > > Two adjacent IP addresses each hosting malware each will get a score of 90 > (50+30+10). > > If malware goes away, score decreases each day. For each day host is still > up, score increases.... > > > > > On Mon, Oct 20, 2008 at 1:48 PM, Matt Jonkman wrote: > >> I agree. I know a lot of installs that use rotating tcpdump files which >> allows the analyst to go back and see everything that happened related >> to an incident. And as long as you have good disks it's a very feasible >> thing to do. If you're running inline you can even do the detection from >> these files to avoid packet loss at peak traffic flows. >> >> Definitely adding this to the list! >> >> Matt >> >> Jeremy wrote: >> > Thought of another feature I would like to see.... >> > >> > [not sure of the number] >> > I would like to see a single capture engine have the ability to >> > perform the analysis of the data (however that may be), and also this >> > same engine should be able to capture session statistics (flow data), >> > and full packet captures. What I mean by this is that currently I run >> > 3 applications on all of my IDS' to get this data. I use SANCP for >> > session data, Tcpdump for full packet capture and snort for IDS >> > functionality. I believe it would improve overall system performance >> > to only have one application read the packets from the capture library >> > such as libpcap instead of several applications having to read this >> > data. I know that I ran into a wall with the 32 bit architecture in >> > that I could only allocate small amounts of Low memory space to each >> > application, as the kernel can only be allocated 1 GB. This 1 GB has >> > to be divided up between with these three applications and well all >> > other applications. I recently was told you could actually allocate 2 >> > GB low memory on a 32 bit architecture, but as of today I have not >> > figured out how. If you over run the low memory space reserved by the >> > kernel you end up with oom kills which can be really frustrating as it >> > will kill off applications based off some really odd/cryptic >> > algorithm. Anyways getting back to one application performing all >> > three tasks, the packet only needs to be copied once and therefore I >> > could allocate a substantial amount of the low memory reserved by the >> > kernel to this application. Now I know not everyone will want to >> > capture flow data, and full packet data on their IDS boxes, but if >> > this was a configurable option it may help some of us. I also noticed >> > someone else posted about SANCP indexing abilities, this would be nice >> > to have in the new engine as well to make correlating Session data >> > (flow data) to full packet captures very fast and easy. Having the >> > full packet captures on my IDS boxes has really helped in identifying >> > false positives and signature tweaking. The session data (flow data) >> > has proven to be an outstanding addition to our anomaly detection >> > capabilities. >> > >> > >> > --jeremy >> > >> > On Sat, Oct 18, 2008 at 9:17 PM, th3 m0nq wrote: >> >> >> >> As Robert and Martin eluded in an other >> >> thread, any sort of correlation and IP analysis, scoring, >> >> severity-bumping, etc, is best done in the analysis or portal engine. >> >> >> >> >> >> I agree here, but does next generation mean an advanced version of >> >> snort? Does next generation mean an evolution in terms of thinking >> >> about IDS/IPS? Do we leave things as they are and say that there >> >> exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system >> >> for my network? I would like to see an evolution of such technologies >> >> be more biological in nature. >> >> >> >> >> >> I don't think any existing detection engine of an IDS is the proper >> place. >> >> But perhaps a parallel engine that collects network stream data and >> does >> >> matching on hostile IP's, and then checks (perhaps by packet/session >> >> timestamp) if the IDS did produce an alert in a recent time window >> >> (since the session-IP-eval-engine will surely lag behind) >> >> >> >> >> >> I am with you here as well. Why is the parallel engine not part of >> >> the IDS? Why should next generation IDS mean a better snort? Why >> >> can't it mean a better way of looking at network security? >> >> http://en.wikipedia.org/wiki/Immune_system#Layered_defense What about >> >> an all encompassing IDS/IPS _system_? >> >> >> >> I am not saying I have the answers, or something similar to an expert >> >> system is the answer. I am saying I think the issue should be looked >> >> at as we dump the current way we do things in a temporary holding >> >> cell, and think in evolutionary terms. This makes 4 cents ;-). >> >> >> >> >> >> Adrian >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe wrote: >> >>> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: >> >>>> [..] I think there is >> >>>> room, at minimum, to have packets that do not necessarily trigger a >> >>>> rule "flagged" in some way in an IDS. This could be implemented as a >> >>>> plug-in component of the system. This may be way outside of the >> scope >> >>>> of what you guys are looking at, but that's my 2 cents. >> >>> Receiving alerts (with session content) on IP's that don't trip >> existing >> >>> signatures, but are present in a hostile-IP list, is a good idea (for >> >>> example, to detect new evasion techniques of existing badware). >> >>> >> >>> However, I don't think the sensor is the proper place for that due to >> >>> the immense IP-matching workload that would have to occur on a >> >>> per-packet or per-session basis (remember, we're talking at least tens >> >>> of thousands hostile IP's). As Robert and Martin eluded in an other >> >>> thread, any sort of correlation and IP analysis, scoring, >> >>> severity-bumping, etc, is best done in the analysis or portal engine. >> >>> (We do those things in our shop as well, with back-end scripts and our >> >>> portal). >> >>> >> >>> While you can make decisions on the IP "badness value" for analysis, >> you >> >>> would of course miss the content of the IP not receiving an alert. I >> >>> don't think any existing detection engine of an IDS is the proper >> place. >> >>> But perhaps a parallel engine that collects network stream data and >> does >> >>> matching on hostile IP's, and then checks (perhaps by packet/session >> >>> timestamp) if the IDS did produce an alert in a recent time window >> >>> (since the session-IP-eval-engine will surely lag behind). That check >> >>> would probably be best done against the IDS/portal database. A simple >> >>> bpf filter with hostile 20,000+ IP's to capture content will probably >> >>> not cut it :) >> >>> >> >>> -Frank >> >>> >> >>> >> >>> -- >> >>> It is said that the Internet is a public utility. As such, it is best >> >>> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >> >>> against your ports. >> >>> >> >>> >> >> _______________________________________________ >> >> Discussion mailing list >> >> Discussion at openinfosecfoundation.org >> >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/0fa32426/attachment.html From david.glosser at gmail.com Wed Oct 22 21:29:59 2008 From: david.glosser at gmail.com (David Glosser) Date: Wed, 22 Oct 2008 21:29:59 -0400 Subject: [Discussion] Features In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <48F91733.6050008@inliniac.net> <1224367303.13106.17.camel@localhost> <1a0f92ae0810191746h5ed6b5e6yeb2109f89e173884@mail.gmail.com> <48FCC484.9080308@jonkmans.com> Message-ID: it will also give the "Btrivos" of the world an incentive to clean up their act, as the more IPs in their netblock have a bad reputation, the worse off ALL IPs in their netblock are. Sort of like what hostexploit did by calling out the hosting provider..... Let the hosting providers know that their entire range will be negativelt impacted, with a sliding scale based on number of "bad" hosts and other parameters.... On Wed, Oct 22, 2008 at 8:19 PM, Martin Holste wrote: > I really like that idea. It won't directly lead to blocking innocent IP's, > but will still give the good guys a simple and reliable predictive > capability. > > > On Wed, Oct 22, 2008 at 7:04 PM, David Glosser wrote: > >> Back to the idea of Spam-assassin scoring: >> >> Once a bad host is identified, then I'm wondering if IP reputation could >> maybe using a "halo effect" whereby other IPs by the same provider are given >> lower scores. >> >> Say reputation is from 0-100 (with 100 being bad) >> >> So if an IP on hosting provider "Btrivo" contains malware, that IP gets a >> reputation score of 50. >> Adjacent IPs are given a reputation score of 30. >> All of the IPs in "Btrivo's" netblock are given a reputation score of 10. >> >> Two adjacent IP addresses each hosting malware each will get a score of 90 >> (50+30+10). >> >> If malware goes away, score decreases each day. For each day host is still >> up, score increases.... >> >> >> >> >> On Mon, Oct 20, 2008 at 1:48 PM, Matt Jonkman wrote: >> >>> I agree. I know a lot of installs that use rotating tcpdump files which >>> allows the analyst to go back and see everything that happened related >>> to an incident. And as long as you have good disks it's a very feasible >>> thing to do. If you're running inline you can even do the detection from >>> these files to avoid packet loss at peak traffic flows. >>> >>> Definitely adding this to the list! >>> >>> Matt >>> >>> Jeremy wrote: >>> > Thought of another feature I would like to see.... >>> > >>> > [not sure of the number] >>> > I would like to see a single capture engine have the ability to >>> > perform the analysis of the data (however that may be), and also this >>> > same engine should be able to capture session statistics (flow data), >>> > and full packet captures. What I mean by this is that currently I run >>> > 3 applications on all of my IDS' to get this data. I use SANCP for >>> > session data, Tcpdump for full packet capture and snort for IDS >>> > functionality. I believe it would improve overall system performance >>> > to only have one application read the packets from the capture library >>> > such as libpcap instead of several applications having to read this >>> > data. I know that I ran into a wall with the 32 bit architecture in >>> > that I could only allocate small amounts of Low memory space to each >>> > application, as the kernel can only be allocated 1 GB. This 1 GB has >>> > to be divided up between with these three applications and well all >>> > other applications. I recently was told you could actually allocate 2 >>> > GB low memory on a 32 bit architecture, but as of today I have not >>> > figured out how. If you over run the low memory space reserved by the >>> > kernel you end up with oom kills which can be really frustrating as it >>> > will kill off applications based off some really odd/cryptic >>> > algorithm. Anyways getting back to one application performing all >>> > three tasks, the packet only needs to be copied once and therefore I >>> > could allocate a substantial amount of the low memory reserved by the >>> > kernel to this application. Now I know not everyone will want to >>> > capture flow data, and full packet data on their IDS boxes, but if >>> > this was a configurable option it may help some of us. I also noticed >>> > someone else posted about SANCP indexing abilities, this would be nice >>> > to have in the new engine as well to make correlating Session data >>> > (flow data) to full packet captures very fast and easy. Having the >>> > full packet captures on my IDS boxes has really helped in identifying >>> > false positives and signature tweaking. The session data (flow data) >>> > has proven to be an outstanding addition to our anomaly detection >>> > capabilities. >>> > >>> > >>> > --jeremy >>> > >>> > On Sat, Oct 18, 2008 at 9:17 PM, th3 m0nq wrote: >>> >> >>> >> As Robert and Martin eluded in an other >>> >> thread, any sort of correlation and IP analysis, scoring, >>> >> severity-bumping, etc, is best done in the analysis or portal engine. >>> >> >>> >> >>> >> I agree here, but does next generation mean an advanced version of >>> >> snort? Does next generation mean an evolution in terms of thinking >>> >> about IDS/IPS? Do we leave things as they are and say that there >>> >> exist NIDs, DIDs, HIDs, and they don't form a cohesive immune system >>> >> for my network? I would like to see an evolution of such technologies >>> >> be more biological in nature. >>> >> >>> >> >>> >> I don't think any existing detection engine of an IDS is the proper >>> place. >>> >> But perhaps a parallel engine that collects network stream data and >>> does >>> >> matching on hostile IP's, and then checks (perhaps by packet/session >>> >> timestamp) if the IDS did produce an alert in a recent time window >>> >> (since the session-IP-eval-engine will surely lag behind) >>> >> >>> >> >>> >> I am with you here as well. Why is the parallel engine not part of >>> >> the IDS? Why should next generation IDS mean a better snort? Why >>> >> can't it mean a better way of looking at network security? >>> >> http://en.wikipedia.org/wiki/Immune_system#Layered_defense What >>> about >>> >> an all encompassing IDS/IPS _system_? >>> >> >>> >> I am not saying I have the answers, or something similar to an expert >>> >> system is the answer. I am saying I think the issue should be looked >>> >> at as we dump the current way we do things in a temporary holding >>> >> cell, and think in evolutionary terms. This makes 4 cents ;-). >>> >> >>> >> >>> >> Adrian >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> On Sat, Oct 18, 2008 at 5:01 PM, Frank Knobbe >>> wrote: >>> >>> On Sat, 2008-10-18 at 16:28 -0500, th3 m0nq wrote: >>> >>>> [..] I think there is >>> >>>> room, at minimum, to have packets that do not necessarily trigger a >>> >>>> rule "flagged" in some way in an IDS. This could be implemented as >>> a >>> >>>> plug-in component of the system. This may be way outside of the >>> scope >>> >>>> of what you guys are looking at, but that's my 2 cents. >>> >>> Receiving alerts (with session content) on IP's that don't trip >>> existing >>> >>> signatures, but are present in a hostile-IP list, is a good idea (for >>> >>> example, to detect new evasion techniques of existing badware). >>> >>> >>> >>> However, I don't think the sensor is the proper place for that due to >>> >>> the immense IP-matching workload that would have to occur on a >>> >>> per-packet or per-session basis (remember, we're talking at least >>> tens >>> >>> of thousands hostile IP's). As Robert and Martin eluded in an other >>> >>> thread, any sort of correlation and IP analysis, scoring, >>> >>> severity-bumping, etc, is best done in the analysis or portal engine. >>> >>> (We do those things in our shop as well, with back-end scripts and >>> our >>> >>> portal). >>> >>> >>> >>> While you can make decisions on the IP "badness value" for analysis, >>> you >>> >>> would of course miss the content of the IP not receiving an alert. I >>> >>> don't think any existing detection engine of an IDS is the proper >>> place. >>> >>> But perhaps a parallel engine that collects network stream data and >>> does >>> >>> matching on hostile IP's, and then checks (perhaps by packet/session >>> >>> timestamp) if the IDS did produce an alert in a recent time window >>> >>> (since the session-IP-eval-engine will surely lag behind). That check >>> >>> would probably be best done against the IDS/portal database. A simple >>> >>> bpf filter with hostile 20,000+ IP's to capture content will probably >>> >>> not cut it :) >>> >>> >>> >>> -Frank >>> >>> >>> >>> >>> >>> -- >>> >>> It is said that the Internet is a public utility. As such, it is best >>> >>> compared to a sewer. A big, fat pipe with a bunch of crap sloshing >>> >>> against your ports. >>> >>> >>> >>> >>> >> _______________________________________________ >>> >> Discussion mailing list >>> >> Discussion at openinfosecfoundation.org >>> >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >> >>> > _______________________________________________ >>> > Discussion mailing list >>> > Discussion at openinfosecfoundation.org >>> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >>> -- >>> -------------------------------------------- >>> Matthew Jonkman >>> Emerging Threats >>> Phone 765-429-0398 >>> Fax 312-264-0205 >>> http://www.emergingthreats.net >>> -------------------------------------------- >>> >>> PGP: http://www.jonkmans.com/mattjonkman.asc >>> >>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081022/f20e8099/attachment-0001.html From jonkman at jonkmans.com Fri Oct 24 11:43:01 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 24 Oct 2008 11:43:01 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> Message-ID: <4901ED05.2000609@jonkmans.com> Ya, we're definitely interested in getting all ideas on the board. We have time to iron them out and choose what we go after now vs later vs never. Or things we spawn into sub-projects. Matt David Glosser wrote: > I think listing all possible actions is a good exercise, then, after > brainstorming is over, deciding what is best. > > For example, for granny, there's blocking/preventing at the browser > level, blocking/preventing at the OS, at her home router, via DNS, via > IP, etc. then there's blocking (or null routing, etc) at her ISP, > bringing down the host, contacting the hosting provider, etc... > > > > > On Wed, Oct 22, 2008 at 7:35 PM, Martin Holste > wrote: > > I like the idea, but are there really that many different actions to > be taken, and aren't they going to be org specific? If I know that > an IP is spamming, I don't just want to block them from emailing, I > want to block all access from that IP since it is untrustworthy. > But I do think there is a lot of value in developing and > distributing better language for describing why the given IP/host is > now on the list and other descriptions. I'm more for giving orgs > the most information that we can, and leaving it to them to > implement the actual blocking decisions. > > --Martin > > > On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein > wrote: > > What if we focus on developing and distributing a better > language for > communicating actionable events? > The idea is to make all intelligence more valuable and > immediate. If I > see this input event, alert, network, ISP, javascript, URL, how > does it > impact me, and what do I do about it? Instead of just collecting and > distributing, the goal is to direct the actions for (ISP takedown, > firewall, admin action, more). This enhances all of the prior > research > we've already done. > > > Blake > > > > robert.jamison at bt.com wrote: > > It seems we're a split camp with: > > > > [Keynesian CAMP] > > Client Side Product/Service with ability to protect/detect > compromise on > > grannyx home user > > *scope most thoroughly represented by Martin's " RFC: Proposal for > > Analysis Framework" > > > > [Supply Side CAMP] > > Focus on server side protection for net critical assets > > *Andre/Jack "What is absolutely horrible in its current state is > > IDS/IPS" / "Client side is simply not possible due to > political and > > religious issues." > > > > Additional notes gathered (I've just caught up on my reading;-) > > > > (a) Consideration for re-write defanging capability as inline > protection > > (b) Efficiency in stream storage--essentially normalize data > inspection > > so it doesn't have to be redone by multiple engines > > (c) XML vs. Binary distribution of verbose alerts vs. instruction > > inferred datapoints > > (d) Consideration for extending existing project Bro > > > > Anything I'm missing? > > > > Rob > > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Oct 24 11:45:49 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 24 Oct 2008 11:45:49 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> Message-ID: <4901EDAD.9040606@jonkmans.com> Martin Holste wrote: > I like the idea, but are there really that many different actions to be > taken, and aren't they going to be org specific? If I know that an IP > is spamming, I don't just want to block them from emailing, I want to > block all access from that IP since it is untrustworthy. I want the ip reputation data to be categorized. For instance the IP would have a reputation number in spam, phishing, malware host, CnC, known bot, scanner, etc. Maybe stuff to cover underground forums, whatever. There'd be an average score amassing all of the categories, and the end user could weight each category to be more an influence on the average, or even just make block decisions on a few categories they're interested in. That make sense? So if I ran a mail farm I could weight the spam category very high, or just look at that alone. Or if I were protecting a net of users I'd probably take the average but weight the CnC and Malware hosts higher. Matt But I do think > there is a lot of value in developing and distributing better language > for describing why the given IP/host is now on the list and other > descriptions. I'm more for giving orgs the most information that we > can, and leaving it to them to implement the actual blocking decisions. > > --Martin > > On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein > wrote: > > What if we focus on developing and distributing a better language for > communicating actionable events? > The idea is to make all intelligence more valuable and immediate. If I > see this input event, alert, network, ISP, javascript, URL, how does it > impact me, and what do I do about it? Instead of just collecting and > distributing, the goal is to direct the actions for (ISP takedown, > firewall, admin action, more). This enhances all of the prior research > we've already done. > > > Blake > > > > robert.jamison at bt.com wrote: > > It seems we're a split camp with: > > > > [Keynesian CAMP] > > Client Side Product/Service with ability to protect/detect > compromise on > > grannyx home user > > *scope most thoroughly represented by Martin's " RFC: Proposal for > > Analysis Framework" > > > > [Supply Side CAMP] > > Focus on server side protection for net critical assets > > *Andre/Jack "What is absolutely horrible in its current state is > > IDS/IPS" / "Client side is simply not possible due to political and > > religious issues." > > > > Additional notes gathered (I've just caught up on my reading;-) > > > > (a) Consideration for re-write defanging capability as inline > protection > > (b) Efficiency in stream storage--essentially normalize data > inspection > > so it doesn't have to be redone by multiple engines > > (c) XML vs. Binary distribution of verbose alerts vs. instruction > > inferred datapoints > > (d) Consideration for extending existing project Bro > > > > Anything I'm missing? > > > > Rob > > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From rMslade at shaw.ca Sat Oct 25 19:24:06 2008 From: rMslade at shaw.ca (Rob, grandpa of Ryan, Trevor, Devon & Hannah) Date: Sat, 25 Oct 2008 15:24:06 -0800 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <4901EDAD.9040606@jonkmans.com> References: Message-ID: <49033A16.17891.499622@localhost> Date sent: Fri, 24 Oct 2008 11:45:49 -0400 From: Matt Jonkman > I want the ip reputation data to be categorized. For instance the IP > would have a reputation number in spam, phishing, malware host, CnC, > known bot, scanner, etc. Maybe stuff to cover underground forums, whatever. Amen. As I keep telling people in malware seminars, because of the different characteristics and differing categories of protection, it becomes more important to know what specific *type* of naughtiness we are dealing with, even as more of it converges into mixed forms. ====================== (quote inserted randomly by Pegasus Mailer) rslade at vcn.bc.ca slade at victoria.tc.ca rslade at computercrime.org The Internet may promise to improve the way we educate and learn, but so did early television. TV technology has instead reduced our attention spans, reduced intellectual conversations to sound bits, and left us with the impression that in order to be informed, we must first be entertained. - Lew Platt, of HP victoria.tc.ca/techrev/rms.htm blogs.securiteam.com/index.php/archives/author/p1/ From jonkman at jonkmans.com Mon Oct 27 10:25:44 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 27 Oct 2008 11:25:44 -0400 Subject: [Discussion] Text in Msgs Message-ID: <4905DD78.5010701@jonkmans.com> Would anyone be interested in the ability to insert captured text into the alert text of an event? For instance, I was just looking at a few hits on " ET POLICY exe download via HTTP". It'd be nice for that to say: ET POLICY exe download via HTTP (down.onlinedowns.net/page/image/yahoons.exe) Quick way to decide if that was something of interest or not without having to dig into payload. What does everyone think? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From aludwig at packetspy.com Mon Oct 27 10:34:03 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Mon, 27 Oct 2008 11:34:03 -0400 Subject: [Discussion] Text in Msgs In-Reply-To: <4905DD78.5010701@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> Message-ID: <4905DF6B.2050205@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Good idea, as you could use this to pass of binaries (with the right signature set) into a sandnet. Andre Ludwig Matt Jonkman wrote: > Would anyone be interested in the ability to insert captured text into > the alert text of an event? > > For instance, I was just looking at a few hits on " ET POLICY exe > download via HTTP". It'd be nice for that to say: > > ET POLICY exe download via HTTP > (down.onlinedowns.net/page/image/yahoons.exe) > > Quick way to decide if that was something of interest or not without > having to dig into payload. > > What does everyone think? > > Matt > > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSQXfbcjAfVnRK9hXAQg57ggAxqkdTrkVLuBvd6NKgS2TX3BwW/MG89y5 MKUq/6fxvSloTLNguG7lhgzrelIaf5C724OAocJrb2WYMqRmmaKoEHdpuGOnt0y3 9zOZaTbYUzNoubv4CBjUvk1TmzLjoTzRKSvj8BI4e7IpqXzj23FgOzkWryvgTYSu XwJIPq+jurphKm+8R8/S4AsD25igBYu40ULkHErsroPbcP3yWgjGcjtA0Yuu/WBD zoHXazS5dLApuBM+EBmCh02rKz7W5N2+fJ+RB9NmqqA4wFBv0nGS76q6A9NibfqS cL6nY5hYenraMisvK2j5a5US8HSustpmV2ln/6JXJSS7HY06fqyp3w== =bJkv -----END PGP SIGNATURE----- From frank at knobbe.us Mon Oct 27 11:16:49 2008 From: frank at knobbe.us (Frank Knobbe) Date: Mon, 27 Oct 2008 11:16:49 -0500 Subject: [Discussion] Text in Msgs In-Reply-To: <4905DD78.5010701@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> Message-ID: <1225124209.2872.3.camel@localhost> On Mon, 2008-10-27 at 11:25 -0400, Matt Jonkman wrote: > Would anyone be interested in the ability to insert captured text into > the alert text of an event? No, that's a bad idea (at least if you talk about Snort). If you create new/different message texts, Snort will create a new entry in the signature table (unique to msg+gid+sid+rev). Also, you would not get the same text with barnyard or in barnyard (and probably flop) based installs since BY only reports the sid (the msg is pulled from the sid-msg.map file). While you could of course fork barnyard, my concern would be the bloat of the signature table due to unique msg texts. -Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081027/27b44ac6/attachment.bin From jonkman at jonkmans.com Mon Oct 27 11:46:29 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 27 Oct 2008 12:46:29 -0400 Subject: [Discussion] Text in Msgs In-Reply-To: <1225124209.2872.3.camel@localhost> References: <4905DD78.5010701@jonkmans.com> <1225124209.2872.3.camel@localhost> Message-ID: <4905F065.5090405@jonkmans.com> Frank Knobbe wrote: > No, that's a bad idea (at least if you talk about Snort). If you create > new/different message texts, Snort will create a new entry in the > signature table (unique to msg+gid+sid+rev). Also, you would not get the > same text with barnyard or in barnyard (and probably flop) based > installs since BY only reports the sid (the msg is pulled from the > sid-msg.map file). We are not talking snort. This is totally different. And we'll definitely not use a db schema with this issue. Matt > > While you could of course fork barnyard, my concern would be the bloat > of the signature table due to unique msg texts. > No forking here, all new. Everything from the pattern matcher on up. :) Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From frank at knobbe.us Mon Oct 27 12:04:39 2008 From: frank at knobbe.us (Frank Knobbe) Date: Mon, 27 Oct 2008 12:04:39 -0500 Subject: [Discussion] Text in Msgs In-Reply-To: <4905F065.5090405@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> <1225124209.2872.3.camel@localhost> <4905F065.5090405@jonkmans.com> Message-ID: <1225127079.2872.30.camel@localhost> On Mon, 2008-10-27 at 12:46 -0400, Matt Jonkman wrote: > We are not talking snort. This is totally different. Okay, then I don't have objections :) While you are add it, here's a feature I meant to add to Snort years ago: The ability to copy matched text into a variable used for further matching. My idea to implement two new keywords, one to store data into a temp variable, and one to match the temp variable against data that was valid for the lifetime of the session (and only available for that session). Too bad I don't remember the actual use case for it.... but it should have looked something like this: content:"http\://"; store:7,2,' ',1; Different rule (in the same session) could then use: content:"blah"; match:12,1; Parameters for store: 1st: Offset from last match (content, isdataat, etc) 2nd: Type of end match: 1: 3rd param is length 2: 3rd para is ending char match 3rd: End match: Either a length (copy 10 bytes) or an end match (in above example, a space) 4th: Number of session variable. Parameters for match: 1st: Offset from previous match. 2nd: Variable number to match. Above example would store the domain name after http:// (until reaching a space) in variable 1. The second rule using match would check if the data contained in variable 1 is present 12 bytes after blah. I think one of the reasons it never got implemented (besides there being workarounds for the specific issue in the past) was that it could easily bloat the memory of session streams since any data matching the first rule would alloc memory for the variable and tag it into the data associated with the stream. It only would be removed if the stream data is discarded. Considering there could be dozens/hundreds/thousand of matches on the first rule, it would mean dozen/hundreds/thousand times the length of the variable (plus some pointers) of memory overhead. Anyway, since you working on something new, perhaps implementing the ability to save data from sniffed traffic for later matching could be implemented more efficiently. Cheers, Frank -- It is said that the Internet is a public utility. As such, it is best compared to a sewer. A big, fat pipe with a bunch of crap sloshing against your ports. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081027/52e09fb9/attachment.bin From robert.jamison at bt.com Mon Oct 27 12:30:49 2008 From: robert.jamison at bt.com (robert.jamison@bt.com) Date: Mon, 27 Oct 2008 17:30:49 -0000 Subject: [Discussion] Text in Msgs In-Reply-To: <4905DD78.5010701@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> Message-ID: <3839DD5BC9EE23459E6FAC536A280537064E8F98@E03MVY2-UKDY.domain1.systemhost.net> Very much so. Capturing context with syslog based alert format is a major limitation from a remote monitoring standpoint. Rob -----Original Message----- From: discussion-bounces at openinfosecfoundation.org [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt Jonkman Sent: Monday, October 27, 2008 11:26 AM To: discussion at openinfosecfoundation.org Subject: [Discussion] Text in Msgs Would anyone be interested in the ability to insert captured text into the alert text of an event? For instance, I was just looking at a few hits on " ET POLICY exe download via HTTP". It'd be nice for that to say: ET POLICY exe download via HTTP (down.onlinedowns.net/page/image/yahoons.exe) Quick way to decide if that was something of interest or not without having to dig into payload. What does everyone think? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc _______________________________________________ Discussion mailing list Discussion at openinfosecfoundation.org http://lists.openinfosecfoundation.org/mailman/listinfo/discussion From john.r.pritchard at gmail.com Mon Oct 27 12:41:12 2008 From: john.r.pritchard at gmail.com (John Pritchard) Date: Mon, 27 Oct 2008 10:41:12 -0700 Subject: [Discussion] Text in Msgs In-Reply-To: <4905F065.5090405@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> <1225124209.2872.3.camel@localhost> <4905F065.5090405@jonkmans.com> Message-ID: I think this could kind of functionality could open up some very interesting avenues for additional post-alarm processing of alerts. Agree that rolling into an alert "name" could be potentially problematic. This not only applies to Snort but also to a number of SEM/SIM/SIEM/ESM applications. But, passing some additional 'high level" (overview) in new data field(s) could open some interesting avenues for further innovation. For example, the returned "match" could be sent on to additional tasks that would either additionally validate the alarm or invoke some other automated form of action. Take a signature designed to detect possible credit card patterns. Once a potential pattern match is caught, the string could then be passed to other processes to further validate or refute the trigger. For a VISA credit card sig, the initial match might simply be on a 16 digit number sequence beginning with a "four" (which would be highly prone to false positives). But, once "matched" the string could be passed to another process to run a Luhn checksum algorithm on the string; thereby, providing a significantly higher measure of confidence whether the initial alert was "real" or just "noise". John On Mon, Oct 27, 2008 at 9:46 AM, Matt Jonkman wrote: > > Frank Knobbe wrote: >> No, that's a bad idea (at least if you talk about Snort). If you create >> new/different message texts, Snort will create a new entry in the >> signature table (unique to msg+gid+sid+rev). Also, you would not get the >> same text with barnyard or in barnyard (and probably flop) based >> installs since BY only reports the sid (the msg is pulled from the >> sid-msg.map file). > > We are not talking snort. This is totally different. > > And we'll definitely not use a db schema with this issue. > > Matt > >> >> While you could of course fork barnyard, my concern would be the bloat >> of the signature table due to unique msg texts. >> > > No forking here, all new. > > Everything from the pattern matcher on up. :) > > Matt > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From cunningpike at gmail.com Mon Oct 27 14:34:58 2008 From: cunningpike at gmail.com (CunningPike) Date: Mon, 27 Oct 2008 12:34:58 -0700 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <4901EDAD.9040606@jonkmans.com> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> <4901EDAD.9040606@jonkmans.com> Message-ID: <1225136098.15430.1.camel@arodgers-panasonic> I would like to see some aggregation of alert categorizations from sguil consoles. This could provide information on false positives, new spyware hosts etc. CP On Fri, 2008-10-24 at 11:45 -0400, Matt Jonkman wrote: > Martin Holste wrote: > > I like the idea, but are there really that many different actions to be > > taken, and aren't they going to be org specific? If I know that an IP > > is spamming, I don't just want to block them from emailing, I want to > > block all access from that IP since it is untrustworthy. > > I want the ip reputation data to be categorized. For instance the IP > would have a reputation number in spam, phishing, malware host, CnC, > known bot, scanner, etc. Maybe stuff to cover underground forums, whatever. > > There'd be an average score amassing all of the categories, and the end > user could weight each category to be more an influence on the average, > or even just make block decisions on a few categories they're interested > in. > > That make sense? So if I ran a mail farm I could weight the spam > category very high, or just look at that alone. Or if I were protecting > a net of users I'd probably take the average but weight the CnC and > Malware hosts higher. > > Matt > > > But I do think > > there is a lot of value in developing and distributing better language > > for describing why the given IP/host is now on the list and other > > descriptions. I'm more for giving orgs the most information that we > > can, and leaving it to them to implement the actual blocking decisions. > > > > --Martin > > > > On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein > > wrote: > > > > What if we focus on developing and distributing a better language for > > communicating actionable events? > > The idea is to make all intelligence more valuable and immediate. If I > > see this input event, alert, network, ISP, javascript, URL, how does it > > impact me, and what do I do about it? Instead of just collecting and > > distributing, the goal is to direct the actions for (ISP takedown, > > firewall, admin action, more). This enhances all of the prior research > > we've already done. > > > > > > Blake > > > > > > > > robert.jamison at bt.com wrote: > > > It seems we're a split camp with: > > > > > > [Keynesian CAMP] > > > Client Side Product/Service with ability to protect/detect > > compromise on > > > grannyx home user > > > *scope most thoroughly represented by Martin's " RFC: Proposal for > > > Analysis Framework" > > > > > > [Supply Side CAMP] > > > Focus on server side protection for net critical assets > > > *Andre/Jack "What is absolutely horrible in its current state is > > > IDS/IPS" / "Client side is simply not possible due to political and > > > religious issues." > > > > > > Additional notes gathered (I've just caught up on my reading;-) > > > > > > (a) Consideration for re-write defanging capability as inline > > protection > > > (b) Efficiency in stream storage--essentially normalize data > > inspection > > > so it doesn't have to be redone by multiple engines > > > (c) XML vs. Binary distribution of verbose alerts vs. instruction > > > inferred datapoints > > > (d) Consideration for extending existing project Bro > > > > > > Anything I'm missing? > > > > > > Rob > > > > > From jim.mcquaid at gmail.com Mon Oct 27 22:25:53 2008 From: jim.mcquaid at gmail.com (James McQuaid) Date: Mon, 27 Oct 2008 23:25:53 -0400 Subject: [Discussion] not a db schema Message-ID: I'm *still* listening... please continue. Message: 3 Frank Knobbe wrote: > No, that's a bad idea (at least if you talk about Snort). If you create > new/different message texts, Snort will create a new entry in the > signature table (unique to msg+gid+sid+rev). Also, you would not get the > same text with barnyard or in barnyard (and probably flop) based > installs since BY only reports the sid (the msg is pulled from the > sid-msg.map file). We are not talking snort. This is totally different. And we'll definitely not use a db schema with this issue. Matt > > While you could of course fork barnyard, my concern would be the bloat > of the signature table due to unique msg texts. > No forking here, all new. Everything from the pattern matcher on up. :) Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc -- James McQuaid http://www.jamesmcquaid.com From hall.692 at osu.edu Mon Oct 27 23:20:48 2008 From: hall.692 at osu.edu (Seth Hall) Date: Tue, 28 Oct 2008 00:20:48 -0400 Subject: [Discussion] Text in Msgs In-Reply-To: References: <4905DD78.5010701@jonkmans.com> <1225124209.2872.3.camel@localhost> <4905F065.5090405@jonkmans.com> Message-ID: <807E99B8-E40F-4D6A-BF35-F4A50571E948@osu.edu> On Oct 27, 2008, at 1:41 PM, John Pritchard wrote: > For example, the returned "match" could be sent on to additional tasks > that would either additionally validate the alarm or invoke some other > automated form of action. This is exactly how Bro works. There is a notice framework that basically allows you to write code where you can define how you'd like to handle a notice after it has already been thrown (e.g. email, don't email, run some shell command, etc.). > Take a signature designed to detect possible credit card patterns. > Once a potential pattern match is caught, the string could then be > passed to other processes to further validate or refute the trigger. > For a VISA credit card sig, the initial match might simply be on a 16 > digit number sequence beginning with a "four" (which would be highly > prone to false positives). But, once "matched" the string could be > passed to another process to run a Luhn checksum algorithm on the > string; thereby, providing a significantly higher measure of > confidence whether the initial alert was "real" or just "noise". Assuming that we already have something raising a notice named "CC_VISA_Number" and a function named "check_luhn", this would more-or- less look like this... redef notice_policy += { [$pred(n: notice_info) = { return (n$note == CC_VISA_Number && check_luhn(n$sub)); }, $result = NOTICE_EMAIL], }; This is saying that if the notice type being processed is CC_VISA_Number and the "sub" field (where I'm assuming the potential card number would be placed) passes the luhn check, send the notice in an email. From my own personal experience, there is really something to be said for being able to so much aggregation and correlation in a single language and with a single tool. For more information about the notice framework in Bro, you can refer to this post on the ICIR blog by Robin Sommer: http://blog.icir.org/2008/03/telling-bro-what-important.html .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From lists at inliniac.net Tue Oct 28 09:39:52 2008 From: lists at inliniac.net (Victor Julien) Date: Tue, 28 Oct 2008 15:39:52 +0100 Subject: [Discussion] Text in Msgs In-Reply-To: <1225127079.2872.30.camel@localhost> References: <4905DD78.5010701@jonkmans.com> <1225124209.2872.3.camel@localhost> <4905F065.5090405@jonkmans.com> <1225127079.2872.30.camel@localhost> Message-ID: <49072438.8000506@inliniac.net> Frank Knobbe wrote: > On Mon, 2008-10-27 at 12:46 -0400, Matt Jonkman wrote: > >> We are not talking snort. This is totally different. >> > > Okay, then I don't have objections :) > > While you are add it, here's a feature I meant to add to Snort years > ago: The ability to copy matched text into a variable used for further > matching. > > My idea to implement two new keywords, one to store data into a temp > variable, and one to match the temp variable against data that was valid > for the lifetime of the session (and only available for that session). > Too bad I don't remember the actual use case for it.... but it should > have looked something like this: > > content:"http\://"; store:7,2,' ',1; > > Different rule (in the same session) could then use: > content:"blah"; match:12,1; > > Parameters for store: > 1st: Offset from last match (content, isdataat, etc) > 2nd: Type of end match: 1: 3rd param is length > 2: 3rd para is ending char match > 3rd: End match: Either a length (copy 10 bytes) or an end match (in > above example, a space) > 4th: Number of session variable. > > Parameters for match: > 1st: Offset from previous match. > 2nd: Variable number to match. > > Above example would store the domain name after http:// (until reaching > a space) in variable 1. The second rule using match would check if the > data contained in variable 1 is present 12 bytes after blah. > > > I think one of the reasons it never got implemented (besides there being > workarounds for the specific issue in the past) was that it could easily > bloat the memory of session streams since any data matching the first > rule would alloc memory for the variable and tag it into the data > associated with the stream. It only would be removed if the stream data > is discarded. Considering there could be dozens/hundreds/thousand of > matches on the first rule, it would mean dozen/hundreds/thousand times > the length of the variable (plus some pointers) of memory overhead. > > > Anyway, since you working on something new, perhaps implementing the > ability to save data from sniffed traffic for later matching could be > This is quite similar to the stuff I wrote about last week. I think it can be very useful to do so. One other way to do so would be using pcre substring capturing. I'd like to extend your idea to being able to do the capture and matching per packet, flow/stream, host and globally. And be able to compare captured data with each other... Memory will be an issue, yes. But we'll see how efficient it can be done. Quality of rules/sigs will be a big factor there as well... Cheers, Victor > implemented more efficiently. > > Cheers, > Frank > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > From jonkman at jonkmans.com Tue Oct 28 11:54:57 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Tue, 28 Oct 2008 12:54:57 -0400 Subject: [Discussion] not a db schema In-Reply-To: References: Message-ID: <490743E1.2040309@jonkmans.com> I wonder if we ought to build several db schema's for different goals. I'm not a deep-down sql expert (we'll get one involved when we get here), but I suspect if your goal is different a different schema will do best. For instance, if you intend to keep every alert ever generated for historical comparison you need one schema to keep it snappy. If you intend to have a smaller number of alerts but a massive number of sensors inserting frequently then you may need a different structure. So perhaps we should build several schemas and let you choose at setup. Then the engine just reads the version tag and inserts in that form. Any sql guru's that could speak more to this? Matt James McQuaid wrote: > I'm *still* listening... please continue. > > Message: 3 > Frank Knobbe wrote: >> No, that's a bad idea (at least if you talk about Snort). If you create >> new/different message texts, Snort will create a new entry in the >> signature table (unique to msg+gid+sid+rev). Also, you would not get the >> same text with barnyard or in barnyard (and probably flop) based >> installs since BY only reports the sid (the msg is pulled from the >> sid-msg.map file). > > We are not talking snort. This is totally different. > > And we'll definitely not use a db schema with this issue. > > Matt > >> While you could of course fork barnyard, my concern would be the bloat >> of the signature table due to unique msg texts. >> > > No forking here, all new. > > Everything from the pattern matcher on up. :) > > Matt > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > > > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > > > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Tue Oct 28 12:42:56 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Tue, 28 Oct 2008 13:42:56 -0400 Subject: [Discussion] What are we making? -- CLIENT Side In-Reply-To: <1225136098.15430.1.camel@arodgers-panasonic> References: <48FA1839.14684.23E77B8@localhost> <48FA9C15.3040505@packetspy.com> <48FB64C1.4000803@jonkmans.com> <3839DD5BC9EE23459E6FAC536A2805370644E40A@E03MVY2-UKDY.domain1.systemhost.net> <48FFAAB9.7010801@gmail.com> <4901EDAD.9040606@jonkmans.com> <1225136098.15430.1.camel@arodgers-panasonic> Message-ID: <49074F20.9070404@jonkmans.com> I don't follow. Can you give an example? Matt CunningPike wrote: > I would like to see some aggregation of alert categorizations from sguil > consoles. This could provide information on false positives, new spyware > hosts etc. > > CP > > > On Fri, 2008-10-24 at 11:45 -0400, Matt Jonkman wrote: >> Martin Holste wrote: >>> I like the idea, but are there really that many different actions to be >>> taken, and aren't they going to be org specific? If I know that an IP >>> is spamming, I don't just want to block them from emailing, I want to >>> block all access from that IP since it is untrustworthy. >> I want the ip reputation data to be categorized. For instance the IP >> would have a reputation number in spam, phishing, malware host, CnC, >> known bot, scanner, etc. Maybe stuff to cover underground forums, whatever. >> >> There'd be an average score amassing all of the categories, and the end >> user could weight each category to be more an influence on the average, >> or even just make block decisions on a few categories they're interested >> in. >> >> That make sense? So if I ran a mail farm I could weight the spam >> category very high, or just look at that alone. Or if I were protecting >> a net of users I'd probably take the average but weight the CnC and >> Malware hosts higher. >> >> Matt >> >> >> But I do think >>> there is a lot of value in developing and distributing better language >>> for describing why the given IP/host is now on the list and other >>> descriptions. I'm more for giving orgs the most information that we >>> can, and leaving it to them to implement the actual blocking decisions. >>> >>> --Martin >>> >>> On Wed, Oct 22, 2008 at 5:35 PM, Blake Hartstein >> > wrote: >>> >>> What if we focus on developing and distributing a better language for >>> communicating actionable events? >>> The idea is to make all intelligence more valuable and immediate. If I >>> see this input event, alert, network, ISP, javascript, URL, how does it >>> impact me, and what do I do about it? Instead of just collecting and >>> distributing, the goal is to direct the actions for (ISP takedown, >>> firewall, admin action, more). This enhances all of the prior research >>> we've already done. >>> >>> >>> Blake >>> >>> >>> >>> robert.jamison at bt.com wrote: >>> > It seems we're a split camp with: >>> > >>> > [Keynesian CAMP] >>> > Client Side Product/Service with ability to protect/detect >>> compromise on >>> > grannyx home user >>> > *scope most thoroughly represented by Martin's " RFC: Proposal for >>> > Analysis Framework" >>> > >>> > [Supply Side CAMP] >>> > Focus on server side protection for net critical assets >>> > *Andre/Jack "What is absolutely horrible in its current state is >>> > IDS/IPS" / "Client side is simply not possible due to political and >>> > religious issues." >>> > >>> > Additional notes gathered (I've just caught up on my reading;-) >>> > >>> > (a) Consideration for re-write defanging capability as inline >>> protection >>> > (b) Efficiency in stream storage--essentially normalize data >>> inspection >>> > so it doesn't have to be redone by multiple engines >>> > (c) XML vs. Binary distribution of verbose alerts vs. instruction >>> > inferred datapoints >>> > (d) Consideration for extending existing project Bro >>> > >>> > Anything I'm missing? >>> > >>> > Rob >>> >>> > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From mcholste at gmail.com Tue Oct 28 20:41:17 2008 From: mcholste at gmail.com (Martin Holste) Date: Tue, 28 Oct 2008 20:41:17 -0500 Subject: [Discussion] not a db schema In-Reply-To: <490743E1.2040309@jonkmans.com> References: <490743E1.2040309@jonkmans.com> Message-ID: I think that schema discussions are extremely premature since we still haven't decided what we're even recording. --Martin On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman wrote: > I wonder if we ought to build several db schema's for different goals. > > I'm not a deep-down sql expert (we'll get one involved when we get > here), but I suspect if your goal is different a different schema will > do best. > > For instance, if you intend to keep every alert ever generated for > historical comparison you need one schema to keep it snappy. If you > intend to have a smaller number of alerts but a massive number of > sensors inserting frequently then you may need a different structure. > > So perhaps we should build several schemas and let you choose at setup. > Then the engine just reads the version tag and inserts in that form. > > Any sql guru's that could speak more to this? > > Matt > > James McQuaid wrote: > > I'm *still* listening... please continue. > > > > Message: 3 > > Frank Knobbe wrote: > >> No, that's a bad idea (at least if you talk about Snort). If you create > >> new/different message texts, Snort will create a new entry in the > >> signature table (unique to msg+gid+sid+rev). Also, you would not get the > >> same text with barnyard or in barnyard (and probably flop) based > >> installs since BY only reports the sid (the msg is pulled from the > >> sid-msg.map file). > > > > We are not talking snort. This is totally different. > > > > And we'll definitely not use a db schema with this issue. > > > > Matt > > > >> While you could of course fork barnyard, my concern would be the bloat > >> of the signature table due to unique msg texts. > >> > > > > No forking here, all new. > > > > Everything from the pattern matcher on up. :) > > > > Matt > > > > > > -- > > -------------------------------------------- > > Matthew Jonkman > > Emerging Threats > > Phone 765-429-0398 > > Fax 312-264-0205 > > http://www.emergingthreats.net > > > > > > -------------------------------------------- > > > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > > > > > > > > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081028/af3e10c8/attachment.html From jonkman at jonkmans.com Wed Oct 29 12:27:32 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Wed, 29 Oct 2008 13:27:32 -0400 Subject: [Discussion] not a db schema In-Reply-To: References: <490743E1.2040309@jonkmans.com> Message-ID: <49089D04.8090509@jonkmans.com> Agreed. I don't want to get deep into the schema itself. Just want to explore the idea of not being tied to one schema. A question for the sql guru's though: Is there a large enough difference in how you'd structure a scheme for large scale vs small/speedy to justify thinking about this? Matt Martin Holste wrote: > I think that schema discussions are extremely premature since we still > haven't decided what we're even recording. > > --Martin > > On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman > wrote: > > I wonder if we ought to build several db schema's for different goals. > > I'm not a deep-down sql expert (we'll get one involved when we get > here), but I suspect if your goal is different a different schema will > do best. > > For instance, if you intend to keep every alert ever generated for > historical comparison you need one schema to keep it snappy. If you > intend to have a smaller number of alerts but a massive number of > sensors inserting frequently then you may need a different structure. > > So perhaps we should build several schemas and let you choose at setup. > Then the engine just reads the version tag and inserts in that form. > > Any sql guru's that could speak more to this? > > Matt > > James McQuaid wrote: > > I'm *still* listening... please continue. > > > > Message: 3 > > Frank Knobbe wrote: > >> No, that's a bad idea (at least if you talk about Snort). If you > create > >> new/different message texts, Snort will create a new entry in the > >> signature table (unique to msg+gid+sid+rev). Also, you would not > get the > >> same text with barnyard or in barnyard (and probably flop) based > >> installs since BY only reports the sid (the msg is pulled from the > >> sid-msg.map file). > > > > We are not talking snort. This is totally different. > > > > And we'll definitely not use a db schema with this issue. > > > > Matt > > > >> While you could of course fork barnyard, my concern would be the > bloat > >> of the signature table due to unique msg texts. > >> > > > > No forking here, all new. > > > > Everything from the pattern matcher on up. :) > > > > Matt > > > > > > -- > > -------------------------------------------- > > Matthew Jonkman > > Emerging Threats > > Phone 765-429-0398 > > Fax 312-264-0205 > > http://www.emergingthreats.net > > > > > > -------------------------------------------- > > > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > > > > > > > > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From david at vorant.com Wed Oct 29 12:48:12 2008 From: david at vorant.com (David J. Bianco) Date: Wed, 29 Oct 2008 13:48:12 -0400 Subject: [Discussion] Text in Msgs In-Reply-To: <4905DD78.5010701@jonkmans.com> References: <4905DD78.5010701@jonkmans.com> Message-ID: <4908A1DC.6080606@vorant.com> Let me please be the first to download a file called %27%29%3B%20DROP%20DATABASE%20%27secdb%27%3B.exe from your network! David Matt Jonkman wrote: > Would anyone be interested in the ability to insert captured text into > the alert text of an event? > > For instance, I was just looking at a few hits on " ET POLICY exe > download via HTTP". It'd be nice for that to say: > > ET POLICY exe download via HTTP > (down.onlinedowns.net/page/image/yahoons.exe) > > Quick way to decide if that was something of interest or not without > having to dig into payload. > > What does everyone think? > > Matt > From david at vorant.com Wed Oct 29 13:13:53 2008 From: david at vorant.com (David J. Bianco) Date: Wed, 29 Oct 2008 14:13:53 -0400 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: <48F7E3B0.20002@jonkmans.com> References: <48F7E3B0.20002@jonkmans.com> Message-ID: <4908A7E1.4090308@vorant.com> I just now got caught up on all my OISF discussion list reading, so I'd like to make a few points. Some of you probably know that I'm affiliated with the Sguil project, so many of my comments spring from that background. We consider ourselves to have moved past IDS years ago, when we started to implement our NSM model. The additional network forensic capabilities built into Sguil (network session database, full content packet capture and passive network service detection) are really key to our ability to do good event analysis. In fact, from our point of view, the success of our monitoring program depends in large part upon our ability to : 1) Find new ways of detecting events 2) Get those detected events into the Sguil database 3) Use the built-in Sguil research tools to investigate them For the most part, getting things into Sguil (#2) is quite simple, so that really leaves us with #1 and #3. There was some discussion on this list about Bro. I'm a big fan of Bro. It performs very well in the pattern analysis and policy-based detection arenas, in which Snort (and thus, Sguil) do poorly. Conversely, Bro's signature matching abilities aren't very good, and it certainly doesn't have a robust sig library like Snort does. My view is that they are both complementary technologies, and should be deployed in tandem. It has been suggested on this list that Bro be the engine for future OISF implementation. Great idea! But I propose that we consider our project a framework, or suite of tools, and that Bro be a critical *piece* of that framework, but not the only framework. Further, we should strive to be "sensor-agnostic", accepting inputs from different types of detection engines and integrating them into a common analysis portal. There are many advantages to this approach, including the fact that it allows each site to choose the exact mix of detection engines that make most sense in their environment. I can envision a scenario where I'd deploy, for example, signature based IDS (Snort), policy based IDS (Bro), correlated event alerts (OSSEC) and perhaps even application specific sensors (mod_security, database IDS, spear phishing detection, etc). All these events should go to the same analysis tool, they should be organized effectively, searchable, cross-referenced, and tied to forensic data (network sessions, packet captures, log files, etc). In fact, we're much of the way there already. I released an agent to slurp OSSEC events into Sguil quite some time ago, and we have done the same with other data sources as well. Communication between Bro and Sguil is on my TODO list as well. So really, the idea of deploying multiple types of sensors feeding into the same analysis tool is a practical one. I think this should be a design goal of the project. David From david at vorant.com Wed Oct 29 13:39:25 2008 From: david at vorant.com (David J. Bianco) Date: Wed, 29 Oct 2008 14:39:25 -0400 Subject: [Discussion] Multi-tiered detection, and some feature suggestions Message-ID: <4908ADDD.3030103@vorant.com> In my organization, we're collecting lots of network forensic information to support our intrusion analysis process. In addition to IDS alerts, we routinely collect network session information, full packet captures, network asset info (OS and services) as well as a few other things. At first, we really did just use these things to answer questions when we were investigating IDS alerts, but over time we have turned these additional sources of information into detection tools in their own right. For example, our network session database is really good at detecting scans and sweeps. It can tell us when two hosts communicate that have never communicated before, or when a box starts talking on a new port. It can even tell us when someone's using BitTorrent or similar P2P applications, or when unusual amounts of data are being download to or exfiltrated from our network. We look for these things by running reports, so if our IDS can't catch them in real-time, we can still have a good chance of detecting them via offline analysis. Similarly, our pcap data is a really good source to turn to when we need to extract copies of transferred files. The thing I'm most excited about for purposes of this project, though, is that it allows us to look back in time and apply what we know *now* to what happened *then*. For example, if I hear of a new vulnerability, and I write a Snort rule to detect it, I often go through and run that rule retroactively against the stored traffic for the last few days or weeks. This is very effective in closing the window between vulnerability discovery and signature availability. Given these, I would like to suggest that a design goal of this new system we're working on would be to allow for both online & offline detection of security events. I would also like to suggest that as new signatures become known, the system should be able to retroactively check them against things which have gone before. In fact, I would like to go a step further. One thing that Sguil, Bro and pretty much every other analysis tools I've ever used fail at is helping analysts detect patterns that might link multiple attacks to the same source. Way back in February 2007, I wrote a blog post about how I'd like to see a Web 2.0-style tagging feature implemented in Sguil: http://blog.vorant.com/2007/02/ip-tagging.html I would like to put that here for discussion, but I'd also like to extend it to include the capability of an analyst to flag "indicators" that might tie an attacker's various attempts together. For example, two attacks that originate from the same IP might reasonably be considered related, even if they are separated by days, weeks or months. If we find two pieces of malware exfiltrating data to the same DNS name, it's a good bet they're under the same control. Email addresses, attachement names, URL patterns... there are probably a lot of different types of indicators I'm not even considering. I'd like to see a system that could keep track of indicators, tie them to specific "incidents", and then flag additional uses of these same indicators, either in future or historical occurrences, including situations where the historical events weren't flagged as security alerts at the time. Sound Crazy? Well, it's not. We're doing this manually now, and only for a few cases where we think it's worth the time. However, the fact that we *can* do it manually implies that we can probably do it automatically, and in my opinion, that's when things really start to get interesting! David From aludwig at packetspy.com Wed Oct 29 13:44:18 2008 From: aludwig at packetspy.com (Andre Ludwig) Date: Wed, 29 Oct 2008 14:44:18 -0400 Subject: [Discussion] not a db schema In-Reply-To: <49089D04.8090509@jonkmans.com> References: <490743E1.2040309@jonkmans.com> <49089D04.8090509@jonkmans.com> Message-ID: <4908AF02.2000701@packetspy.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 What is being discussed on this list in regards to technology and requirements for data rentetion and access just dont mesh well with current "standard RDBMS". I would suggest looking at distributed file systems and some of the databases that sit on top of them (hypertable, hbase, etc come to mind). Andre Ludwig Matt Jonkman wrote: > Agreed. I don't want to get deep into the schema itself. Just want to > explore the idea of not being tied to one schema. > > A question for the sql guru's though: Is there a large enough difference > in how you'd structure a scheme for large scale vs small/speedy to > justify thinking about this? > > Matt > > > Martin Holste wrote: > >> I think that schema discussions are extremely premature since we still >> haven't decided what we're even recording. >> >> --Martin >> >> On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman > > wrote: >> >> I wonder if we ought to build several db schema's for different goals. >> >> I'm not a deep-down sql expert (we'll get one involved when we get >> here), but I suspect if your goal is different a different schema will >> do best. >> >> For instance, if you intend to keep every alert ever generated for >> historical comparison you need one schema to keep it snappy. If you >> intend to have a smaller number of alerts but a massive number of >> sensors inserting frequently then you may need a different structure. >> >> So perhaps we should build several schemas and let you choose at setup. >> Then the engine just reads the version tag and inserts in that form. >> >> Any sql guru's that could speak more to this? >> >> Matt >> >> James McQuaid wrote: >> > I'm *still* listening... please continue. >> > >> > Message: 3 >> > Frank Knobbe wrote: >> >> No, that's a bad idea (at least if you talk about Snort). If you >> create >> >> new/different message texts, Snort will create a new entry in the >> >> signature table (unique to msg+gid+sid+rev). Also, you would not >> get the >> >> same text with barnyard or in barnyard (and probably flop) based >> >> installs since BY only reports the sid (the msg is pulled from the >> >> sid-msg.map file). >> > >> > We are not talking snort. This is totally different. >> > >> > And we'll definitely not use a db schema with this issue. >> > >> > Matt >> > >> >> While you could of course fork barnyard, my concern would be the >> bloat >> >> of the signature table due to unique msg texts. >> >> >> > >> > No forking here, all new. >> > >> > Everything from the pattern matcher on up. :) >> > >> > Matt >> > >> > >> > -- >> > -------------------------------------------- >> > Matthew Jonkman >> > Emerging Threats >> > Phone 765-429-0398 >> > Fax 312-264-0205 >> > http://www.emergingthreats.net >> > >> > >> > -------------------------------------------- >> > >> > PGP: http://www.jonkmans.com/mattjonkman.asc >> > >> > >> > >> > >> > >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> >> _______________________________________________ >> Discussion mailing list >> Discussion at openinfosecfoundation.org >> >> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> >> > > -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.9.0 (Build 397) Charset: ISO-8859-1 wsBVAwUBSQivBcjAfVnRK9hXAQjszAgAuo00m4GoBrroLxj6zcRjpEyiNP/S831u TWPwiF5YjEoDMpGDBlVqJiOowJ2LvO2bAJCPvR9dXnQPK7pOVgRy1ODt0cdeAV5a 18ALVe6P9SNCQXpt3Qnay8sSHC7fNcUxvLpyVwlRXm+2OnC/UKEjtGtBlrbXn0eO Aapm8i4IH53QysFuuWAEFq3tPH6T1XbrnAHEdXrUiC972gqFFy2G8i2UqxN92076 KX0OhdeWS7QPJOTXiBRBftK9ZLvLQ42zXUtW/I7/Cmh2+hMT6/ime+QblCwX3k9q 9+8YOLiVTG1/KRtXSJakZ/2f4h7dK55j7az4UoD3hdzbgzLUNUagZQ== =6IAU -----END PGP SIGNATURE----- From scheidell at secnap.net Wed Oct 29 13:52:34 2008 From: scheidell at secnap.net (Michael Scheidell) Date: Wed, 29 Oct 2008 14:52:34 -0400 Subject: [Discussion] not a db schema In-Reply-To: <4908AF02.2000701@packetspy.com> References: <490743E1.2040309@jonkmans.com> <49089D04.8090509@jonkmans.com> <4908AF02.2000701@packetspy.com> Message-ID: <4908B0F2.2030102@secnap.net> look at mynetwatchman.. it was an early attempt do do both. Andre Ludwig wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > What is being discussed on this list in regards to technology and > requirements for data rentetion and access just dont mesh well with > current "standard RDBMS". I would suggest looking at distributed file > systems and some of the databases that sit on top of them (hypertable, > hbase, etc come to mind). > > Andre Ludwig > > Matt Jonkman wrote: > >> Agreed. I don't want to get deep into the schema itself. Just want to >> explore the idea of not being tied to one schema. >> >> A question for the sql guru's though: Is there a large enough difference >> in how you'd structure a scheme for large scale vs small/speedy to >> justify thinking about this? >> >> Matt >> >> >> Martin Holste wrote: >> >> >>> I think that schema discussions are extremely premature since we still >>> haven't decided what we're even recording. >>> >>> --Martin >>> >>> On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman >> > wrote: >>> >>> I wonder if we ought to build several db schema's for different goals. >>> >>> I'm not a deep-down sql expert (we'll get one involved when we get >>> here), but I suspect if your goal is different a different schema will >>> do best. >>> >>> For instance, if you intend to keep every alert ever generated for >>> historical comparison you need one schema to keep it snappy. If you >>> intend to have a smaller number of alerts but a massive number of >>> sensors inserting frequently then you may need a different structure. >>> >>> So perhaps we should build several schemas and let you choose at setup. >>> Then the engine just reads the version tag and inserts in that form. >>> >>> Any sql guru's that could speak more to this? >>> >>> Matt >>> >>> James McQuaid wrote: >>> > I'm *still* listening... please continue. >>> > >>> > Message: 3 >>> > Frank Knobbe wrote: >>> >> No, that's a bad idea (at least if you talk about Snort). If you >>> create >>> >> new/different message texts, Snort will create a new entry in the >>> >> signature table (unique to msg+gid+sid+rev). Also, you would not >>> get the >>> >> same text with barnyard or in barnyard (and probably flop) based >>> >> installs since BY only reports the sid (the msg is pulled from the >>> >> sid-msg.map file). >>> > >>> > We are not talking snort. This is totally different. >>> > >>> > And we'll definitely not use a db schema with this issue. >>> > >>> > Matt >>> > >>> >> While you could of course fork barnyard, my concern would be the >>> bloat >>> >> of the signature table due to unique msg texts. >>> >> >>> > >>> > No forking here, all new. >>> > >>> > Everything from the pattern matcher on up. :) >>> > >>> > Matt >>> > >>> > >>> > -- >>> > -------------------------------------------- >>> > Matthew Jonkman >>> > Emerging Threats >>> > Phone 765-429-0398 >>> > Fax 312-264-0205 >>> > http://www.emergingthreats.net >>> > >>> > >>> > -------------------------------------------- >>> > >>> > PGP: http://www.jonkmans.com/mattjonkman.asc >>> > >>> > >>> > >>> > >>> > >>> >>> -- >>> -------------------------------------------- >>> Matthew Jonkman >>> Emerging Threats >>> Phone 765-429-0398 >>> Fax 312-264-0205 >>> http://www.emergingthreats.net >>> -------------------------------------------- >>> >>> PGP: http://www.jonkmans.com/mattjonkman.asc >>> >>> >>> _______________________________________________ >>> Discussion mailing list >>> Discussion at openinfosecfoundation.org >>> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >>> >>> >>> >>> >> >> > > > -----BEGIN PGP SIGNATURE----- > Version: PGP Desktop 9.9.0 (Build 397) > Charset: ISO-8859-1 > > wsBVAwUBSQivBcjAfVnRK9hXAQjszAgAuo00m4GoBrroLxj6zcRjpEyiNP/S831u > TWPwiF5YjEoDMpGDBlVqJiOowJ2LvO2bAJCPvR9dXnQPK7pOVgRy1ODt0cdeAV5a > 18ALVe6P9SNCQXpt3Qnay8sSHC7fNcUxvLpyVwlRXm+2OnC/UKEjtGtBlrbXn0eO > Aapm8i4IH53QysFuuWAEFq3tPH6T1XbrnAHEdXrUiC972gqFFy2G8i2UqxN92076 > KX0OhdeWS7QPJOTXiBRBftK9ZLvLQ42zXUtW/I7/Cmh2+hMT6/ime+QblCwX3k9q > 9+8YOLiVTG1/KRtXSJakZ/2f4h7dK55j7az4UoD3hdzbgzLUNUagZQ== > =6IAU > -----END PGP SIGNATURE----- > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- Michael Scheidell, CTO Phone: 561-999-5000, x 1259 > *| *SECNAP Network Security Corporation * Certified SNORT Integrator * King of Spam Filters, SC Magazine 2008 * Information Security Award 2008, Info Security Products Guide * CRN Magazine Top 40 Emerging Security Vendors _________________________________________________________________________ This email has been scanned and certified safe by SpammerTrap(r). For Information please see http://www.spammertrap.com _________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081029/d3cd66c2/attachment-0001.html From hall.692 at osu.edu Wed Oct 29 14:55:29 2008 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 29 Oct 2008 15:55:29 -0400 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: <4908A7E1.4090308@vorant.com> References: <48F7E3B0.20002@jonkmans.com> <4908A7E1.4090308@vorant.com> Message-ID: On Oct 29, 2008, at 2:13 PM, David J. Bianco wrote: > There was some discussion on this list about Bro. I'm a big fan of > Bro. > It performs very well in the pattern analysis and policy-based > detection > arenas, in which Snort (and thus, Sguil) do poorly. Conversely, Bro's > signature matching abilities aren't very good, and it certainly > doesn't have > a robust sig library like Snort does. Bro's signature matching is only poor because it isn't used much by anyone that uses Bro heavily. I don't believe that it would be too much of a project to flesh out the signature matching capabilities of Bro to mirror Snort. One area where Bro's signature matching already bests Snort is in its ability to have bidirectional signatures. You can define a signature that only triggers if another signature is seen in the opposite direction of the connection. For example, if you're matching something that requests a certain URL, maybe you only care about it the server responds to the client with a 200 status code. This is easy to do with bidirectional signatures. You can also have signature matches thrown into Bro policy script code so that you can do any number of extended heuristics based on other information Bro has detected. > My view is that they are both complementary technologies, and should > be > deployed in tandem. It has been suggested on this list that Bro be > the > engine for future OISF implementation. Great idea! But I propose > that > we consider our project a framework, or suite of tools, and that Bro > be > a critical *piece* of that framework, but not the only framework. In my opinion, Bro would actually make a very general framework to build upon. Bro is already more or less a general network analysis framework. It has support for lots of various technologies that people want in order to do network traffic analysis and typically it's fairly easy to add new functionality building on top of existing functionality. As an example of this, yesterday I wrote some small extensions that will identify windows executables being transferred over HTTP with libmagic and once a windows executable is identified it will begin building an MD5 hash of the data. It then takes the MD5 hash and checks it with the newly announced Team Cymru Malware Hash Repository[1] in realtime to see if they have that binary file in their malware repository. If you would like to see how little code I had to write to make it work, here's the link to my changes: http://github.com/sethhall/bro_scripts/tree/master/md5_hash_malware Malware Hash Registry docs: http://www.team-cymru.org/Services/MHR/ My point is just that Bro is very much in a position to be a central framework. Much more so than any other tools currently available. > I can envision a scenario where I'd deploy, for example, signature > based > IDS (Snort), policy based IDS (Bro), correlated event alerts (OSSEC) > and > perhaps even application specific sensors (mod_security, database IDS, > spear phishing detection, etc). The problem is that you're replicating effort there. You'd have to maintain multiple machines just to run different tools (assuming you're looking at too much traffic to analyze on a single machine with multiple tools). As I mentioned above, the problems with Bro's signature support are not due to a major failing in the software it's just temporarily suffering due to a lack of interest from the Bro community. :) It probably makes sense to feed the data from other tools like OSSEC into Bro first too because it has very good support for writing scripts that deal with data expiration over time and catching time based events. We use it to look for anomalies and normalize our authentication logs by pushing a lot of them into Bro with the Broccoli library (it's included with Bro). I know of another location that was using instrumented sshd servers to forward the content of all shell sessions to their Bro instance and looking for anomalies with the commands being typed over ssh to catch compromised accounts. After Bro has a chance to look at the data, it can then pass normalized and aggregated data (and maybe more notices if it caught anything strange in the data) into Squil and its database. > All these events should go to the same > analysis tool, they should be organized effectively, searchable, > cross-referenced, and tied to forensic data (network sessions, packet > captures, log files, etc). I definitely agree with this! Bro doesn't have anything yet that is end user focused towards doing forensics with the data being generated. If Squil could give it that frontend, that would be awesome. > In fact, we're much of the way there already. I released an agent to > slurp OSSEC events into Sguil quite some time ago, and we have done > the > same with other data sources as well. Communication between Bro and > Sguil > is on my TODO list as well. So really, the idea of deploying multiple > types of sensors feeding into the same analysis tool is a practical > one. > I think this should be a design goal of the project. If you'd like assistance with integrating Squil and Bro, I'd be glad to give some pointers or lend a hand with it. :) .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From jim.mcquaid at gmail.com Wed Oct 29 18:37:54 2008 From: jim.mcquaid at gmail.com (James McQuaid) Date: Wed, 29 Oct 2008 19:37:54 -0400 Subject: [Discussion] db schema Message-ID: My MySQL experience is relatively recent and rather limited, having used SQL Server and Oracle in employment, so I'm no guru. "Is there a large enough difference in how you'd structure a scheme for large scale vs small/speedy to justify thinking about this?" Yes. MySQL's most important feature is its storage-engine architecture, which separates query processing and other server tasks from data storage and retrieval. MySQL 6.0 supports 13 different runtime plug-in storage engines. This allows one to determine on a table by table basis how the data is stored and what performance characteristics are sought. Optimizing schema "can improve performance by orders of magnitude". For high performance, schema and indexes must be designed for specific queries. Performance requirements must vary for different kinds of queries. So, we should allow a user to select one of several sets of schema combinations. These sets of schema combinations might be based upon environmental criteria such as hardware, home user vs. enterprise, etc. Andre's point is very well taken with respect to requirements for data retention and access: distributed file systems may be a necessity, rather than an option, for regulated environments. Jim > Agreed. I don't want to get deep into the schema itself. Just want to > explore the idea of not being tied to one schema. > > A question for the sql guru's though: Is there a large enough difference > in how you'd structure a scheme for large scale vs small/speedy to > justify thinking about this? > > Matt >> On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman > > wrote: >> >> I wonder if we ought to build several db schema's for different goals. >> >> I'm not a deep-down sql expert (we'll get one involved when we get >> here), but I suspect if your goal is different a different schema will >> do best. >> >> For instance, if you intend to keep every alert ever generated for >> historical comparison you need one schema to keep it snappy. If you >> intend to have a smaller number of alerts but a massive number of >> sensors inserting frequently then you may need a different structure. >> >> So perhaps we should build several schemas and let you choose at setup. >> Then the engine just reads the version tag and inserts in that form. >> >> Any sql guru's that could speak more to this? > What is being discussed on this list in regards to technology and > requirements for data rentetion and access just dont mesh well with > current "standard RDBMS". I would suggest looking at distributed file > systems and some of the databases that sit on top of them (hypertable, > hbase, etc come to mind). > > Andre Ludwig >>> I wonder if we ought to build several db schema's for different goals. >>> >>> I'm not a deep-down sql expert (we'll get one involved when we get >>> here), but I suspect if your goal is different a different schema will >>> do best. >>> >>> For instance, if you intend to keep every alert ever generated for >>> historical comparison you need one schema to keep it snappy. If you >>> intend to have a smaller number of alerts but a massive number of >>> sensors inserting frequently then you may need a different structure. >>> >>> So perhaps we should build several schemas and let you choose at setup. >>> Then the engine just reads the version tag and inserts in that form. >>> >>> Any sql guru's that could speak more to this? >>> >>> Matt -- James McQuaid http://www.jamesmcquaid.com From mcholste at gmail.com Wed Oct 29 20:43:24 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 29 Oct 2008 20:43:24 -0500 Subject: [Discussion] not a db schema In-Reply-To: <4908B0F2.2030102@secnap.net> References: <490743E1.2040309@jonkmans.com> <49089D04.8090509@jonkmans.com> <4908AF02.2000701@packetspy.com> <4908B0F2.2030102@secnap.net> Message-ID: Matt: As long as the schema obeys (or approaches obeying) basic third normal form, then scalability will not be a major issue. The Snort schema is a decent example of this. I think that the key difference when talking big db/small db is in making sure that you can join the tables together to get the larger picture when necessary, but keeping the individual tables with as few columns as possible so that you can query them independently very efficiently. A good general rule would be that if we make sure that there's always a path to join tables together (maybe even through an intermediate table), then it makes the schema very extensible. This is probably the most important quality to me in a potential schema. Andre: Hypertable looks pretty cool, but I think it would be overkill for a lot of situations. A tiny amount of app-level coding can allow for very conventional databases to do some amazing things. Breaking one big query into two small and quick queries can sometimes make a huge difference. There are also a lot of arithemetical tricks to play in network-based queries. For instance, I keep the entire MaxMind GeoIP database in MySQL and do 1:1 joins on it against flows. This means I can efficiently sort my traffic by country, which is very cool. I do this by storing the country table not as individual IP address, but by class C address (1.1.1.0). Then, when I query through the traffic, I do this: JOIN geoip ON flows.srcip-MOD(flows.srcip, 256)=geoip.subnet. I can sort a 50 million row flows table by country in under a minute that way because the join is an "eq ref." --Martin On Wed, Oct 29, 2008 at 1:52 PM, Michael Scheidell wrote: > look at mynetwatchman.. it was an early attempt do do both. > > > > Andre Ludwig wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > What is being discussed on this list in regards to technology and > requirements for data rentetion and access just dont mesh well with > current "standard RDBMS". I would suggest looking at distributed file > systems and some of the databases that sit on top of them (hypertable, > hbase, etc come to mind). > > Andre Ludwig > > Matt Jonkman wrote: > > > Agreed. I don't want to get deep into the schema itself. Just want to > explore the idea of not being tied to one schema. > > A question for the sql guru's though: Is there a large enough difference > in how you'd structure a scheme for large scale vs small/speedy to > justify thinking about this? > > Matt > > > Martin Holste wrote: > > > > I think that schema discussions are extremely premature since we still > haven't decided what we're even recording. > > --Martin > > On Tue, Oct 28, 2008 at 11:54 AM, Matt Jonkman > wrote: > > I wonder if we ought to build several db schema's for different goals. > > I'm not a deep-down sql expert (we'll get one involved when we get > here), but I suspect if your goal is different a different schema will > do best. > > For instance, if you intend to keep every alert ever generated for > historical comparison you need one schema to keep it snappy. If you > intend to have a smaller number of alerts but a massive number of > sensors inserting frequently then you may need a different structure. > > So perhaps we should build several schemas and let you choose at setup. > Then the engine just reads the version tag and inserts in that form. > > Any sql guru's that could speak more to this? > > Matt > > James McQuaid wrote: > > I'm *still* listening... please continue. > > > > Message: 3 > > Frank Knobbe wrote: > >> No, that's a bad idea (at least if you talk about Snort). If you > create > >> new/different message texts, Snort will create a new entry in the > >> signature table (unique to msg+gid+sid+rev). Also, you would not > get the > >> same text with barnyard or in barnyard (and probably flop) based > >> installs since BY only reports the sid (the msg is pulled from the > >> sid-msg.map file). > > > > We are not talking snort. This is totally different. > > > > And we'll definitely not use a db schema with this issue. > > > > Matt > > > >> While you could of course fork barnyard, my concern would be the > bloat > >> of the signature table due to unique msg texts. > >> > > > > No forking here, all new. > > > > Everything from the pattern matcher on up. :) > > > > Matt > > > > > > -- > > -------------------------------------------- > > Matthew Jonkman > > Emerging Threats > > Phone 765-429-0398 > > Fax 312-264-0205 > > http://www.emergingthreats.net > > > > > > -------------------------------------------- > > > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > > > > > > > > > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > -----BEGIN PGP SIGNATURE----- > Version: PGP Desktop 9.9.0 (Build 397) > Charset: ISO-8859-1 > > wsBVAwUBSQivBcjAfVnRK9hXAQjszAgAuo00m4GoBrroLxj6zcRjpEyiNP/S831u > TWPwiF5YjEoDMpGDBlVqJiOowJ2LvO2bAJCPvR9dXnQPK7pOVgRy1ODt0cdeAV5a > 18ALVe6P9SNCQXpt3Qnay8sSHC7fNcUxvLpyVwlRXm+2OnC/UKEjtGtBlrbXn0eO > Aapm8i4IH53QysFuuWAEFq3tPH6T1XbrnAHEdXrUiC972gqFFy2G8i2UqxN92076 > KX0OhdeWS7QPJOTXiBRBftK9ZLvLQ42zXUtW/I7/Cmh2+hMT6/ime+QblCwX3k9q > 9+8YOLiVTG1/KRtXSJakZ/2f4h7dK55j7az4UoD3hdzbgzLUNUagZQ== > =6IAU > -----END PGP SIGNATURE----- > _______________________________________________ > Discussion mailing listDiscussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > -- > Michael Scheidell, CTO > Phone: 561-999-5000, x 1259 > > *| *SECNAP Network Security Corporation > > - Certified SNORT Integrator > - King of Spam Filters, SC Magazine 2008 > - Information Security Award 2008, Info Security Products Guide > - CRN Magazine Top 40 Emerging Security Vendors > > > ------------------------------ > > This email has been scanned and certified safe by SpammerTrap(R). > For Information please see www.spammertrap.com > ------------------------------ > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081029/8b6fd77c/attachment-0001.html From mcholste at gmail.com Wed Oct 29 22:06:45 2008 From: mcholste at gmail.com (Martin Holste) Date: Wed, 29 Oct 2008 22:06:45 -0500 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <4908A7E1.4090308@vorant.com> Message-ID: I find both Sguil and Bro to be venerable platforms, however Seth, I think that Bro is better left as a sniffer/preprocessor. I think that the bulk of the processing (I would even argue things like the libmagic part) are better left to another process that Bro ships interesting things to via Broccoli. While Bro sports a lot of really good plugins and a fair amount of extensibility, I think it's the wrong design path to throw everything into it as a framework. It is no accident that it lacks native database support and has no frontend, because it wasn't designed to be all-encompassing, it was designed to do the heavy lifting and pass the rest off via Broccoli. But Bro does have the great strength of allowing for extremely rich signature scripts, as you pointed out. David, (I am a regular reader of your blog, BTW): I consider Sguil the closest thing to an out-of-the-box solution for NSM out there, and I think that it is 100% on the right track. In my opinion, though, the entire Sguil architecture feels archaic and a bit kludgy, and I definitely require a web console as opposed to a Tcl-based console. I completely understand why Tcl was the original design choice, but AJAX has really made it obsolete. My vote would be to extend the NSM ideas of Sguil to a platform based on a language with a wider audience, like Perl. Admittedly, since I know Perl the best, I will of course advocate for it (just as you advocated Sguil and Seth advocated Bro). That said, you can take a Perl class (or Java, .NET, or probably even Python) at any tech college, but very few places will teach Tcl. Perl also has a much richer library of modules to draw on to do all of the really interesting things that you talked about in your last post. As an example, as of this morning, all packed files downloaded in my org are extracted from the network and sent to VirusTotal, Symantec, and now CWSandbox, and the results are collected and stored in a local database for data mining purposes. In any case, my goal is not to start a flame war over which programming language is best, but many vendors (like VMWare, HP, IBM, Cisco) chose to use Perl as the language in which to publish modules because of its ubiquity and extensibility. I think that most of our work will be analytical in nature, and this work is best done away from the packet sniffing/recording environment in a framework based on a dynamic language. I furthermore think that the best-suited dynamic language is Perl (maybe even Perl 6's Parrot platform). All of this is of course debatable, and I welcome rebuttals. --Martin On Wed, Oct 29, 2008 at 2:55 PM, Seth Hall wrote: > > On Oct 29, 2008, at 2:13 PM, David J. Bianco wrote: > > > There was some discussion on this list about Bro. I'm a big fan of > > Bro. > > It performs very well in the pattern analysis and policy-based > > detection > > arenas, in which Snort (and thus, Sguil) do poorly. Conversely, Bro's > > signature matching abilities aren't very good, and it certainly > > doesn't have > > a robust sig library like Snort does. > > Bro's signature matching is only poor because it isn't used much by > anyone that uses Bro heavily. I don't believe that it would be too > much of a project to flesh out the signature matching capabilities of > Bro to mirror Snort. One area where Bro's signature matching already > bests Snort is in its ability to have bidirectional signatures. You > can define a signature that only triggers if another signature is seen > in the opposite direction of the connection. For example, if you're > matching something that requests a certain URL, maybe you only care > about it the server responds to the client with a 200 status code. > This is easy to do with bidirectional signatures. You can also have > signature matches thrown into Bro policy script code so that you can > do any number of extended heuristics based on other information Bro > has detected. > > > My view is that they are both complementary technologies, and should > > be > > deployed in tandem. It has been suggested on this list that Bro be > > the > > engine for future OISF implementation. Great idea! But I propose > > that > > we consider our project a framework, or suite of tools, and that Bro > > be > > a critical *piece* of that framework, but not the only framework. > > In my opinion, Bro would actually make a very general framework to > build upon. Bro is already more or less a general network analysis > framework. It has support for lots of various technologies that > people want in order to do network traffic analysis and typically it's > fairly easy to add new functionality building on top of existing > functionality. As an example of this, yesterday I wrote some small > extensions that will identify windows executables being transferred > over HTTP with libmagic and once a windows executable is identified it > will begin building an MD5 hash of the data. It then takes the MD5 > hash and checks it with the newly announced Team Cymru Malware Hash > Repository[1] in realtime to see if they have that binary file in > their malware repository. If you would like to see how little code I > had to write to make it work, here's the link to my changes: > http://github.com/sethhall/bro_scripts/tree/master/md5_hash_malware > > Malware Hash Registry docs: > http://www.team-cymru.org/Services/MHR/ > > My point is just that Bro is very much in a position to be a central > framework. Much more so than any other tools currently available. > > > I can envision a scenario where I'd deploy, for example, signature > > based > > IDS (Snort), policy based IDS (Bro), correlated event alerts (OSSEC) > > and > > perhaps even application specific sensors (mod_security, database IDS, > > spear phishing detection, etc). > > The problem is that you're replicating effort there. You'd have to > maintain multiple machines just to run different tools (assuming > you're looking at too much traffic to analyze on a single machine with > multiple tools). As I mentioned above, the problems with Bro's > signature support are not due to a major failing in the software it's > just temporarily suffering due to a lack of interest from the Bro > community. :) > > It probably makes sense to feed the data from other tools like OSSEC > into Bro first too because it has very good support for writing > scripts that deal with data expiration over time and catching time > based events. We use it to look for anomalies and normalize our > authentication logs by pushing a lot of them into Bro with the > Broccoli library (it's included with Bro). I know of another location > that was using instrumented sshd servers to forward the content of all > shell sessions to their Bro instance and looking for anomalies with > the commands being typed over ssh to catch compromised accounts. > > After Bro has a chance to look at the data, it can then pass > normalized and aggregated data (and maybe more notices if it caught > anything strange in the data) into Squil and its database. > > > All these events should go to the same > > analysis tool, they should be organized effectively, searchable, > > cross-referenced, and tied to forensic data (network sessions, packet > > captures, log files, etc). > > I definitely agree with this! Bro doesn't have anything yet that is > end user focused towards doing forensics with the data being > generated. If Squil could give it that frontend, that would be awesome. > > > In fact, we're much of the way there already. I released an agent to > > slurp OSSEC events into Sguil quite some time ago, and we have done > > the > > same with other data sources as well. Communication between Bro and > > Sguil > > is on my TODO list as well. So really, the idea of deploying multiple > > types of sensors feeding into the same analysis tool is a practical > > one. > > I think this should be a design goal of the project. > > If you'd like assistance with integrating Squil and Bro, I'd be glad > to give some pointers or lend a hand with it. :) > > .Seth > > --- > Seth Hall > Network Security - Office of the CIO > The Ohio State University > Phone: 614-292-9721 > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081029/f0c0b223/attachment.html From david at vorant.com Thu Oct 30 08:01:17 2008 From: david at vorant.com (David J. Bianco) Date: Thu, 30 Oct 2008 09:01:17 -0400 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <4908A7E1.4090308@vorant.com> Message-ID: <4909B01D.5060002@vorant.com> Seth Hall wrote: > > Bro's signature matching is only poor because it isn't used much by > anyone that uses Bro heavily. I don't believe that it would be too much > of a project to flesh out the signature matching capabilities of Bro to > mirror Snort. I'm sure it's possible to correct. I wouldn't underestimate the difficulty, though. Snort really has a lot going for it already, including the functional equivalent of the bidirectional matching (at least, if I understand Bro's capability correctly). Snort just uses the flowbits construct for that. > > My point is just that Bro is very much in a position to be a central > framework. Much more so than any other tools currently available. > Believe me, I'm not dissing Bro at all. You're right that it does a great job as both a suite of detectors and a policy engine. And I think I may have given the mistaken impression that I'm advocating that we use Sguil for this. In fact, I don't care what we use as the framework basis, or if we invent something new. In fact, I really want to see the good parts of both Bro and Sguil, merged into a unified framework, and let the end users figure out which detectors they need. >> All these events should go to the same >> analysis tool, they should be organized effectively, searchable, >> cross-referenced, and tied to forensic data (network sessions, packet >> captures, log files, etc). > > I definitely agree with this! Bro doesn't have anything yet that is end > user focused towards doing forensics with the data being generated. If > Squil could give it that frontend, that would be awesome. Yeah, I'm pretty hopeful about this. If I can get them working together properly, the combination could be very powerful. "Join me, and together we can rule the galaxy!" David From david at vorant.com Thu Oct 30 08:17:20 2008 From: david at vorant.com (David J. Bianco) Date: Thu, 30 Oct 2008 09:17:20 -0400 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: References: <48F7E3B0.20002@jonkmans.com> <4908A7E1.4090308@vorant.com> Message-ID: <4909B3E0.2090506@vorant.com> Martin Holste wrote: > I find both Sguil and Bro to be venerable platforms, however Seth, I > think that Bro is better left as a sniffer/preprocessor. I think that > the bulk of the processing (I would even argue things like the libmagic > part) are better left to another process that Bro ships interesting > things to via Broccoli. While Bro sports a lot of really good plugins > and a fair amount of extensibility, I think it's the wrong design path > to throw everything into it as a framework. It is no accident that it > lacks native database support and has no frontend, because it wasn't > designed to be all-encompassing, it was designed to do the heavy lifting > and pass the rest off via Broccoli. But Bro does have the great > strength of allowing for extremely rich signature scripts, as you > pointed out. I think that this point of view isn't really very far off from what Seth and I have been discussing (if I understand you correctly, Seth). I think we all agree Bro has a better policy engine, and some great non-signature based detection mechanisms. And I think we also agree that Sguil has a better front end and better analysis support. A combination of these two strengths would be a formidable new tool. > > David, (I am a regular reader of your blog, BTW): I have a reader! Yay! > I consider Sguil the > closest thing to an out-of-the-box solution for NSM out there, and I > think that it is 100% on the right track. In my opinion, though, the > entire Sguil architecture feels archaic and a bit kludgy, and I > definitely require a web console as opposed to a Tcl-based console. It's definitely a complicated beast, and yeah, there are better ways to do a lot of what it does. There has been some recent discussion on how to change a lot of this (see our mailing list). Richard Bejtlich has even announced that he's going to be underwriting some major updates. All this is basically because we agree with your statement above, even if we didn't put it in those words exactly. David From jonkman at jonkmans.com Thu Oct 30 08:26:39 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Thu, 30 Oct 2008 09:26:39 -0400 Subject: [Discussion] Virtues of a Multi-Sensor Environment In-Reply-To: <4909B3E0.2090506@vorant.com> References: <48F7E3B0.20002@jonkmans.com> <4908A7E1.4090308@vorant.com> <4909B3E0.2090506@vorant.com> Message-ID: <4909B60F.8070702@jonkmans.com> I also agree that Bro is a great tool, and does some of what we want to do. Unfortunately I think we may get to a point where it can't do some of the things that make our final feature list. The extreme multi-threading may be one of those. But there are several others that will be a challenge to Bro. And Bro already has it's own roadmap and goals to hit. Its definitely not out of the question that we look to start with Bro, but if we pursue some of our major goals I don't think that'll be the best option. Interoperability or data sharing with Bro may be a more feasible option. Thoughts? Matt David J. Bianco wrote: > Martin Holste wrote: >> I find both Sguil and Bro to be venerable platforms, however Seth, I >> think that Bro is better left as a sniffer/preprocessor. I think that >> the bulk of the processing (I would even argue things like the libmagic >> part) are better left to another process that Bro ships interesting >> things to via Broccoli. While Bro sports a lot of really good plugins >> and a fair amount of extensibility, I think it's the wrong design path >> to throw everything into it as a framework. It is no accident that it >> lacks native database support and has no frontend, because it wasn't >> designed to be all-encompassing, it was designed to do the heavy lifting >> and pass the rest off via Broccoli. But Bro does have the great >> strength of allowing for extremely rich signature scripts, as you >> pointed out. > > I think that this point of view isn't really very far off from what Seth > and I have been discussing (if I understand you correctly, Seth). I think > we all agree Bro has a better policy engine, and some great non-signature > based detection mechanisms. And I think we also agree that Sguil has a > better front end and better analysis support. A combination of these > two strengths would be a formidable new tool. > >> David, (I am a regular reader of your blog, BTW): > > I have a reader! Yay! > >> I consider Sguil the >> closest thing to an out-of-the-box solution for NSM out there, and I >> think that it is 100% on the right track. In my opinion, though, the >> entire Sguil architecture feels archaic and a bit kludgy, and I >> definitely require a web console as opposed to a Tcl-based console. > > It's definitely a complicated beast, and yeah, there are better ways to do > a lot of what it does. There has been some recent discussion on how to > change a lot of this (see our mailing list). Richard Bejtlich has even > announced that he's going to be underwriting some major updates. All this > is basically because we agree with your statement above, even if we didn't > put it in those words exactly. > > David > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From taosecurity at gmail.com Thu Oct 30 11:35:26 2008 From: taosecurity at gmail.com (Richard Bejtlich) Date: Thu, 30 Oct 2008 12:35:26 -0400 Subject: [Discussion] Multi-tiered detection, and some feature suggestions In-Reply-To: <4908ADDD.3030103@vorant.com> References: <4908ADDD.3030103@vorant.com> Message-ID: <120ef0530810300935j34ae19b3s5bb03623b0d2da32@mail.gmail.com> On Wed, Oct 29, 2008 at 2:39 PM, David J. Bianco wrote: > In my organization, we're collecting lots of network forensic information to > support our intrusion analysis process. ... > I would like to put that here for discussion, but I'd also like to extend it > to include the capability of an analyst to flag "indicators" that might > tie an attacker's various attempts together. For example, two attacks > that originate from the same IP might reasonably be considered related, > even if they are separated by days, weeks or months. If we find two pieces > of malware exfiltrating data to the same DNS name, it's a good bet they're > under the same control. Email addresses, attachement names, URL patterns... > there are probably a lot of different types of indicators I'm not even > considering. > > I'd like to see a system that could keep track of indicators, tie them to > specific "incidents", and then flag additional uses of these same indicators, > either in future or historical occurrences, including situations where the > historical events weren't flagged as security alerts at the time. > David and everyone, I think Splunk can serve this function: http://marc.info/?l=sguil-users&m=122468353412622&w=2 However, I realize Splunk is not free. SplunkBase applications are, since they are released through the Creative Commons license. Sincerely, Richard