From victor at inliniac.net Wed Feb 1 03:32:28 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 01 Feb 2012 09:32:28 +0100 Subject: [Oisf-users] PCRE JIT compiler does not support ... In-Reply-To: <1327991891.2955.0.camel@sknaumov.altell.local> References: <1327991891.2955.0.camel@sknaumov.altell.local> Message-ID: <4F28F89C.4070604@inliniac.net> On 01/31/2012 07:38 AM, Sergey Naumov wrote: > Hello. > > I would like to ask whether rules with "PCRE JIT compiler does not > support ..." warning will be processed as if they weren't precompiled > (so as without PCRE enabled)? > Log level of these messages in syslog are "warning", not "notice" or > "info", therefore it pushes me to ask this question. The rules will be processed normally, only without the PCRE JIT optimization. If possible, please share the rules. We can pass on the regex' to the PCRE JIT authors to see if they can add the support. Btw, make sure to use the latests pcre version, 8.21 currently. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From jonkman at emergingthreatspro.com Mon Feb 6 11:14:23 2012 From: jonkman at emergingthreatspro.com (Matthew Jonkman) Date: Mon, 6 Feb 2012 11:14:23 -0500 Subject: [Oisf-users] Suricata Brainstorming Session Tomorrow! Message-ID: <2AF275BB-E321-4AFB-9B54-042452468AB3@emergingthreatspro.com> Don't forget to mark your calendar to attend either in person or remotely the next Suricata Brainstorming Session! http://www.openinfosecfoundation.org/index.php/component/content/article/1-latest-news/146-suricata-brainstorming-session-feb-7-2012 1:00pm CET (GMT+1), or 7am EST. Don't worry if that's early where you are, join when you can, it'll be informal. IT-Defense Conference 2012, Munich (http://it-defense.de) February 7th, 2012 Leonardo Royal Hotel, Munich www.leonardo-hotels.com The primary goal of this Brainstorming Session is to review and adjust the Suricata Development Roadmap. To do this we will outline the current complete features and development status, proposed features from public and private sources, and seek input on these items. This is an open discussion. Let us know what you?d like your IDS/IPS engine to do! Remote attendance will be available. A link will be posted on the OISF website (http://www.openinfosecfoundation.org) and on this list as well prior to the meeting. A full agenda is now available: http://www.openinfosecfoundation.org/images/oisf_it-defense_munich_2012_agenda.pdf We hope to see you there! ---------------------------------------------------- Matt Jonkman Emerging Threats Pro Open Information Security Foundation (OISF) Phone 866-504-2523 x110 http://www.emergingthreatspro.com http://www.openinfosecfoundation.org ---------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4399 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120206/0267a5e5/smime.bin From jonkman at emergingthreatspro.com Tue Feb 7 06:58:52 2012 From: jonkman at emergingthreatspro.com (Matthew Jonkman) Date: Tue, 7 Feb 2012 06:58:52 -0500 Subject: [Oisf-users] Join the Suricata Brainstorming Session Message-ID: <62FAA005-6645-4CCC-9043-00D1ADFA0315@emergingthreatspro.com> Starts soon, 1pm CET or 7am EST. We will be here for quite a few hours, so join when it's convenient to your local time zone! https://plus.google.com/100017416165194672237/posts/ZajBLSSiBC9 We hope to see you there! ---------------------------------------------------- Matt Jonkman Emerging Threats Pro Open Information Security Foundation (OISF) Phone 866-504-2523 x110 http://www.emergingthreatspro.com http://www.openinfosecfoundation.org ---------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4399 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120207/d274c265/smime.bin From eric at regit.org Tue Feb 7 07:55:13 2012 From: eric at regit.org (Eric Leblond) Date: Tue, 07 Feb 2012 13:55:13 +0100 Subject: [Oisf-users] [Discussion] Join the Suricata Brainstorming Session In-Reply-To: <62FAA005-6645-4CCC-9043-00D1ADFA0315@emergingthreatspro.com> References: <62FAA005-6645-4CCC-9043-00D1ADFA0315@emergingthreatspro.com> Message-ID: <1328619313.3089.13.camel@tiger.regit.org> Hello, On Tue, 2012-02-07 at 06:58 -0500, Matthew Jonkman wrote: > Starts soon, 1pm CET or 7am EST. We will be here for quite a few hours, so join when it's convenient to your local time zone! > > https://plus.google.com/100017416165194672237/posts/ZajBLSSiBC9 > > We hope to see you there! Second hangout with sound only is available here: https://plus.google.com/hangouts/37d3daf435df99dc50172c0231901f4f084f4d40 BR, > > ---------------------------------------------------- > Matt Jonkman > Emerging Threats Pro > Open Information Security Foundation (OISF) > Phone 866-504-2523 x110 > http://www.emergingthreatspro.com > http://www.openinfosecfoundation.org > ---------------------------------------------------- > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- Eric Leblond Blog: http://home.regit.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120207/b64322e1/attachment.bin From jonkman at emergingthreatspro.com Wed Feb 8 05:05:30 2012 From: jonkman at emergingthreatspro.com (Matthew Jonkman) Date: Wed, 8 Feb 2012 05:05:30 -0500 Subject: [Oisf-users] [Emerging-Sigs] Join the Suricata Brainstorming Session In-Reply-To: <4F311DAF.9050303@gmail.com> References: <62FAA005-6645-4CCC-9043-00D1ADFA0315@emergingthreatspro.com> <4F311DAF.9050303@gmail.com> Message-ID: <7D4483E2-4321-410C-86A5-C94BB41F7466@emergingthreatspro.com> An update: For those who weren't there: We had some major bandwidth and room size issues. We filled up two Google+ hangouts in seconds (the 10 or so attendee limit is apparently back in place unfortunately), and once we moved to Webex we had local bandwidth issues. We did get some great conversations through, but the frequent drops were a barrier. We had over 50 folks that were able to get in and hear parts, I think we would have had many more if we hadn't had the technical issues. I'll get notes and a summary out for what we were able to cover, and we will get a date rescheduled in the next couple of weeks to have the brainstorming session again. We've resolved to focus on more remote attendance meetings on a regular basis and not have those at the same time as our onsite sessions at conferences. The technical issues of relying on hotel or remote bandwidth have been too much a frustration. That allows us though to start having more regular brainstorming sessions not dependent on travel and conferences. We're thinking about having a regular talk around each release (approximately every 2 months at this point). that would allow the dev guys to discuss and explain what's new in the release, what's left, and then we can dive into what we want them to get into next! Ideas welcome, more info soon! Matt On Feb 7, 2012, at 7:48 AM, Edward Fjellsk?l wrote: > On 02/07/2012 12:58 PM, Matthew Jonkman wrote: >> Starts soon, 1pm CET or 7am EST. We will be here for quite a few hours, so join when it's convenient to your local time zone! >> >> >> https://plus.google.com/100017416165194672237/posts/ZajBLSSiBC9 >> >> >> We hope to see you there! >> >> ---------------------------------------------------- >> Matt Jonkman >> Emerging Threats Pro >> Open Information Security Foundation (OISF) >> Phone 866-504-2523 x110 >> >> http://www.emergingthreatspro.com >> http://www.openinfosecfoundation.org >> >> ---------------------------------------------------- >> >> >> >> _______________________________________________ >> Emerging-sigs mailing list >> >> Emerging-sigs at emergingthreats.net >> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs >> >> >> Support Emerging Threats! Subscribe to Emerging Threats Pro >> http://www.emergingthreatspro.com >> >> The ONLY place to get complete premium rulesets for Snort 2.4.0 through Current! >> > > > > Aww... > We're sorry, the hangout looks to be jam-packed already. There's no more room! > > _______________________________________________ > Emerging-sigs mailing list > Emerging-sigs at emergingthreats.net > http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs > > Support Emerging Threats! Subscribe to Emerging Threats Pro http://www.emergingthreatspro.com > The ONLY place to get complete premium rulesets for Snort 2.4.0 through Current! ---------------------------------------------------- Matt Jonkman Emerging Threats Pro Open Information Security Foundation (OISF) Phone 866-504-2523 x110 http://www.emergingthreatspro.com http://www.openinfosecfoundation.org ---------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4399 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120208/dc682403/smime.bin From josh at securemind.org Wed Feb 8 20:33:05 2012 From: josh at securemind.org (Josh White) Date: Wed, 8 Feb 2012 20:33:05 -0500 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring Message-ID: When I run Suri to monitor multiple interfaces like "suricata -c /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log file has multiple entries for each stat. "one entry for each interface being monitored" Is there an easy way to consolidate the stats so all the interface stats are consolidated? Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120208/59605ee9/attachment.html From ndenev at gmail.com Thu Feb 9 15:03:12 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 9 Feb 2012 22:03:12 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint Message-ID: Hi all, It's probably stupid question and I'm missing something but I don't seem to be able to generate alert immediately when for example a given string is found inside a TCP stream. When the TCP connection closes, suricata immediately prints the alert in fast.log. How can I make the alert be generated immediately when the rule condition is matched? Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, while I have the string that should fire the alert several times in the stream. Here's an example : alert tcp $HOME_NET 6666 -> any any \ (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) alert tcp $HOME_NET 6666 -> any any \ (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) This never works, I just have the first rule fire once when the TCP session is terminated. P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples suricata complains about duplicated rules. Thanks, From ndenev at gmail.com Thu Feb 9 15:04:17 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 9 Feb 2012 22:04:17 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: Message-ID: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: > Hi all, > > It's probably stupid question and I'm missing something but I don't seem to be able > to generate alert immediately when for example a given string is found inside a TCP stream. > When the TCP connection closes, suricata immediately prints the alert in fast.log. > How can I make the alert be generated immediately when the rule condition is matched? > > Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, > while I have the string that should fire the alert several times in the stream. > > Here's an example : > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) > > This never works, I just have the first rule fire once when the TCP session is terminated. > > > P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples > suricata complains about duplicated rules. > > Thanks, > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. From petermanev at gmail.com Thu Feb 9 20:44:23 2012 From: petermanev at gmail.com (Peter Manev) Date: Fri, 10 Feb 2012 02:44:23 +0100 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: References: Message-ID: Hi, I don't think this is possible(in suri), you could of course use some bash/perl/your choice of scripting to achieve that. Thanks On Thu, Feb 9, 2012 at 2:33 AM, Josh White wrote: > When I run Suri to monitor multiple interfaces like "suricata -c > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log file has > multiple entries for each stat. "one entry for each interface being > monitored" > > Is there an easy way to consolidate the stats so all the interface stats > are consolidated? > > Josh > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120210/2a88836a/attachment.html From ndenev at gmail.com Fri Feb 10 00:43:03 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Fri, 10 Feb 2012 07:43:03 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> Message-ID: <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: > >> Hi all, >> >> It's probably stupid question and I'm missing something but I don't seem to be able >> to generate alert immediately when for example a given string is found inside a TCP stream. >> When the TCP connection closes, suricata immediately prints the alert in fast.log. >> How can I make the alert be generated immediately when the rule condition is matched? >> >> Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, >> while I have the string that should fire the alert several times in the stream. >> >> Here's an example : >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) >> >> This never works, I just have the first rule fire once when the TCP session is terminated. >> >> >> P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples >> suricata complains about duplicated rules. >> >> Thanks, >> > > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. This seems to work : alert tcp $HOME_NET 6666 -> any any \ (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; noalert; sid:10; priority: 1;) alert tcp $HOME_NET 6666 -> any any \ (msg:"got more"; content:"something"; flowint:something,isset; flowint:something,+,1; noalert; sid:11; priority: 2;) alert tcp $HOME_NET 6666 -> any any \ (msg:"got too many"; content:"something"; flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) From victor at inliniac.net Fri Feb 10 09:20:40 2012 From: victor at inliniac.net (Victor Julien) Date: Fri, 10 Feb 2012 15:20:40 +0100 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: References: Message-ID: <4F3527B8.6020307@inliniac.net> On 02/10/2012 02:44 AM, Peter Manev wrote: > Hi, > > I don't think this is possible(in suri), you could of course use some > bash/perl/your choice of scripting to achieve that. It's indeed not possible right now. I'm a bit torn on it, as I see use for both cases. Ideally we're have it both simultaneously. Maybe we should an easily parseble (csv or something) output option. Cheers, Victor > > Thanks > > On Thu, Feb 9, 2012 at 2:33 AM, Josh White > wrote: > > When I run Suri to monitor multiple interfaces like "suricata -c > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log file > has multiple entries for each stat. "one entry for each interface > being monitored" > > Is there an easy way to consolidate the stats so all the interface > stats are consolidated? > > Josh > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > -- > Peter Manev > > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From petermanev at gmail.com Sat Feb 11 05:11:24 2012 From: petermanev at gmail.com (Peter Manev) Date: Sat, 11 Feb 2012 11:11:24 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: > > On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: > > > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: > > > >> Hi all, > >> > >> It's probably stupid question and I'm missing something but I don't > seem to be able > >> to generate alert immediately when for example a given string is found > inside a TCP stream. > >> When the TCP connection closes, suricata immediately prints the alert > in fast.log. > >> How can I make the alert be generated immediately when the rule > condition is matched? > >> > >> Also I don't know if its because of this I don't seem to be able to > trigger the rule to match several times on the same stream, > >> while I have the string that should fire the alert several times in the > stream. > >> > >> Here's an example : > >> > >> alert tcp $HOME_NET 6666 -> any any \ > >> (msg:"got one"; content:"something"; flowint:something,notset; > flowint:something,=,1; sid:10;) > >> > >> alert tcp $HOME_NET 6666 -> any any \ > >> (msg:"got five or more"; content:"something"; > flowint:something,isset; flowint:something,+,1; flowint:something,>,5; > sid:11;) > >> > >> This never works, I just have the first rule fire once when the TCP > session is terminated. > >> > >> > >> P.S.: As a side note the wiki should be updated to include probably > "sid"s for the rules, as currently when I try to run the examples > >> suricata complains about duplicated rules. > >> > >> Thanks, > >> > > > > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. > > This seems to work : > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got one"; content:"something"; flowint:something,notset; > flowint:something,=,1; noalert; sid:10; priority: 1;) > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got more"; content:"something"; flowint:something,isset; > flowint:something,+,1; noalert; sid:11; priority: 2;) > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got too many"; content:"something"; flowint:something,isset; > flowint:something,>,2; sid:12; priority: 3;) > > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > Hi Nikolay, I think this is the way it is supposed to work. (last example, by you). When you take out "noalert" form sid 11 - does it fire ? And are these the only rules that are loaded in terms of flowint or you have others before that? thanks -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/7d0cce6a/attachment.html From ndenev at gmail.com Sat Feb 11 10:31:26 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Sat, 11 Feb 2012 17:31:26 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: > > > On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: > > On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: > > > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: > > > >> Hi all, > >> > >> It's probably stupid question and I'm missing something but I don't seem to be able > >> to generate alert immediately when for example a given string is found inside a TCP stream. > >> When the TCP connection closes, suricata immediately prints the alert in fast.log. > >> How can I make the alert be generated immediately when the rule condition is matched? > >> > >> Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, > >> while I have the string that should fire the alert several times in the stream. > >> > >> Here's an example : > >> > >> alert tcp $HOME_NET 6666 -> any any \ > >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) > >> > >> alert tcp $HOME_NET 6666 -> any any \ > >> (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) > >> > >> This never works, I just have the first rule fire once when the TCP session is terminated. > >> > >> > >> P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples > >> suricata complains about duplicated rules. > >> > >> Thanks, > >> > > > > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. > > This seems to work : > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; noalert; sid:10; priority: 1;) > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got more"; content:"something"; flowint:something,isset; flowint:something,+,1; noalert; sid:11; priority: 2;) > > alert tcp $HOME_NET 6666 -> any any \ > (msg:"got too many"; content:"something"; flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) > > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > Hi Nikolay, > I think this is the way it is supposed to work. (last example, by you). > > When you take out "noalert" form sid 11 - does it fire ? > > And are these the only rules that are loaded in terms of flowint or you have others before that? > > thanks > > > > -- > Peter Manev Yes, It fires, the problem I have is that it doesn't fire for each occurence of "content". Is alert supposed to fire once per packet if it matches, or for each match in the stream? For example now I'm using these rules to catch if there are more than some defined amount of email addresses in a given stream : alert tcp $HOME_NET 80 -> any any \ (msg:"got one email addr"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3; noalert;) alert tcp $HOME_NET 80 -> any any \ (msg:"got more email addrs"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2; noalert;) alert tcp $HOME_NET 80 -> any any \ (msg:"Got too many email addrs!"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ flow:established,from_server; flowint:something,isset; flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) This for example works, but would not match for a simple plain text file with 10 email adresses, I need to have maybe 40-50 or more for this to match. Maybe I'm missing something? And yes, these are my only rules that I'm testing with. No other rules with or without flowint whatsoever. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/ec10da87/attachment.html From petermanev at gmail.com Sat Feb 11 12:52:51 2012 From: petermanev at gmail.com (Peter Manev) Date: Sat, 11 Feb 2012 18:52:51 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: > > > > On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: > >> >> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >> >> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >> > >> >> Hi all, >> >> >> >> It's probably stupid question and I'm missing something but I don't >> seem to be able >> >> to generate alert immediately when for example a given string is found >> inside a TCP stream. >> >> When the TCP connection closes, suricata immediately prints the alert >> in fast.log. >> >> How can I make the alert be generated immediately when the rule >> condition is matched? >> >> >> >> Also I don't know if its because of this I don't seem to be able to >> trigger the rule to match several times on the same stream, >> >> while I have the string that should fire the alert several times in >> the stream. >> >> >> >> Here's an example : >> >> >> >> alert tcp $HOME_NET 6666 -> any any \ >> >> (msg:"got one"; content:"something"; flowint:something,notset; >> flowint:something,=,1; sid:10;) >> >> >> >> alert tcp $HOME_NET 6666 -> any any \ >> >> (msg:"got five or more"; content:"something"; >> flowint:something,isset; flowint:something,+,1; flowint:something,>,5; >> sid:11;) >> >> >> >> This never works, I just have the first rule fire once when the TCP >> session is terminated. >> >> >> >> >> >> P.S.: As a side note the wiki should be updated to include probably >> "sid"s for the rules, as currently when I try to run the examples >> >> suricata complains about duplicated rules. >> >> >> >> Thanks, >> >> >> > >> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >> >> This seems to work : >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got one"; content:"something"; flowint:something,notset; >> flowint:something,=,1; noalert; sid:10; priority: 1;) >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got more"; content:"something"; flowint:something,isset; >> flowint:something,+,1; noalert; sid:11; priority: 2;) >> > >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got too many"; content:"something"; >> flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >> >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> > > > Hi Nikolay, > I think this is the way it is supposed to work. (last example, by you). > > When you take out "noalert" form sid 11 - does it fire ? > > And are these the only rules that are loaded in terms of flowint or you > have others before that? > > thanks > > > > -- > Peter Manev > > > > Yes, It fires, the problem I have is that it doesn't fire for each > occurence of "content". > Is alert supposed to fire once per packet if it matches, or for each match > in the stream? > > For example now I'm using these rules to catch if there are more than some > defined amount of email addresses in a given stream : > > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got one email addr"; content:"|40|"; > pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,notset; > flowint:something,=,1; sid:10; priority:3; noalert;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got more email addrs"; content:"|40|"; > pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,isset; > flowint:something,+,1; sid:11; priority:2; noalert;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"Got too many email addrs!"; content:"|40|"; > pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,isset; > flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) > > > This for example works, but would not match for a simple plain text file > with 10 email adresses, I need to have maybe 40-50 or more for this to > match. > Maybe I'm missing something? > And yes, these are my only rules that I'm testing with. No other rules with > or without flowint whatsoever. > > Hi , Just so I understand you correctly - you have a text file (in the stream) and in that text file you have 10 e-mail addresses and it wold not fire. correct ? thanks -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/79473a0b/attachment.html From ndenev at gmail.com Sat Feb 11 13:42:38 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Sat, 11 Feb 2012 20:42:38 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: > > > On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: > >> >> >> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >> >> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >> >> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >> > >> >> Hi all, >> >> >> >> It's probably stupid question and I'm missing something but I don't seem to be able >> >> to generate alert immediately when for example a given string is found inside a TCP stream. >> >> When the TCP connection closes, suricata immediately prints the alert in fast.log. >> >> How can I make the alert be generated immediately when the rule condition is matched? >> >> >> >> Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, >> >> while I have the string that should fire the alert several times in the stream. >> >> >> >> Here's an example : >> >> >> >> alert tcp $HOME_NET 6666 -> any any \ >> >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) >> >> >> >> alert tcp $HOME_NET 6666 -> any any \ >> >> (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) >> >> >> >> This never works, I just have the first rule fire once when the TCP session is terminated. >> >> >> >> >> >> P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples >> >> suricata complains about duplicated rules. >> >> >> >> Thanks, >> >> >> > >> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >> >> This seems to work : >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; noalert; sid:10; priority: 1;) >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got more"; content:"something"; flowint:something,isset; flowint:something,+,1; noalert; sid:11; priority: 2;) >> >> alert tcp $HOME_NET 6666 -> any any \ >> (msg:"got too many"; content:"something"; flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >> >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >> >> Hi Nikolay, >> I think this is the way it is supposed to work. (last example, by you). >> >> When you take out "noalert" form sid 11 - does it fire ? >> >> And are these the only rules that are loaded in terms of flowint or you have others before that? >> >> thanks >> >> >> >> -- >> Peter Manev > > > Yes, It fires, the problem I have is that it doesn't fire for each occurence of "content". > Is alert supposed to fire once per packet if it matches, or for each match in the stream? > > For example now I'm using these rules to catch if there are more than some defined amount of email addresses in a given stream : > > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got one email addr"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3; noalert;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got more email addrs"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2; noalert;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"Got too many email addrs!"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > flow:established,from_server; flowint:something,isset; flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) > > > This for example works, but would not match for a simple plain text file with 10 email adresses, I need to have maybe 40-50 or more for this to match. > Maybe I'm missing something? > And yes, these are my only rules that I'm testing with. No other rules with or without flowint whatsoever. > > > Hi , > Just so I understand you correctly - you have a text file (in the stream) and in that text file you have 10 e-mail addresses and it wold not fire. correct ? > > > thanks > > > -- > Peter Manev Exactly. For example if I try to fetch the file emails.txt via http which has the following content : # cat emails.txt edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com $ curl http://testserver/emails.txt edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com edin at email.com $ And I also remove the "noalert" option from the rules, this is what I get in fast.log : 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 If I change the third rule to fire if the flowint var is more than 1, it is being triggered. If I insert some random data between the email addresses in the text file, then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/75dbe11a/attachment-0001.html From petermanev at gmail.com Sat Feb 11 14:14:08 2012 From: petermanev at gmail.com (Peter Manev) Date: Sat, 11 Feb 2012 20:14:08 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: > > > > On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: > >> >> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >> >> >> >> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >> >>> >>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>> >>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>> > >>> >> Hi all, >>> >> >>> >> It's probably stupid question and I'm missing something but I don't >>> seem to be able >>> >> to generate alert immediately when for example a given string is >>> found inside a TCP stream. >>> >> When the TCP connection closes, suricata immediately prints the alert >>> in fast.log. >>> >> How can I make the alert be generated immediately when the rule >>> condition is matched? >>> >> >>> >> Also I don't know if its because of this I don't seem to be able to >>> trigger the rule to match several times on the same stream, >>> >> while I have the string that should fire the alert several times in >>> the stream. >>> >> >>> >> Here's an example : >>> >> >>> >> alert tcp $HOME_NET 6666 -> any any \ >>> >> (msg:"got one"; content:"something"; flowint:something,notset; >>> flowint:something,=,1; sid:10;) >>> >> >>> >> alert tcp $HOME_NET 6666 -> any any \ >>> >> (msg:"got five or more"; content:"something"; >>> flowint:something,isset; flowint:something,+,1; flowint:something,>,5; >>> sid:11;) >>> >> >>> >> This never works, I just have the first rule fire once when the TCP >>> session is terminated. >>> >> >>> >> >>> >> P.S.: As a side note the wiki should be updated to include probably >>> "sid"s for the rules, as currently when I try to run the examples >>> >> suricata complains about duplicated rules. >>> >> >>> >> Thanks, >>> >> >>> > >>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>> >>> This seems to work : >>> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got one"; content:"something"; flowint:something,notset; >>> flowint:something,=,1; noalert; sid:10; priority: 1;) >>> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got more"; content:"something"; flowint:something,isset; >>> flowint:something,+,1; noalert; sid:11; priority: 2;) >>> >> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got too many"; content:"something"; >>> flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >>> >>> >>> _______________________________________________ >>> Oisf-users mailing list >>> Oisf-users at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> >> >> >> Hi Nikolay, >> I think this is the way it is supposed to work. (last example, by you). >> >> When you take out "noalert" form sid 11 - does it fire ? >> >> And are these the only rules that are loaded in terms of flowint or you >> have others before that? >> >> thanks >> >> >> >> -- >> Peter Manev >> >> >> >> Yes, It fires, the problem I have is that it doesn't fire for each >> occurence of "content". >> Is alert supposed to fire once per packet if it matches, or for each >> match in the stream? >> >> For example now I'm using these rules to catch if there are more than >> some defined amount of email addresses in a given stream : >> >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got one email addr"; content:"|40|"; >> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,notset; >> flowint:something,=,1; sid:10; priority:3; noalert;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got more email addrs"; content:"|40|"; >> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,+,1; sid:11; priority:2; noalert;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"Got too many email addrs!"; content:"|40|"; >> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) >> >> >> This for example works, but would not match for a simple plain text file >> with 10 email adresses, I need to have maybe 40-50 or more for this to >> match. >> Maybe I'm missing something? >> > And yes, these are my only rules that I'm testing with. No other rules >> with or without flowint whatsoever. >> >> > Hi , > Just so I understand you correctly - you have a text file (in the stream) > and in that text file you have 10 e-mail addresses and it wold not fire. > correct ? > > > thanks > > > -- > Peter Manev > > > Exactly. > > For example if I try to fetch the file emails.txt via http which has the > following content : > > # cat emails.txt > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > > $ curl http://testserver/emails.txt > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > $ > > And I also remove the "noalert" option from the rules, this is what I get > in fast.log : > > 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] > [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 > 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] > [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 > > > If I change the third rule to fire if the flowint var is more than 1, it > is being triggered. > > If I insert some random data between the email addresses in the text file, > then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? > 1. What happens if you take out the PCRE expressions from all the rules ? 2. sid:12 - should not fire because you have >10 , and there are exactly 10 e-mails in the file 3. how big is the stream itself? i think it is below 2KB, correct? 4. is the PCRE matching the e-mails, under the unix shell ? 5. yes i think you should get more sid:11 alerts - but first lets investigate the above 4. thanks -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/ffbd21ac/attachment.html From ndenev at gmail.com Sat Feb 11 14:27:00 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Sat, 11 Feb 2012 21:27:00 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> Message-ID: <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: > > > On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: > >> >> >> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: >> >> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >> >>> >>> >>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >>> >>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>> >>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>> > >>> >> Hi all, >>> >> >>> >> It's probably stupid question and I'm missing something but I don't seem to be able >>> >> to generate alert immediately when for example a given string is found inside a TCP stream. >>> >> When the TCP connection closes, suricata immediately prints the alert in fast.log. >>> >> How can I make the alert be generated immediately when the rule condition is matched? >>> >> >>> >> Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, >>> >> while I have the string that should fire the alert several times in the stream. >>> >> >>> >> Here's an example : >>> >> >>> >> alert tcp $HOME_NET 6666 -> any any \ >>> >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) >>> >> >>> >> alert tcp $HOME_NET 6666 -> any any \ >>> >> (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) >>> >> >>> >> This never works, I just have the first rule fire once when the TCP session is terminated. >>> >> >>> >> >>> >> P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples >>> >> suricata complains about duplicated rules. >>> >> >>> >> Thanks, >>> >> >>> > >>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>> >>> This seems to work : >>> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; noalert; sid:10; priority: 1;) >>> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got more"; content:"something"; flowint:something,isset; flowint:something,+,1; noalert; sid:11; priority: 2;) >>> >>> alert tcp $HOME_NET 6666 -> any any \ >>> (msg:"got too many"; content:"something"; flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >>> >>> >>> _______________________________________________ >>> Oisf-users mailing list >>> Oisf-users at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> >>> >>> Hi Nikolay, >>> I think this is the way it is supposed to work. (last example, by you). >>> >>> When you take out "noalert" form sid 11 - does it fire ? >>> >>> And are these the only rules that are loaded in terms of flowint or you have others before that? >>> >>> thanks >>> >>> >>> >>> -- >>> Peter Manev >> >> >> Yes, It fires, the problem I have is that it doesn't fire for each occurence of "content". >> Is alert supposed to fire once per packet if it matches, or for each match in the stream? >> >> For example now I'm using these rules to catch if there are more than some defined amount of email addresses in a given stream : >> >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got one email addr"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3; noalert;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got more email addrs"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2; noalert;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"Got too many email addrs!"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> flow:established,from_server; flowint:something,isset; flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) >> >> >> This for example works, but would not match for a simple plain text file with 10 email adresses, I need to have maybe 40-50 or more for this to match. >> Maybe I'm missing something? >> And yes, these are my only rules that I'm testing with. No other rules with or without flowint whatsoever. >> >> >> Hi , >> Just so I understand you correctly - you have a text file (in the stream) and in that text file you have 10 e-mail addresses and it wold not fire. correct ? >> >> >> thanks >> >> >> -- >> Peter Manev > > Exactly. > > For example if I try to fetch the file emails.txt via http which has the following content : > > # cat emails.txt > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > > $ curl http://testserver/emails.txt > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > edin at email.com > $ > > And I also remove the "noalert" option from the rules, this is what I get in fast.log : > > 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 > 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 > > > If I change the third rule to fire if the flowint var is more than 1, it is being triggered. > > If I insert some random data between the email addresses in the text file, then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? > > 1. What happens if you take out the PCRE expressions from all the rules ? > 2. sid:12 - should not fire because you have >10 , and there are exactly 10 e-mails in the file > 3. how big is the stream itself? i think it is below 2KB, correct? > 4. is the PCRE matching the e-mails, under the unix shell ? > 5. yes i think you should get more sid:11 alerts - but first lets investigate the above 4. > > thanks > > -- > Peter Manev The file with only the 10 emails is 160 bytes. Even without pcre I get the same result : alert tcp $HOME_NET 80 -> any any \ (msg:"got one email addr"; content:"|40|"; \ flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3;) alert tcp $HOME_NET 80 -> any any \ (msg:"got more email addrs"; content:"|40|"; \ flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2;) alert tcp $HOME_NET 80 -> any any \ (msg:"Got too many email addrs!"; content:"|40|"; \ flow:established,from_server; flowint:something,isset; flowint:something,>,9; sid:12; priority:1; classtype:policy-violation;) alerts I get : 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 If I put some '#' symbols between the emails in the file so that it gets about 9K big and I fetch it I get these alerts : 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/81dc8a60/attachment-0001.html From petermanev at gmail.com Sat Feb 11 15:11:23 2012 From: petermanev at gmail.com (Peter Manev) Date: Sat, 11 Feb 2012 21:11:23 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> Message-ID: On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: > > > > On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev wrote: > >> >> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >> >> >> >> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: >> >>> >>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>> >>> >>> >>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >>> >>>> >>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>> >>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>> > >>>> >> Hi all, >>>> >> >>>> >> It's probably stupid question and I'm missing something but I don't >>>> seem to be able >>>> >> to generate alert immediately when for example a given string is >>>> found inside a TCP stream. >>>> >> When the TCP connection closes, suricata immediately prints the >>>> alert in fast.log. >>>> >> How can I make the alert be generated immediately when the rule >>>> condition is matched? >>>> >> >>>> >> Also I don't know if its because of this I don't seem to be able to >>>> trigger the rule to match several times on the same stream, >>>> >> while I have the string that should fire the alert several times in >>>> the stream. >>>> >> >>>> >> Here's an example : >>>> >> >>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>> >> (msg:"got one"; content:"something"; flowint:something,notset; >>>> flowint:something,=,1; sid:10;) >>>> >> >>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>> >> (msg:"got five or more"; content:"something"; >>>> flowint:something,isset; flowint:something,+,1; flowint:something,>,5; >>>> sid:11;) >>>> >> >>>> >> This never works, I just have the first rule fire once when the TCP >>>> session is terminated. >>>> >> >>>> >> >>>> >> P.S.: As a side note the wiki should be updated to include probably >>>> "sid"s for the rules, as currently when I try to run the examples >>>> >> suricata complains about duplicated rules. >>>> >> >>>> >> Thanks, >>>> >> >>>> > >>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>> >>>> This seems to work : >>>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got one"; content:"something"; flowint:something,notset; >>>> flowint:something,=,1; noalert; sid:10; priority: 1;) >>>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got more"; content:"something"; flowint:something,isset; >>>> flowint:something,+,1; noalert; sid:11; priority: 2;) >>>> >>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got too many"; content:"something"; >>>> flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >>>> >>>> >>>> _______________________________________________ >>>> Oisf-users mailing list >>>> Oisf-users at openinfosecfoundation.org >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>> >>> >>> >>> Hi Nikolay, >>> I think this is the way it is supposed to work. (last example, by you). >>> >>> When you take out "noalert" form sid 11 - does it fire ? >>> >>> And are these the only rules that are loaded in terms of flowint or you >>> have others before that? >>> >>> thanks >>> >>> >>> >>> -- >>> Peter Manev >>> >>> >>> >>> Yes, It fires, the problem I have is that it doesn't fire for each >>> occurence of "content". >>> Is alert supposed to fire once per packet if it matches, or for each >>> match in the stream? >>> >>> For example now I'm using these rules to catch if there are more than >>> some defined amount of email addresses in a given stream : >>> >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got one email addr"; content:"|40|"; >>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,notset; >>> flowint:something,=,1; sid:10; priority:3; noalert;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got more email addrs"; content:"|40|"; >>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,isset; >>> flowint:something,+,1; sid:11; priority:2; noalert;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"Got too many email addrs!"; content:"|40|"; >>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,isset; >>> flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) >>> >>> >>> This for example works, but would not match for a simple plain text file >>> with 10 email adresses, I need to have maybe 40-50 or more for this to >>> match. >>> Maybe I'm missing something? >>> >> And yes, these are my only rules that I'm testing with. No other rules >>> with or without flowint whatsoever. >>> >>> >> Hi , >> Just so I understand you correctly - you have a text file (in the stream) >> and in that text file you have 10 e-mail addresses and it wold not fire. >> correct ? >> >> >> thanks >> >> >> -- >> Peter Manev >> >> >> Exactly. >> >> For example if I try to fetch the file emails.txt via http which has the >> following content : >> >> # cat emails.txt >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> >> $ curl http://testserver/emails.txt >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> $ >> >> And I also remove the "noalert" option from the rules, this is what I get >> in fast.log : >> >> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] >> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >> >> >> If I change the third rule to fire if the flowint var is more than 1, it >> is being triggered. >> >> If I insert some random data between the email addresses in the text >> file, then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? >> > > 1. What happens if you take out the PCRE expressions from all the rules ? > 2. sid:12 - should not fire because you have >10 , and there are exactly > 10 e-mails in the file > 3. how big is the stream itself? i think it is below 2KB, correct? > 4. is the PCRE matching the e-mails, under the unix shell ? > 5. yes i think you should get more sid:11 alerts - but first lets > investigate the above 4. > > thanks > > -- > Peter Manev > > > The file with only the 10 emails is 160 bytes. Even without pcre I get the > same result : > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got one email addr"; content:"|40|"; \ > flow:established,from_server; flowint:something,notset; > flowint:something,=,1; sid:10; priority:3;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got more email addrs"; content:"|40|"; \ > flow:established,from_server; flowint:something,isset; > flowint:something,+,1; sid:11; priority:2;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"Got too many email addrs!"; content:"|40|"; \ > flow:established,from_server; flowint:something,isset; > flowint:something,>,9; sid:12; priority:1; classtype:policy-violation;) > > > alerts I get : > > 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] > [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 > 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs [**] > [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 > > If I put some '#' symbols between the emails in the file so that it gets > about 9K big and I fetch it I get these alerts : > > 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] > [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs [**] > [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs [**] > [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs [**] > [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > > Hi Nikolay, Can you please post this as a bug - please be detailed (as you were in your 2 previous e-mails). Personally i think here sid 11 is the problem , may be it does not count/increment correctly.... thanks -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/7ed38155/attachment.html From josh at securemind.org Sat Feb 11 19:04:57 2012 From: josh at securemind.org (Josh White) Date: Sat, 11 Feb 2012 19:04:57 -0500 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: <4F3527B8.6020307@inliniac.net> References: <4F3527B8.6020307@inliniac.net> Message-ID: That would work, I was originally thinking even an option to append the interface name and have have multiple stats files like stats.log.em1 or the reverse em1.stats.log. However if it was more of a csv format then it would be easier to graph in some cases. On Fri, Feb 10, 2012 at 9:20 AM, Victor Julien wrote: > On 02/10/2012 02:44 AM, Peter Manev wrote: > > Hi, > > > > I don't think this is possible(in suri), you could of course use some > > bash/perl/your choice of scripting to achieve that. > > It's indeed not possible right now. I'm a bit torn on it, as I see use > for both cases. Ideally we're have it both simultaneously. Maybe we > should an easily parseble (csv or something) output option. > > Cheers, > Victor > > > > > Thanks > > > > On Thu, Feb 9, 2012 at 2:33 AM, Josh White > > wrote: > > > > When I run Suri to monitor multiple interfaces like "suricata -c > > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log file > > has multiple entries for each stat. "one entry for each interface > > being monitored" > > > > Is there an easy way to consolidate the stats so all the interface > > stats are consolidated? > > > > Josh > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > > > > > -- > > Peter Manev > > > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120211/3fc8ef6b/attachment-0001.html From ndenev at gmail.com Sun Feb 12 02:15:46 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Sun, 12 Feb 2012 09:15:46 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> Message-ID: <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: > > > On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev wrote: > > On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: > >> >> >> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev wrote: >> >> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >> >>> >>> >>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: >>> >>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>> >>>> >>>> >>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >>>> >>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>> >>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>> > >>>> >> Hi all, >>>> >> >>>> >> It's probably stupid question and I'm missing something but I don't seem to be able >>>> >> to generate alert immediately when for example a given string is found inside a TCP stream. >>>> >> When the TCP connection closes, suricata immediately prints the alert in fast.log. >>>> >> How can I make the alert be generated immediately when the rule condition is matched? >>>> >> >>>> >> Also I don't know if its because of this I don't seem to be able to trigger the rule to match several times on the same stream, >>>> >> while I have the string that should fire the alert several times in the stream. >>>> >> >>>> >> Here's an example : >>>> >> >>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>> >> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; sid:10;) >>>> >> >>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>> >> (msg:"got five or more"; content:"something"; flowint:something,isset; flowint:something,+,1; flowint:something,>,5; sid:11;) >>>> >> >>>> >> This never works, I just have the first rule fire once when the TCP session is terminated. >>>> >> >>>> >> >>>> >> P.S.: As a side note the wiki should be updated to include probably "sid"s for the rules, as currently when I try to run the examples >>>> >> suricata complains about duplicated rules. >>>> >> >>>> >> Thanks, >>>> >> >>>> > >>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>> >>>> This seems to work : >>>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got one"; content:"something"; flowint:something,notset; flowint:something,=,1; noalert; sid:10; priority: 1;) >>>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got more"; content:"something"; flowint:something,isset; flowint:something,+,1; noalert; sid:11; priority: 2;) >>>> >>>> alert tcp $HOME_NET 6666 -> any any \ >>>> (msg:"got too many"; content:"something"; flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >>>> >>>> >>>> _______________________________________________ >>>> Oisf-users mailing list >>>> Oisf-users at openinfosecfoundation.org >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>> >>>> >>>> Hi Nikolay, >>>> I think this is the way it is supposed to work. (last example, by you). >>>> >>>> When you take out "noalert" form sid 11 - does it fire ? >>>> >>>> And are these the only rules that are loaded in terms of flowint or you have others before that? >>>> >>>> thanks >>>> >>>> >>>> >>>> -- >>>> Peter Manev >>> >>> >>> Yes, It fires, the problem I have is that it doesn't fire for each occurence of "content". >>> Is alert supposed to fire once per packet if it matches, or for each match in the stream? >>> >>> For example now I'm using these rules to catch if there are more than some defined amount of email addresses in a given stream : >>> >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got one email addr"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3; noalert;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got more email addrs"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2; noalert;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"Got too many email addrs!"; content:"|40|"; pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>> flow:established,from_server; flowint:something,isset; flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) >>> >>> >>> This for example works, but would not match for a simple plain text file with 10 email adresses, I need to have maybe 40-50 or more for this to match. >>> Maybe I'm missing something? >>> And yes, these are my only rules that I'm testing with. No other rules with or without flowint whatsoever. >>> >>> >>> Hi , >>> Just so I understand you correctly - you have a text file (in the stream) and in that text file you have 10 e-mail addresses and it wold not fire. correct ? >>> >>> >>> thanks >>> >>> >>> -- >>> Peter Manev >> >> Exactly. >> >> For example if I try to fetch the file emails.txt via http which has the following content : >> >> # cat emails.txt >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> >> $ curl http://testserver/emails.txt >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> edin at email.com >> $ >> >> And I also remove the "noalert" option from the rules, this is what I get in fast.log : >> >> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >> >> >> If I change the third rule to fire if the flowint var is more than 1, it is being triggered. >> >> If I insert some random data between the email addresses in the text file, then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? >> >> 1. What happens if you take out the PCRE expressions from all the rules ? >> 2. sid:12 - should not fire because you have >10 , and there are exactly 10 e-mails in the file >> 3. how big is the stream itself? i think it is below 2KB, correct? >> 4. is the PCRE matching the e-mails, under the unix shell ? >> 5. yes i think you should get more sid:11 alerts - but first lets investigate the above 4. >> >> thanks >> >> -- >> Peter Manev > > The file with only the 10 emails is 160 bytes. Even without pcre I get the same result : > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got one email addr"; content:"|40|"; \ > flow:established,from_server; flowint:something,notset; flowint:something,=,1; sid:10; priority:3;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"got more email addrs"; content:"|40|"; \ > flow:established,from_server; flowint:something,isset; flowint:something,+,1; sid:11; priority:2;) > > alert tcp $HOME_NET 80 -> any any \ > (msg:"Got too many email addrs!"; content:"|40|"; \ > flow:established,from_server; flowint:something,isset; flowint:something,>,9; sid:12; priority:1; classtype:policy-violation;) > > > alerts I get : > > 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 > 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 > > If I put some '#' symbols between the emails in the file so that it gets about 9K big and I fetch it I get these alerts : > > 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 > > > > Hi Nikolay, > > > Can you please post this as a bug - please be detailed (as you were in your 2 previous e-mails). > Personally i think here sid 11 is the problem , may be it does not count/increment correctly.... > thanks > > > -- > Peter Manev Yes I will post this as a bug. But I've just found a much simpler case. Let's for example have only this rule in suricata : alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) Then on a monitored machine from the $HOME_NET range I do : echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 And on different host I do : nc testserver 6666 This gets the ten @ chars transferred, and I get only one alert. But for example if I echo more @ chars, like 5000 or something, I get 3-6 alerts. I have to check what is actually the number of packets with payload, probably the rule is matched once per packet? But this could not explain that I get different number of alerts on different runs. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120212/c7fc1da5/attachment.html From petermanev at gmail.com Sun Feb 12 04:48:10 2012 From: petermanev at gmail.com (Peter Manev) Date: Sun, 12 Feb 2012 10:48:10 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> Message-ID: On Sun, Feb 12, 2012 at 8:15 AM, Nikolay Denev wrote: > > On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: > > > > On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev wrote: > >> >> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >> >> >> >> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev wrote: >> >>> >>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>> >>> >>> >>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev wrote: >>> >>>> >>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>> >>>> >>>> >>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev wrote: >>>> >>>>> >>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>> >>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>> > >>>>> >> Hi all, >>>>> >> >>>>> >> It's probably stupid question and I'm missing something but I don't >>>>> seem to be able >>>>> >> to generate alert immediately when for example a given string is >>>>> found inside a TCP stream. >>>>> >> When the TCP connection closes, suricata immediately prints the >>>>> alert in fast.log. >>>>> >> How can I make the alert be generated immediately when the rule >>>>> condition is matched? >>>>> >> >>>>> >> Also I don't know if its because of this I don't seem to be able to >>>>> trigger the rule to match several times on the same stream, >>>>> >> while I have the string that should fire the alert several times in >>>>> the stream. >>>>> >> >>>>> >> Here's an example : >>>>> >> >>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>> >> (msg:"got one"; content:"something"; >>>>> flowint:something,notset; flowint:something,=,1; sid:10;) >>>>> >> >>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>> >> (msg:"got five or more"; content:"something"; >>>>> flowint:something,isset; flowint:something,+,1; flowint:something,>,5; >>>>> sid:11;) >>>>> >> >>>>> >> This never works, I just have the first rule fire once when the TCP >>>>> session is terminated. >>>>> >> >>>>> >> >>>>> >> P.S.: As a side note the wiki should be updated to include probably >>>>> "sid"s for the rules, as currently when I try to run the examples >>>>> >> suricata complains about duplicated rules. >>>>> >> >>>>> >> Thanks, >>>>> >> >>>>> > >>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>> >>>>> This seems to work : >>>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got one"; content:"something"; flowint:something,notset; >>>>> flowint:something,=,1; noalert; sid:10; priority: 1;) >>>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got more"; content:"something"; flowint:something,isset; >>>>> flowint:something,+,1; noalert; sid:11; priority: 2;) >>>>> >>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got too many"; content:"something"; >>>>> flowint:something,isset; flowint:something,>,2; sid:12; priority: 3;) >>>>> >>>>> >>>>> _______________________________________________ >>>>> Oisf-users mailing list >>>>> Oisf-users at openinfosecfoundation.org >>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>> >>>> >>>> >>>> Hi Nikolay, >>>> I think this is the way it is supposed to work. (last example, by you). >>>> >>>> When you take out "noalert" form sid 11 - does it fire ? >>>> >>>> And are these the only rules that are loaded in terms of flowint or you >>>> have others before that? >>>> >>>> thanks >>>> >>>> >>>> >>>> -- >>>> Peter Manev >>>> >>>> >>>> >>>> Yes, It fires, the problem I have is that it doesn't fire for each >>>> occurence of "content". >>>> Is alert supposed to fire once per packet if it matches, or for each >>>> match in the stream? >>>> >>>> For example now I'm using these rules to catch if there are more than >>>> some defined amount of email addresses in a given stream : >>>> >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got one email addr"; content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; flowint:something,notset; >>>> flowint:something,=,1; sid:10; priority:3; noalert;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got more email addrs"; content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; flowint:something,isset; >>>> flowint:something,+,1; sid:11; priority:2; noalert;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"Got too many email addrs!"; content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; flowint:something,isset; >>>> flowint:something,>,10; sid:12; priority:1; classtype:policy-violation;) >>>> >>>> >>>> This for example works, but would not match for a simple plain text >>>> file with 10 email adresses, I need to have maybe 40-50 or more for this to >>>> match. >>>> Maybe I'm missing something? >>>> >>> And yes, these are my only rules that I'm testing with. No other rules >>>> with or without flowint whatsoever. >>>> >>>> >>> Hi , >>> Just so I understand you correctly - you have a text file (in the >>> stream) and in that text file you have 10 e-mail addresses and it wold not >>> fire. correct ? >>> >>> >>> thanks >>> >>> >>> -- >>> Peter Manev >>> >>> >>> Exactly. >>> >>> For example if I try to fetch the file emails.txt via http which has the >>> following content : >>> >>> # cat emails.txt >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> >>> $ curl http://testserver/emails.txt >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> $ >>> >>> And I also remove the "noalert" option from the rules, this is what I >>> get in fast.log : >>> >>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr [**] >>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email addrs [**] >>> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:57923 >>> >>> >>> If I change the third rule to fire if the flowint var is more than 1, it >>> is being triggered. >>> >>> If I insert some random data between the email addresses in the text >>> file, then I get 4 maybe 5 matches. Doesn't it have to match all 10 of them? >>> >> >> 1. What happens if you take out the PCRE expressions from all the rules ? >> 2. sid:12 - should not fire because you have >10 , and there are exactly >> 10 e-mails in the file >> 3. how big is the stream itself? i think it is below 2KB, correct? >> 4. is the PCRE matching the e-mails, under the unix shell ? >> 5. yes i think you should get more sid:11 alerts - but first lets >> investigate the above 4. >> >> thanks >> >> -- >> Peter Manev >> >> >> The file with only the 10 emails is 160 bytes. Even without pcre I get >> the same result : >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got one email addr"; content:"|40|"; \ >> flow:established,from_server; flowint:something,notset; >> flowint:something,=,1; sid:10; priority:3;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got more email addrs"; content:"|40|"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,+,1; sid:11; priority:2;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"Got too many email addrs!"; content:"|40|"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,>,9; sid:12; priority:1; classtype:policy-violation;) >> >> >> alerts I get : >> >> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 >> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs [**] >> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58158 >> >> If I put some '#' symbols between the emails in the file so that it gets >> about 9K big and I fetch it I get these alerts : >> >> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs [**] >> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs [**] >> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs [**] >> [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> Y.Y.Y.Y:58166 >> >> > > Hi Nikolay, > > > Can you please post this as a bug - please be detailed (as you were in > your 2 previous e-mails). > Personally i think here sid 11 is the problem , may be it does not > count/increment correctly.... > thanks > > > -- > Peter Manev > > > Yes I will post this as a bug. But I've just found a much simpler case. > > Let's for example have only this rule in suricata : > > alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) > > Then on a monitored machine from the $HOME_NET range I do : > > echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 > > And on different host I do : > > nc testserver 6666 > > This gets the ten @ chars transferred, and I get only one alert. > But for example if I echo more @ chars, like 5000 or something, I get 3-6 > alerts. > I have to check what is actually the number of packets with payload, > probably the rule > is matched once per packet? But this could not explain that I get > different number of alerts on different runs. > > > Sure, that works too :) (as an example i mean). Thanks -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120212/78ab5d7b/attachment-0001.html From petermanev at gmail.com Sun Feb 12 11:54:51 2012 From: petermanev at gmail.com (Peter Manev) Date: Sun, 12 Feb 2012 17:54:51 +0100 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: References: <4F3527B8.6020307@inliniac.net> Message-ID: <4F37EEDB.1010407@gmail.com> On 2/12/2012 1:04 AM, Josh White wrote: > That would work, I was originally thinking even an option to append > the interface name and have have multiple stats files like > stats.log.em1 or the reverse em1.stats.log. However if it was more of > a csv format then it would be easier to graph in some cases. > > On Fri, Feb 10, 2012 at 9:20 AM, Victor Julien > wrote: > > On 02/10/2012 02:44 AM, Peter Manev wrote: > > Hi, > > > > I don't think this is possible(in suri), you could of course use > some > > bash/perl/your choice of scripting to achieve that. > > It's indeed not possible right now. I'm a bit torn on it, as I see use > for both cases. Ideally we're have it both simultaneously. Maybe we > should an easily parseble (csv or something) output option. > Actually I am very fond of the csv availability (in yaml maybe? ) for the different log files output. I agree with Josh - there are plenty of tools that make graphing possible (using csv files) and it would also come in handy for GeoIP visualization. > Cheers, > Victor > > > > > Thanks > > > > On Thu, Feb 9, 2012 at 2:33 AM, Josh White > > >> wrote: > > > > When I run Suri to monitor multiple interfaces like "suricata -c > > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the > stats.log file > > has multiple entries for each stat. "one entry for each > interface > > being monitored" > > > > Is there an easy way to consolidate the stats so all the > interface > > stats are consolidated? > > > > Josh > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > > > > > -- > > Peter Manev > > > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120212/eb24b05f/attachment.html From toastyguy at gmail.com Sun Feb 12 14:23:43 2012 From: toastyguy at gmail.com (toasty) Date: Sun, 12 Feb 2012 19:23:43 +0000 Subject: [Oisf-users] config testing Message-ID: Hi, has anyone come across a way to validate a [suricata.yaml] config kind of like how snort had the -T option? Tried looking around for this some, and while there might be something in the unit tests, figured asking might quicker than going through them all... ...use-case I have in mind is for when doing automated updates, and wanting to test that a new ruleset won't result in just killing the sensor (would rather have it tell me that it was not going to work). Thanks! --james From jonkman at emergingthreatspro.com Sun Feb 12 15:25:48 2012 From: jonkman at emergingthreatspro.com (Matthew Jonkman) Date: Sun, 12 Feb 2012 15:25:48 -0500 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: <4F37EEDB.1010407@gmail.com> References: <4F3527B8.6020307@inliniac.net> <4F37EEDB.1010407@gmail.com> Message-ID: How about we just define a log format like you can for an apache customlog? Then we only have to solve the problem once.... Matt On Feb 12, 2012, at 11:54 AM, Peter Manev wrote: > On 2/12/2012 1:04 AM, Josh White wrote: >> That would work, I was originally thinking even an option to append the interface name and have have multiple stats files like stats.log.em1 or the reverse em1.stats.log. However if it was more of a csv format then it would be easier to graph in some cases. >> >> On Fri, Feb 10, 2012 at 9:20 AM, Victor Julien wrote: >> On 02/10/2012 02:44 AM, Peter Manev wrote: >> > Hi, >> > >> > I don't think this is possible(in suri), you could of course use some >> > bash/perl/your choice of scripting to achieve that. >> >> It's indeed not possible right now. I'm a bit torn on it, as I see use >> for both cases. Ideally we're have it both simultaneously. Maybe we >> should an easily parseble (csv or something) output option. >> > Actually I am very fond of the csv availability (in yaml maybe? ) for the different log files output. I agree with Josh - there are plenty of tools that make graphing possible (using csv files) and it would also come in handy for GeoIP visualization. > > >> Cheers, >> Victor >> >> > >> > Thanks >> > >> > On Thu, Feb 9, 2012 at 2:33 AM, Josh White > > > wrote: >> > >> > When I run Suri to monitor multiple interfaces like "suricata -c >> > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log file >> > has multiple entries for each stat. "one entry for each interface >> > being monitored" >> > >> > Is there an easy way to consolidate the stats so all the interface >> > stats are consolidated? >> > >> > Josh >> > >> > _______________________________________________ >> > Oisf-users mailing list >> > Oisf-users at openinfosecfoundation.org >> > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> > >> > >> > >> > >> > -- >> > Peter Manev >> > >> > >> > _______________________________________________ >> > Oisf-users mailing list >> > Oisf-users at openinfosecfoundation.org >> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >> >> -- >> --------------------------------------------- >> Victor Julien >> http://www.inliniac.net/ >> PGP: http://www.inliniac.net/victorjulien.asc >> --------------------------------------------- >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >> >> >> >> _______________________________________________ >> Oisf-users mailing list >> >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > -- > Regards, > Peter Manev > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users ---------------------------------------------------- Matt Jonkman Emerging Threats Pro Open Information Security Foundation (OISF) Phone 866-504-2523 x110 http://www.emergingthreatspro.com http://www.openinfosecfoundation.org ---------------------------------------------------- From edwardfjellskaal at gmail.com Mon Feb 13 04:09:07 2012 From: edwardfjellskaal at gmail.com (=?ISO-8859-1?Q?Edward_Fjellsk=E5l?=) Date: Mon, 13 Feb 2012 10:09:07 +0100 Subject: [Oisf-users] config testing In-Reply-To: References: Message-ID: There might be a "new and improved" way to do this, but here is a snippet from more or less how I checked it: ----8<---- .... rm $SURILOGDIR/* $ENGINE --runmode single -c $SURIYAML -r $TESTPCAP ERRORS=`grep "ERRCODE:" $SURILOGDIR/suricata.log | wc -l` if [ $ERRORS != 0 ]; then grep "ERRCODE:" $SURILOGDIR/suricata.log exit 1 fi .... ----8<---- E On Sun, Feb 12, 2012 at 8:23 PM, toasty wrote: > Hi, has anyone come across a way to validate a [suricata.yaml] config > kind of like how snort had the -T option? Tried looking around for > this some, and while there might be something in the unit tests, > figured asking might quicker than going through them all... > > ...use-case I have in mind is for when doing automated updates, and > wanting to test that a new ruleset won't result in just killing the > sensor (would rather have it tell me that it was not going to work). > > Thanks! > > > --james > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- Edward Bjarte Fjellsk?l Senior Security Analyst http://www.gamelinux.org/ From josh at securemind.org Mon Feb 13 14:06:36 2012 From: josh at securemind.org (Josh White) Date: Mon, 13 Feb 2012 14:06:36 -0500 Subject: [Oisf-users] [oisf-users] Consolidating Stats File Results from Multiple Interface Monitoring In-Reply-To: References: <4F3527B8.6020307@inliniac.net> <4F37EEDB.1010407@gmail.com> Message-ID: I like that idea.. this way dependent on the specific format of the organization, ie: CEE, CEF, etc. they can set up there own interface to whatever SIM they are using. Otherwise I fear we'll be in the mess of supporting "connectors" to different systems. On Sun, Feb 12, 2012 at 3:25 PM, Matthew Jonkman < jonkman at emergingthreatspro.com> wrote: > How about we just define a log format like you can for an apache > customlog? Then we only have to solve the problem once.... > > Matt > > > On Feb 12, 2012, at 11:54 AM, Peter Manev wrote: > > > On 2/12/2012 1:04 AM, Josh White wrote: > >> That would work, I was originally thinking even an option to append the > interface name and have have multiple stats files like stats.log.em1 or the > reverse em1.stats.log. However if it was more of a csv format then it would > be easier to graph in some cases. > >> > >> On Fri, Feb 10, 2012 at 9:20 AM, Victor Julien > wrote: > >> On 02/10/2012 02:44 AM, Peter Manev wrote: > >> > Hi, > >> > > >> > I don't think this is possible(in suri), you could of course use some > >> > bash/perl/your choice of scripting to achieve that. > >> > >> It's indeed not possible right now. I'm a bit torn on it, as I see use > >> for both cases. Ideally we're have it both simultaneously. Maybe we > >> should an easily parseble (csv or something) output option. > >> > > Actually I am very fond of the csv availability (in yaml maybe? ) for > the different log files output. I agree with Josh - there are plenty of > tools that make graphing possible (using csv files) and it would also come > in handy for GeoIP visualization. > > > > > >> Cheers, > >> Victor > >> > >> > > >> > Thanks > >> > > >> > On Thu, Feb 9, 2012 at 2:33 AM, Josh White >> > > wrote: > >> > > >> > When I run Suri to monitor multiple interfaces like "suricata -c > >> > /etc/suricata/suricata.yaml -i em1 -i em2 -i em3" the stats.log > file > >> > has multiple entries for each stat. "one entry for each interface > >> > being monitored" > >> > > >> > Is there an easy way to consolidate the stats so all the interface > >> > stats are consolidated? > >> > > >> > Josh > >> > > >> > _______________________________________________ > >> > Oisf-users mailing list > >> > Oisf-users at openinfosecfoundation.org > >> > > >> > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >> > > >> > > >> > > >> > > >> > -- > >> > Peter Manev > >> > > >> > > >> > _______________________________________________ > >> > Oisf-users mailing list > >> > Oisf-users at openinfosecfoundation.org > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >> > >> > >> -- > >> --------------------------------------------- > >> Victor Julien > >> http://www.inliniac.net/ > >> PGP: http://www.inliniac.net/victorjulien.asc > >> --------------------------------------------- > >> > >> _______________________________________________ > >> Oisf-users mailing list > >> Oisf-users at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >> > >> > >> > >> > >> _______________________________________________ > >> Oisf-users mailing list > >> > >> Oisf-users at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > -- > > Regards, > > Peter Manev > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > ---------------------------------------------------- > Matt Jonkman > Emerging Threats Pro > Open Information Security Foundation (OISF) > Phone 866-504-2523 x110 > http://www.emergingthreatspro.com > http://www.openinfosecfoundation.org > ---------------------------------------------------- > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120213/ec06b379/attachment.html From toastyguy at gmail.com Mon Feb 13 19:13:10 2012 From: toastyguy at gmail.com (toasty) Date: Tue, 14 Feb 2012 00:13:10 +0000 Subject: [Oisf-users] config testing In-Reply-To: References: Message-ID: E, Thanks - I think using a basic pcap pretty well does what I was hoping! --james On Mon, Feb 13, 2012 at 9:09 AM, Edward Fjellsk?l wrote: > There might be a "new and improved" way to do this, but here is a > snippet from more or less how I checked it: > > ----8<---- > .... > rm $SURILOGDIR/* > $ENGINE --runmode single -c $SURIYAML -r $TESTPCAP > ERRORS=`grep "ERRCODE:" $SURILOGDIR/suricata.log | wc -l` > if [ $ERRORS != 0 ]; then > ? grep "ERRCODE:" $SURILOGDIR/suricata.log > ? exit 1 > fi > .... > ----8<---- > > > E > > > > On Sun, Feb 12, 2012 at 8:23 PM, toasty wrote: >> Hi, has anyone come across a way to validate a [suricata.yaml] config >> kind of like how snort had the -T option? Tried looking around for >> this some, and while there might be something in the unit tests, >> figured asking might quicker than going through them all... >> >> ...use-case I have in mind is for when doing automated updates, and >> wanting to test that a new ruleset won't result in just killing the >> sensor (would rather have it tell me that it was not going to work). >> >> Thanks! >> >> >> --james >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > -- > Edward Bjarte Fjellsk?l > Senior Security Analyst > http://www.gamelinux.org/ -- --james From victor at inliniac.net Tue Feb 14 03:44:17 2012 From: victor at inliniac.net (Victor Julien) Date: Tue, 14 Feb 2012 09:44:17 +0100 Subject: [Oisf-users] config testing In-Reply-To: References: Message-ID: <4F3A1EE1.5060001@inliniac.net> You can add the --init-errors-fatal option as well. Still requires you to pass the pcap though. Cheers, Victor On 02/14/2012 01:13 AM, toasty wrote: > E, > Thanks - I think using a basic pcap pretty well does what I was hoping! > > --james > > On Mon, Feb 13, 2012 at 9:09 AM, Edward Fjellsk?l > wrote: >> There might be a "new and improved" way to do this, but here is a >> snippet from more or less how I checked it: >> >> ----8<---- >> .... >> rm $SURILOGDIR/* >> $ENGINE --runmode single -c $SURIYAML -r $TESTPCAP >> ERRORS=`grep "ERRCODE:" $SURILOGDIR/suricata.log | wc -l` >> if [ $ERRORS != 0 ]; then >> grep "ERRCODE:" $SURILOGDIR/suricata.log >> exit 1 >> fi >> .... >> ----8<---- >> >> >> E >> >> >> >> On Sun, Feb 12, 2012 at 8:23 PM, toasty wrote: >>> Hi, has anyone come across a way to validate a [suricata.yaml] config >>> kind of like how snort had the -T option? Tried looking around for >>> this some, and while there might be something in the unit tests, >>> figured asking might quicker than going through them all... >>> >>> ...use-case I have in mind is for when doing automated updates, and >>> wanting to test that a new ruleset won't result in just killing the >>> sensor (would rather have it tell me that it was not going to work). >>> >>> Thanks! >>> >>> >>> --james >>> _______________________________________________ >>> Oisf-users mailing list >>> Oisf-users at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >> >> >> -- >> Edward Bjarte Fjellsk?l >> Senior Security Analyst >> http://www.gamelinux.org/ > > > -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From victor at inliniac.net Tue Feb 14 04:21:55 2012 From: victor at inliniac.net (Victor Julien) Date: Tue, 14 Feb 2012 10:21:55 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> Message-ID: <4F3A27B3.8020608@inliniac.net> On 02/12/2012 08:15 AM, Nikolay Denev wrote: > > On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: > >> >> >> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev > > wrote: >> >> >> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >> >>> >>> >>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev >> > wrote: >>> >>> >>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>> >>>> >>>> >>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >>>> > wrote: >>>> >>>> >>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>> >>>>> >>>>> >>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >>>>> > wrote: >>>>> >>>>> >>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>> >>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>> > >>>>> >> Hi all, >>>>> >> >>>>> >> It's probably stupid question and I'm missing >>>>> something but I don't seem to be able >>>>> >> to generate alert immediately when for example a >>>>> given string is found inside a TCP stream. >>>>> >> When the TCP connection closes, suricata >>>>> immediately prints the alert in fast.log. >>>>> >> How can I make the alert be generated >>>>> immediately when the rule condition is matched? >>>>> >> >>>>> >> Also I don't know if its because of this I don't >>>>> seem to be able to trigger the rule to match >>>>> several times on the same stream, >>>>> >> while I have the string that should fire the >>>>> alert several times in the stream. >>>>> >> >>>>> >> Here's an example : >>>>> >> >>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>> >> (msg:"got one"; content:"something"; >>>>> flowint:something,notset; flowint:something,=,1; >>>>> sid:10;) >>>>> >> >>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>> >> (msg:"got five or more"; >>>>> content:"something"; flowint:something,isset; >>>>> flowint:something,+,1; flowint:something,>,5; sid:11;) >>>>> >> >>>>> >> This never works, I just have the first rule >>>>> fire once when the TCP session is terminated. >>>>> >> >>>>> >> >>>>> >> P.S.: As a side note the wiki should be updated >>>>> to include probably "sid"s for the rules, as >>>>> currently when I try to run the examples >>>>> >> suricata complains about duplicated rules. >>>>> >> >>>>> >> Thanks, >>>>> >> >>>>> > >>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>> >>>>> This seems to work : >>>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got one"; content:"something"; >>>>> flowint:something,notset; flowint:something,=,1; >>>>> noalert; sid:10; priority: 1;) >>>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got more"; content:"something"; >>>>> flowint:something,isset; flowint:something,+,1; >>>>> noalert; sid:11; priority: 2;) >>>>> >>>>> >>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>> (msg:"got too many"; content:"something"; >>>>> flowint:something,isset; flowint:something,>,2; >>>>> sid:12; priority: 3;) >>>>> >>>>> >>>>> _______________________________________________ >>>>> Oisf-users mailing list >>>>> Oisf-users at openinfosecfoundation.org >>>>> >>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>> >>>>> >>>>> >>>>> Hi Nikolay, >>>>> I think this is the way it is supposed to work. (last >>>>> example, by you). >>>>> >>>>> When you take out "noalert" form sid 11 - does it fire ? >>>>> >>>>> And are these the only rules that are loaded in terms >>>>> of flowint or you have others before that? >>>>> >>>>> thanks >>>>> >>>>> >>>>> >>>>> -- >>>>> Peter Manev >>>> >>>> >>>> Yes, It fires, the problem I have is that it doesn't >>>> fire for each occurence of "content". >>>> Is alert supposed to fire once per packet if it matches, >>>> or for each match in the stream? >>>> >>>> For example now I'm using these rules to catch if there >>>> are more than some defined amount of email addresses in >>>> a given stream : >>>> >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got one email addr"; content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; >>>> flowint:something,notset; flowint:something,=,1; sid:10; >>>> priority:3; noalert;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got more email addrs"; content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; >>>> flowint:something,isset; flowint:something,+,1; sid:11; >>>> priority:2; noalert;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"Got too many email addrs!"; >>>> content:"|40|"; >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>> flow:established,from_server; >>>> flowint:something,isset; flowint:something,>,10; sid:12; >>>> priority:1; classtype:policy-violation;) >>>> >>>> >>>> This for example works, but would not match for a simple >>>> plain text file with 10 email adresses, I need to have >>>> maybe 40-50 or more for this to match. >>>> Maybe I'm missing something? >>>> >>>> And yes, these are my only rules that I'm testing with. >>>> No other rules with or without flowint whatsoever. >>>> >>>> >>>> Hi , >>>> Just so I understand you correctly - you have a text file >>>> (in the stream) and in that text file you have 10 e-mail >>>> addresses and it wold not fire. correct ? >>>> >>>> >>>> thanks >>>> >>>> >>>> -- >>>> Peter Manev >>> >>> Exactly. >>> >>> For example if I try to fetch the file emails.txt via http >>> which has the following content : >>> >>> # cat emails.txt >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> >>> $ curl http://testserver/emails.txt >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> edin at email.com >>> $ >>> >>> And I also remove the "noalert" option from the rules, this >>> is what I get in fast.log : >>> >>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr >>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >>> -> Y.Y.Y.Y:57923 >>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email >>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} >>> X.X.X.X:80 -> Y.Y.Y.Y:57923 >>> >>> >>> If I change the third rule to fire if the flowint var is more >>> than 1, it is being triggered. >>> >>> If I insert some random data between the email addresses in >>> the text file, then I get 4 maybe 5 matches. Doesn't it have >>> to match all 10 of them? >>> >>> >>> 1. What happens if you take out the PCRE expressions from all >>> the rules ? >>> 2. sid:12 - should not fire because you have >10 , and there are >>> exactly 10 e-mails in the file >>> 3. how big is the stream itself? i think it is below 2KB, correct? >>> 4. is the PCRE matching the e-mails, under the unix shell ? >>> 5. yes i think you should get more sid:11 alerts - but first lets >>> investigate the above 4. >>> >>> thanks >>> >>> -- >>> Peter Manev >> >> The file with only the 10 emails is 160 bytes. Even without pcre I >> get the same result : >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got one email addr"; content:"|40|"; \ >> flow:established,from_server; flowint:something,notset; >> flowint:something,=,1; sid:10; priority:3;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"got more email addrs"; content:"|40|"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,+,1; sid:11; priority:2;) >> >> alert tcp $HOME_NET 80 -> any any \ >> (msg:"Got too many email addrs!"; content:"|40|"; \ >> flow:established,from_server; flowint:something,isset; >> flowint:something,>,9; sid:12; priority:1; >> classtype:policy-violation;) >> >> >> alerts I get : >> >> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58158 >> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58158 >> >> If I put some '#' symbols between the emails in the file so that >> it gets about 9K big and I fetch it I get these alerts : >> >> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58166 >> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> Y.Y.Y.Y:58166 >> >> >> >> Hi Nikolay, >> >> >> Can you please post this as a bug - please be detailed (as you were in >> your 2 previous e-mails). >> Personally i think here sid 11 is the problem , may be it does not >> count/increment correctly.... >> thanks >> >> >> -- >> Peter Manev > > Yes I will post this as a bug. But I've just found a much simpler case. > > Let's for example have only this rule in suricata : > > alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) > > Then on a monitored machine from the $HOME_NET range I do : > > echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 > > And on different host I do : > > nc testserver 6666 > > This gets the ten @ chars transferred, and I get only one alert. > But for example if I echo more @ chars, like 5000 or something, I get > 3-6 alerts. > I have to check what is actually the number of packets with payload, > probably the rule > is matched once per packet? But this could not explain that I get > different number of alerts on different runs. The behavior is by design. TCP data by default is inspected in the stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected at once. Suricata will not try to find every possible match in a payload, but just one. The reason you get more alerts if you increase the payload significantly, is that the stream is inspected in chunks. The size of those chunks is determined by your stream toserver_chunk_size setting. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From anoopsaldanha at gmail.com Tue Feb 14 04:22:37 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Tue, 14 Feb 2012 14:52:37 +0530 Subject: [Oisf-users] New MPM available Message-ID: Hello all, We have a new MPM available in our codebase - "ac-bs". This provides compression that's pretty close to ac-gfbs, while performing better than ac-gfbs. To use this mpm, set "mpm-algo: ac-bs" in the conf file. Would appreciate performance numbers with both "sgh-mpm-context:full" and "sgh-mpm-context:single" To give an explanation on what "sgh-mpm-context" and the params "full" and "single" mean, these refer to how we set up mpm contexts. "single" indicates that we use a single context for all the patterns in the engine. "full" indicates that we split the patterns into many mpm contexts, one mpm context per signature group head(sgh). To use "full" with a sufficiently decent ruleset(say > 10k rules with a decent no of patterns) would require a lot of memory, running into a couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case of "ac". "single" solves this with a single context and hence the smaller memory footprint for the engine. If the machine has sufficient memory, "full" is suggested as it provides much better performance than "single", albeit at the cost of increased memory consumption. More of a available_memory vs performance scenario. Looking forward to some performance/memory feedback/benchmarks with this mpm from the community. *mpm - multi pattern matcher *sgh - signature group head -- Anoop Saldanha From petermanev at gmail.com Tue Feb 14 04:29:50 2012 From: petermanev at gmail.com (Peter Manev) Date: Tue, 14 Feb 2012 10:29:50 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <4F3A27B3.8020608@inliniac.net> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> Message-ID: On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien wrote: > On 02/12/2012 08:15 AM, Nikolay Denev wrote: > > > > On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: > > > >> > >> > >> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev >> > wrote: > >> > >> > >> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: > >> > >>> > >>> > >>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev >>> > wrote: > >>> > >>> > >>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: > >>> > >>>> > >>>> > >>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev > >>>> > wrote: > >>>> > >>>> > >>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: > >>>> > >>>>> > >>>>> > >>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev > >>>>> > wrote: > >>>>> > >>>>> > >>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: > >>>>> > >>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: > >>>>> > > >>>>> >> Hi all, > >>>>> >> > >>>>> >> It's probably stupid question and I'm missing > >>>>> something but I don't seem to be able > >>>>> >> to generate alert immediately when for example a > >>>>> given string is found inside a TCP stream. > >>>>> >> When the TCP connection closes, suricata > >>>>> immediately prints the alert in fast.log. > >>>>> >> How can I make the alert be generated > >>>>> immediately when the rule condition is matched? > >>>>> >> > >>>>> >> Also I don't know if its because of this I don't > >>>>> seem to be able to trigger the rule to match > >>>>> several times on the same stream, > >>>>> >> while I have the string that should fire the > >>>>> alert several times in the stream. > >>>>> >> > >>>>> >> Here's an example : > >>>>> >> > >>>>> >> alert tcp $HOME_NET 6666 -> any any \ > >>>>> >> (msg:"got one"; content:"something"; > >>>>> flowint:something,notset; flowint:something,=,1; > >>>>> sid:10;) > >>>>> >> > >>>>> >> alert tcp $HOME_NET 6666 -> any any \ > >>>>> >> (msg:"got five or more"; > >>>>> content:"something"; flowint:something,isset; > >>>>> flowint:something,+,1; flowint:something,>,5; > sid:11;) > >>>>> >> > >>>>> >> This never works, I just have the first rule > >>>>> fire once when the TCP session is terminated. > >>>>> >> > >>>>> >> > >>>>> >> P.S.: As a side note the wiki should be updated > >>>>> to include probably "sid"s for the rules, as > >>>>> currently when I try to run the examples > >>>>> >> suricata complains about duplicated rules. > >>>>> >> > >>>>> >> Thanks, > >>>>> >> > >>>>> > > >>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. > >>>>> > >>>>> This seems to work : > >>>>> > >>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>> (msg:"got one"; content:"something"; > >>>>> flowint:something,notset; flowint:something,=,1; > >>>>> noalert; sid:10; priority: 1;) > >>>>> > >>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>> (msg:"got more"; content:"something"; > >>>>> flowint:something,isset; flowint:something,+,1; > >>>>> noalert; sid:11; priority: 2;) > >>>>> > >>>>> > >>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>> (msg:"got too many"; content:"something"; > >>>>> flowint:something,isset; flowint:something,>,2; > >>>>> sid:12; priority: 3;) > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Oisf-users mailing list > >>>>> Oisf-users at openinfosecfoundation.org > >>>>> > >>>>> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >>>>> > >>>>> > >>>>> > >>>>> Hi Nikolay, > >>>>> I think this is the way it is supposed to work. (last > >>>>> example, by you). > >>>>> > >>>>> When you take out "noalert" form sid 11 - does it fire ? > >>>>> > >>>>> And are these the only rules that are loaded in terms > >>>>> of flowint or you have others before that? > >>>>> > >>>>> thanks > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Peter Manev > >>>> > >>>> > >>>> Yes, It fires, the problem I have is that it doesn't > >>>> fire for each occurence of "content". > >>>> Is alert supposed to fire once per packet if it matches, > >>>> or for each match in the stream? > >>>> > >>>> For example now I'm using these rules to catch if there > >>>> are more than some defined amount of email addresses in > >>>> a given stream : > >>>> > >>>> > >>>> alert tcp $HOME_NET 80 -> any any \ > >>>> (msg:"got one email addr"; content:"|40|"; > >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>> flow:established,from_server; > >>>> flowint:something,notset; flowint:something,=,1; sid:10; > >>>> priority:3; noalert;) > >>>> > >>>> alert tcp $HOME_NET 80 -> any any \ > >>>> (msg:"got more email addrs"; content:"|40|"; > >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>> flow:established,from_server; > >>>> flowint:something,isset; flowint:something,+,1; sid:11; > >>>> priority:2; noalert;) > >>>> > >>>> alert tcp $HOME_NET 80 -> any any \ > >>>> (msg:"Got too many email addrs!"; > >>>> content:"|40|"; > >>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>> flow:established,from_server; > >>>> flowint:something,isset; flowint:something,>,10; sid:12; > >>>> priority:1; classtype:policy-violation;) > >>>> > >>>> > >>>> This for example works, but would not match for a simple > >>>> plain text file with 10 email adresses, I need to have > >>>> maybe 40-50 or more for this to match. > >>>> Maybe I'm missing something? > >>>> > >>>> And yes, these are my only rules that I'm testing with. > >>>> No other rules with or without flowint whatsoever. > >>>> > >>>> > >>>> Hi , > >>>> Just so I understand you correctly - you have a text file > >>>> (in the stream) and in that text file you have 10 e-mail > >>>> addresses and it wold not fire. correct ? > >>>> > >>>> > >>>> thanks > >>>> > >>>> > >>>> -- > >>>> Peter Manev > >>> > >>> Exactly. > >>> > >>> For example if I try to fetch the file emails.txt via http > >>> which has the following content : > >>> > >>> # cat emails.txt > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> > >>> $ curl http://testserver/emails.txt > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> edin at email.com > >>> $ > >>> > >>> And I also remove the "noalert" option from the rules, this > >>> is what I get in fast.log : > >>> > >>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr > >>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 > >>> -> Y.Y.Y.Y:57923 > >>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email > >>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} > >>> X.X.X.X:80 -> Y.Y.Y.Y:57923 > >>> > >>> > >>> If I change the third rule to fire if the flowint var is more > >>> than 1, it is being triggered. > >>> > >>> If I insert some random data between the email addresses in > >>> the text file, then I get 4 maybe 5 matches. Doesn't it have > >>> to match all 10 of them? > >>> > >>> > >>> 1. What happens if you take out the PCRE expressions from all > >>> the rules ? > >>> 2. sid:12 - should not fire because you have >10 , and there are > >>> exactly 10 e-mails in the file > >>> 3. how big is the stream itself? i think it is below 2KB, correct? > >>> 4. is the PCRE matching the e-mails, under the unix shell ? > >>> 5. yes i think you should get more sid:11 alerts - but first lets > >>> investigate the above 4. > >>> > >>> thanks > >>> > >>> -- > >>> Peter Manev > >> > >> The file with only the 10 emails is 160 bytes. Even without pcre I > >> get the same result : > >> > >> alert tcp $HOME_NET 80 -> any any \ > >> (msg:"got one email addr"; content:"|40|"; \ > >> flow:established,from_server; flowint:something,notset; > >> flowint:something,=,1; sid:10; priority:3;) > >> > >> alert tcp $HOME_NET 80 -> any any \ > >> (msg:"got more email addrs"; content:"|40|"; \ > >> flow:established,from_server; flowint:something,isset; > >> flowint:something,+,1; sid:11; priority:2;) > >> > >> alert tcp $HOME_NET 80 -> any any \ > >> (msg:"Got too many email addrs!"; content:"|40|"; \ > >> flow:established,from_server; flowint:something,isset; > >> flowint:something,>,9; sid:12; priority:1; > >> classtype:policy-violation;) > >> > >> > >> alerts I get : > >> > >> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] > >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58158 > >> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs > >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58158 > >> > >> If I put some '#' symbols between the emails in the file so that > >> it gets about 9K big and I fetch it I get these alerts : > >> > >> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] > >> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58166 > >> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs > >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58166 > >> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs > >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58166 > >> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs > >> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >> Y.Y.Y.Y:58166 > >> > >> > >> > >> Hi Nikolay, > >> > >> > >> Can you please post this as a bug - please be detailed (as you were in > >> your 2 previous e-mails). > >> Personally i think here sid 11 is the problem , may be it does not > >> count/increment correctly.... > >> thanks > >> > >> > >> -- > >> Peter Manev > > > > Yes I will post this as a bug. But I've just found a much simpler case. > > > > Let's for example have only this rule in suricata : > > > > alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) > > > > Then on a monitored machine from the $HOME_NET range I do : > > > > echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 > > > > And on different host I do : > > > > nc testserver 6666 > > > > This gets the ten @ chars transferred, and I get only one alert. > > But for example if I echo more @ chars, like 5000 or something, I get > > 3-6 alerts. > > I have to check what is actually the number of packets with payload, > > probably the rule > > is matched once per packet? But this could not explain that I get > > different number of alerts on different runs. > > The behavior is by design. TCP data by default is inspected in the > stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected > at once. > Suricata will not try to find every possible match in a > payload, but just one. > That's good to know - clears out a few questions of mine.... but then a PCRE (matching on 10 "@") should match all of them - correct? having in mind they are in the same "chunk". > > The reason you get more alerts if you increase the payload > significantly, is that the stream is inspected in chunks. The size of > those chunks is determined by your stream toserver_chunk_size setting. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120214/abbe5173/attachment-0001.html From victor at inliniac.net Tue Feb 14 04:44:30 2012 From: victor at inliniac.net (Victor Julien) Date: Tue, 14 Feb 2012 10:44:30 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> Message-ID: <4F3A2CFE.5070302@inliniac.net> On 02/14/2012 10:29 AM, Peter Manev wrote: > > Let's for example have only this rule in suricata : > > > > alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) > > > > Then on a monitored machine from the $HOME_NET range I do : > > > > echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 > > > > And on different host I do : > > > > nc testserver 6666 > > > > This gets the ten @ chars transferred, and I get only one alert. > > But for example if I echo more @ chars, like 5000 or something, I get > > 3-6 alerts. > > I have to check what is actually the number of packets with payload, > > probably the rule > > is matched once per packet? But this could not explain that I get > > different number of alerts on different runs. > > The behavior is by design. TCP data by default is inspected in the > stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected > at once. > > > > Suricata will not try to find every possible match in a > payload, but just one. > > That's good to know - clears out a few questions of mine.... > but then a PCRE (matching on 10 "@") should match all of them - correct? > having in mind they are in the same "chunk". Right, but it will be an expensive rule :) -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From victor at inliniac.net Tue Feb 14 04:47:38 2012 From: victor at inliniac.net (Victor Julien) Date: Tue, 14 Feb 2012 10:47:38 +0100 Subject: [Oisf-users] New MPM available In-Reply-To: References: Message-ID: <4F3A2DBA.9060009@inliniac.net> On 02/14/2012 10:22 AM, Anoop Saldanha wrote: > Hello all, > > We have a new MPM available in our codebase - "ac-bs". This provides > compression that's pretty close to ac-gfbs, while performing better > than ac-gfbs. > > To use this mpm, set > > "mpm-algo: ac-bs" in the conf file. > > Would appreciate performance numbers with both > > "sgh-mpm-context:full" > and > "sgh-mpm-context:single" > > To give an explanation on what "sgh-mpm-context" and the params "full" > and "single" mean, these refer to how we set up mpm contexts. > "single" indicates that we use a single context for all the patterns > in the engine. "full" indicates that we split the patterns into many > mpm contexts, one mpm context per signature group head(sgh). > > To use "full" with a sufficiently decent ruleset(say > 10k rules with > a decent no of patterns) would require a lot of memory, running into a > couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case > of "ac". "single" solves this with a single context and hence the > smaller memory footprint for the engine. > > If the machine has sufficient memory, "full" is suggested as it > provides much better performance than "single", albeit at the cost of > increased memory consumption. More of a available_memory vs > performance scenario. > > Looking forward to some performance/memory feedback/benchmarks with > this mpm from the community. So far from what I have seen, in a default et ruleset with the default suricata.yaml, "ac" is faster than "ac-bs". It would be interesting to set the "detect-engine.profile" to high with "ac-bs", as that settings increases the number of rule groups (sgh). -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From ndenev at gmail.com Tue Feb 14 08:49:24 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Tue, 14 Feb 2012 15:49:24 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <4F3A27B3.8020608@inliniac.net> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> Message-ID: <6428A35C-AA57-479B-B190-F6AD471AD210@gmail.com> On Feb 14, 2012, at 11:21 AM, Victor Julien wrote: > On 02/12/2012 08:15 AM, Nikolay Denev wrote: >> >> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: >> >>> >>> >>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev >> > wrote: >>> >>> >>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >>> >>>> >>>> >>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev >>> > wrote: >>>> >>>> >>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>>> >>>>> >>>>> >>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >>>>> > wrote: >>>>> >>>>> >>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >>>>>> > wrote: >>>>>> >>>>>> >>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>>> >>>>>>> On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> It's probably stupid question and I'm missing >>>>>> something but I don't seem to be able >>>>>>>> to generate alert immediately when for example a >>>>>> given string is found inside a TCP stream. >>>>>>>> When the TCP connection closes, suricata >>>>>> immediately prints the alert in fast.log. >>>>>>>> How can I make the alert be generated >>>>>> immediately when the rule condition is matched? >>>>>>>> >>>>>>>> Also I don't know if its because of this I don't >>>>>> seem to be able to trigger the rule to match >>>>>> several times on the same stream, >>>>>>>> while I have the string that should fire the >>>>>> alert several times in the stream. >>>>>>>> >>>>>>>> Here's an example : >>>>>>>> >>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> (msg:"got one"; content:"something"; >>>>>> flowint:something,notset; flowint:something,=,1; >>>>>> sid:10;) >>>>>>>> >>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> (msg:"got five or more"; >>>>>> content:"something"; flowint:something,isset; >>>>>> flowint:something,+,1; flowint:something,>,5; sid:11;) >>>>>>>> >>>>>>>> This never works, I just have the first rule >>>>>> fire once when the TCP session is terminated. >>>>>>>> >>>>>>>> >>>>>>>> P.S.: As a side note the wiki should be updated >>>>>> to include probably "sid"s for the rules, as >>>>>> currently when I try to run the examples >>>>>>>> suricata complains about duplicated rules. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>> >>>>>>> I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>>> >>>>>> This seems to work : >>>>>> >>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>> (msg:"got one"; content:"something"; >>>>>> flowint:something,notset; flowint:something,=,1; >>>>>> noalert; sid:10; priority: 1;) >>>>>> >>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>> (msg:"got more"; content:"something"; >>>>>> flowint:something,isset; flowint:something,+,1; >>>>>> noalert; sid:11; priority: 2;) >>>>>> >>>>>> >>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>> (msg:"got too many"; content:"something"; >>>>>> flowint:something,isset; flowint:something,>,2; >>>>>> sid:12; priority: 3;) >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Oisf-users mailing list >>>>>> Oisf-users at openinfosecfoundation.org >>>>>> >>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>>> >>>>>> >>>>>> >>>>>> Hi Nikolay, >>>>>> I think this is the way it is supposed to work. (last >>>>>> example, by you). >>>>>> >>>>>> When you take out "noalert" form sid 11 - does it fire ? >>>>>> >>>>>> And are these the only rules that are loaded in terms >>>>>> of flowint or you have others before that? >>>>>> >>>>>> thanks >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Peter Manev >>>>> >>>>> >>>>> Yes, It fires, the problem I have is that it doesn't >>>>> fire for each occurence of "content". >>>>> Is alert supposed to fire once per packet if it matches, >>>>> or for each match in the stream? >>>>> >>>>> For example now I'm using these rules to catch if there >>>>> are more than some defined amount of email addresses in >>>>> a given stream : >>>>> >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"got one email addr"; content:"|40|"; >>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>> flow:established,from_server; >>>>> flowint:something,notset; flowint:something,=,1; sid:10; >>>>> priority:3; noalert;) >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"got more email addrs"; content:"|40|"; >>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>> flow:established,from_server; >>>>> flowint:something,isset; flowint:something,+,1; sid:11; >>>>> priority:2; noalert;) >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"Got too many email addrs!"; >>>>> content:"|40|"; >>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>> flow:established,from_server; >>>>> flowint:something,isset; flowint:something,>,10; sid:12; >>>>> priority:1; classtype:policy-violation;) >>>>> >>>>> >>>>> This for example works, but would not match for a simple >>>>> plain text file with 10 email adresses, I need to have >>>>> maybe 40-50 or more for this to match. >>>>> Maybe I'm missing something? >>>>> >>>>> And yes, these are my only rules that I'm testing with. >>>>> No other rules with or without flowint whatsoever. >>>>> >>>>> >>>>> Hi , >>>>> Just so I understand you correctly - you have a text file >>>>> (in the stream) and in that text file you have 10 e-mail >>>>> addresses and it wold not fire. correct ? >>>>> >>>>> >>>>> thanks >>>>> >>>>> >>>>> -- >>>>> Peter Manev >>>> >>>> Exactly. >>>> >>>> For example if I try to fetch the file emails.txt via http >>>> which has the following content : >>>> >>>> # cat emails.txt >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> >>>> $ curl http://testserver/emails.txt >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> edin at email.com >>>> $ >>>> >>>> And I also remove the "noalert" option from the rules, this >>>> is what I get in fast.log : >>>> >>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr >>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >>>> -> Y.Y.Y.Y:57923 >>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email >>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} >>>> X.X.X.X:80 -> Y.Y.Y.Y:57923 >>>> >>>> >>>> If I change the third rule to fire if the flowint var is more >>>> than 1, it is being triggered. >>>> >>>> If I insert some random data between the email addresses in >>>> the text file, then I get 4 maybe 5 matches. Doesn't it have >>>> to match all 10 of them? >>>> >>>> >>>> 1. What happens if you take out the PCRE expressions from all >>>> the rules ? >>>> 2. sid:12 - should not fire because you have >10 , and there are >>>> exactly 10 e-mails in the file >>>> 3. how big is the stream itself? i think it is below 2KB, correct? >>>> 4. is the PCRE matching the e-mails, under the unix shell ? >>>> 5. yes i think you should get more sid:11 alerts - but first lets >>>> investigate the above 4. >>>> >>>> thanks >>>> >>>> -- >>>> Peter Manev >>> >>> The file with only the 10 emails is 160 bytes. Even without pcre I >>> get the same result : >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got one email addr"; content:"|40|"; \ >>> flow:established,from_server; flowint:something,notset; >>> flowint:something,=,1; sid:10; priority:3;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"got more email addrs"; content:"|40|"; \ >>> flow:established,from_server; flowint:something,isset; >>> flowint:something,+,1; sid:11; priority:2;) >>> >>> alert tcp $HOME_NET 80 -> any any \ >>> (msg:"Got too many email addrs!"; content:"|40|"; \ >>> flow:established,from_server; flowint:something,isset; >>> flowint:something,>,9; sid:12; priority:1; >>> classtype:policy-violation;) >>> >>> >>> alerts I get : >>> >>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] >>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58158 >>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs >>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58158 >>> >>> If I put some '#' symbols between the emails in the file so that >>> it gets about 9K big and I fetch it I get these alerts : >>> >>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] >>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58166 >>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs >>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58166 >>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs >>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58166 >>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs >>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>> Y.Y.Y.Y:58166 >>> >>> >>> >>> Hi Nikolay, >>> >>> >>> Can you please post this as a bug - please be detailed (as you were in >>> your 2 previous e-mails). >>> Personally i think here sid 11 is the problem , may be it does not >>> count/increment correctly.... >>> thanks >>> >>> >>> -- >>> Peter Manev >> >> Yes I will post this as a bug. But I've just found a much simpler case. >> >> Let's for example have only this rule in suricata : >> >> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) >> >> Then on a monitored machine from the $HOME_NET range I do : >> >> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 >> >> And on different host I do : >> >> nc testserver 6666 >> >> This gets the ten @ chars transferred, and I get only one alert. >> But for example if I echo more @ chars, like 5000 or something, I get >> 3-6 alerts. >> I have to check what is actually the number of packets with payload, >> probably the rule >> is matched once per packet? But this could not explain that I get >> different number of alerts on different runs. > > The behavior is by design. TCP data by default is inspected in the > stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected > at once. Suricata will not try to find every possible match in a > payload, but just one. > > The reason you get more alerts if you increase the payload > significantly, is that the stream is inspected in chunks. The size of > those chunks is determined by your stream toserver_chunk_size setting. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > I see now. I was suspecting something like this, and it makes sense performance wise. I was trying to create a rule that will fire only if more than let's say 50 email addresses are contained in the stream, maybe I'll try with lower threshold. Thanks, From petermanev at gmail.com Tue Feb 14 16:30:04 2012 From: petermanev at gmail.com (Peter Manev) Date: Tue, 14 Feb 2012 22:30:04 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <6428A35C-AA57-479B-B190-F6AD471AD210@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> <6428A35C-AA57-479B-B190-F6AD471AD210@gmail.com> Message-ID: <4F3AD25C.7040104@gmail.com> On 2/14/2012 2:49 PM, Nikolay Denev wrote: > On Feb 14, 2012, at 11:21 AM, Victor Julien wrote: > >> On 02/12/2012 08:15 AM, Nikolay Denev wrote: >>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: >>> >>>> >>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev>>> > wrote: >>>> >>>> >>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >>>> >>>>> >>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev>>>> > wrote: >>>>> >>>>> >>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>>>> >>>>>> >>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >>>>>> > wrote: >>>>>> >>>>>> >>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>>>> >>>>>>> >>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >>>>>>> > wrote: >>>>>>> >>>>>>> >>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>>>> >>>>>>>> On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> It's probably stupid question and I'm missing >>>>>>> something but I don't seem to be able >>>>>>>>> to generate alert immediately when for example a >>>>>>> given string is found inside a TCP stream. >>>>>>>>> When the TCP connection closes, suricata >>>>>>> immediately prints the alert in fast.log. >>>>>>>>> How can I make the alert be generated >>>>>>> immediately when the rule condition is matched? >>>>>>>>> Also I don't know if its because of this I don't >>>>>>> seem to be able to trigger the rule to match >>>>>>> several times on the same stream, >>>>>>>>> while I have the string that should fire the >>>>>>> alert several times in the stream. >>>>>>>>> Here's an example : >>>>>>>>> >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> (msg:"got one"; content:"something"; >>>>>>> flowint:something,notset; flowint:something,=,1; >>>>>>> sid:10;) >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> (msg:"got five or more"; >>>>>>> content:"something"; flowint:something,isset; >>>>>>> flowint:something,+,1; flowint:something,>,5; sid:11;) >>>>>>>>> This never works, I just have the first rule >>>>>>> fire once when the TCP session is terminated. >>>>>>>>> >>>>>>>>> P.S.: As a side note the wiki should be updated >>>>>>> to include probably "sid"s for the rules, as >>>>>>> currently when I try to run the examples >>>>>>>>> suricata complains about duplicated rules. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>> I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>>>> This seems to work : >>>>>>> >>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>> (msg:"got one"; content:"something"; >>>>>>> flowint:something,notset; flowint:something,=,1; >>>>>>> noalert; sid:10; priority: 1;) >>>>>>> >>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>> (msg:"got more"; content:"something"; >>>>>>> flowint:something,isset; flowint:something,+,1; >>>>>>> noalert; sid:11; priority: 2;) >>>>>>> >>>>>>> >>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>> (msg:"got too many"; content:"something"; >>>>>>> flowint:something,isset; flowint:something,>,2; >>>>>>> sid:12; priority: 3;) >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Oisf-users mailing list >>>>>>> Oisf-users at openinfosecfoundation.org >>>>>>> >>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Nikolay, >>>>>>> I think this is the way it is supposed to work. (last >>>>>>> example, by you). >>>>>>> >>>>>>> When you take out "noalert" form sid 11 - does it fire ? >>>>>>> >>>>>>> And are these the only rules that are loaded in terms >>>>>>> of flowint or you have others before that? >>>>>>> >>>>>>> thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Peter Manev >>>>>> >>>>>> Yes, It fires, the problem I have is that it doesn't >>>>>> fire for each occurence of "content". >>>>>> Is alert supposed to fire once per packet if it matches, >>>>>> or for each match in the stream? >>>>>> >>>>>> For example now I'm using these rules to catch if there >>>>>> are more than some defined amount of email addresses in >>>>>> a given stream : >>>>>> >>>>>> >>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>> (msg:"got one email addr"; content:"|40|"; >>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>> flow:established,from_server; >>>>>> flowint:something,notset; flowint:something,=,1; sid:10; >>>>>> priority:3; noalert;) >>>>>> >>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>> (msg:"got more email addrs"; content:"|40|"; >>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>> flow:established,from_server; >>>>>> flowint:something,isset; flowint:something,+,1; sid:11; >>>>>> priority:2; noalert;) >>>>>> >>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>> (msg:"Got too many email addrs!"; >>>>>> content:"|40|"; >>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>> flow:established,from_server; >>>>>> flowint:something,isset; flowint:something,>,10; sid:12; >>>>>> priority:1; classtype:policy-violation;) >>>>>> >>>>>> >>>>>> This for example works, but would not match for a simple >>>>>> plain text file with 10 email adresses, I need to have >>>>>> maybe 40-50 or more for this to match. >>>>>> Maybe I'm missing something? >>>>>> >>>>>> And yes, these are my only rules that I'm testing with. >>>>>> No other rules with or without flowint whatsoever. >>>>>> >>>>>> >>>>>> Hi , >>>>>> Just so I understand you correctly - you have a text file >>>>>> (in the stream) and in that text file you have 10 e-mail >>>>>> addresses and it wold not fire. correct ? >>>>>> >>>>>> >>>>>> thanks >>>>>> >>>>>> >>>>>> -- >>>>>> Peter Manev >>>>> Exactly. >>>>> >>>>> For example if I try to fetch the file emails.txt via http >>>>> which has the following content : >>>>> >>>>> # cat emails.txt >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> >>>>> $ curl http://testserver/emails.txt >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> edin at email.com >>>>> $ >>>>> >>>>> And I also remove the "noalert" option from the rules, this >>>>> is what I get in fast.log : >>>>> >>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr >>>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >>>>> -> Y.Y.Y.Y:57923 >>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email >>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} >>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923 >>>>> >>>>> >>>>> If I change the third rule to fire if the flowint var is more >>>>> than 1, it is being triggered. >>>>> >>>>> If I insert some random data between the email addresses in >>>>> the text file, then I get 4 maybe 5 matches. Doesn't it have >>>>> to match all 10 of them? >>>>> >>>>> >>>>> 1. What happens if you take out the PCRE expressions from all >>>>> the rules ? >>>>> 2. sid:12 - should not fire because you have>10 , and there are >>>>> exactly 10 e-mails in the file >>>>> 3. how big is the stream itself? i think it is below 2KB, correct? >>>>> 4. is the PCRE matching the e-mails, under the unix shell ? >>>>> 5. yes i think you should get more sid:11 alerts - but first lets >>>>> investigate the above 4. >>>>> >>>>> thanks >>>>> >>>>> -- >>>>> Peter Manev >>>> The file with only the 10 emails is 160 bytes. Even without pcre I >>>> get the same result : >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got one email addr"; content:"|40|"; \ >>>> flow:established,from_server; flowint:something,notset; >>>> flowint:something,=,1; sid:10; priority:3;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"got more email addrs"; content:"|40|"; \ >>>> flow:established,from_server; flowint:something,isset; >>>> flowint:something,+,1; sid:11; priority:2;) >>>> >>>> alert tcp $HOME_NET 80 -> any any \ >>>> (msg:"Got too many email addrs!"; content:"|40|"; \ >>>> flow:established,from_server; flowint:something,isset; >>>> flowint:something,>,9; sid:12; priority:1; >>>> classtype:policy-violation;) >>>> >>>> >>>> alerts I get : >>>> >>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] >>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58158 >>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs >>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58158 >>>> >>>> If I put some '#' symbols between the emails in the file so that >>>> it gets about 9K big and I fetch it I get these alerts : >>>> >>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] >>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58166 >>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs >>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58166 >>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs >>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58166 >>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs >>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>> Y.Y.Y.Y:58166 >>>> >>>> >>>> >>>> Hi Nikolay, >>>> >>>> >>>> Can you please post this as a bug - please be detailed (as you were in >>>> your 2 previous e-mails). >>>> Personally i think here sid 11 is the problem , may be it does not >>>> count/increment correctly.... >>>> thanks >>>> >>>> >>>> -- >>>> Peter Manev >>> Yes I will post this as a bug. But I've just found a much simpler case. >>> >>> Let's for example have only this rule in suricata : >>> >>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) >>> >>> Then on a monitored machine from the $HOME_NET range I do : >>> >>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 >>> >>> And on different host I do : >>> >>> nc testserver 6666 >>> >>> This gets the ten @ chars transferred, and I get only one alert. >>> But for example if I echo more @ chars, like 5000 or something, I get >>> 3-6 alerts. >>> I have to check what is actually the number of packets with payload, >>> probably the rule >>> is matched once per packet? But this could not explain that I get >>> different number of alerts on different runs. >> The behavior is by design. TCP data by default is inspected in the >> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected >> at once. Suricata will not try to find every possible match in a >> payload, but just one. >> >> The reason you get more alerts if you increase the payload >> significantly, is that the stream is inspected in chunks. The size of >> those chunks is determined by your stream toserver_chunk_size setting. >> >> -- >> --------------------------------------------- >> Victor Julien >> http://www.inliniac.net/ >> PGP: http://www.inliniac.net/victorjulien.asc >> --------------------------------------------- >> >> > I see now. > I was suspecting something like this, and it makes sense performance wise. > > I was trying to create a rule that will fire only if more than let's say 50 email addresses are contained > in the stream, maybe I'll try with lower threshold. > > Thanks, > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users If you ...for a cheap trick ...try the PCRE - would it behave as expected? -- Regards, Peter Manev From anoopsaldanha at gmail.com Tue Feb 14 23:51:21 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Wed, 15 Feb 2012 10:21:21 +0530 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> Message-ID: On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev wrote: > > > On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien wrote: >> >> On 02/12/2012 08:15 AM, Nikolay Denev wrote: >> > >> > On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: >> > >> >> >> >> >> >> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev > >> > wrote: >> >> >> >> >> >> ? ? On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >> >> >> >>> >> >>> >> >>> ? ? On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev > >>> ? ? > wrote: >> >>> >> >>> >> >>> ? ? ? ? On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >> >>> >> >>>> >> >>>> >> >>>> ? ? ? ? On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >> >>>> ? ? ? ? > wrote: >> >>>> >> >>>> >> >>>> ? ? ? ? ? ? On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >> >>>> >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >> >>>>> ? ? ? ? ? ? > wrote: >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >> >>>>> ? ? ? ? ? ? ? ? > >> >>>>> ? ? ? ? ? ? ? ? >> Hi all, >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> It's probably stupid question and I'm missing >> >>>>> ? ? ? ? ? ? ? ? something but I don't seem to be able >> >>>>> ? ? ? ? ? ? ? ? >> to generate alert immediately when for example a >> >>>>> ? ? ? ? ? ? ? ? given string is found inside a TCP stream. >> >>>>> ? ? ? ? ? ? ? ? >> When the TCP connection closes, suricata >> >>>>> ? ? ? ? ? ? ? ? immediately prints the alert in fast.log. >> >>>>> ? ? ? ? ? ? ? ? >> How can I make the alert be generated >> >>>>> ? ? ? ? ? ? ? ? immediately when the rule condition is matched? >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> Also I don't know if its because of this I don't >> >>>>> ? ? ? ? ? ? ? ? seem to be able to trigger the rule to match >> >>>>> ? ? ? ? ? ? ? ? several times on the same stream, >> >>>>> ? ? ? ? ? ? ? ? >> while I have the string that should fire the >> >>>>> ? ? ? ? ? ? ? ? alert several times in the stream. >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> Here's an example : >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> alert tcp $HOME_NET 6666 -> any any \ >> >>>>> ? ? ? ? ? ? ? ? >> ? ? ? (msg:"got one"; content:"something"; >> >>>>> ? ? ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; >> >>>>> ? ? ? ? ? ? ? ? sid:10;) >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> alert tcp $HOME_NET 6666 -> any any \ >> >>>>> ? ? ? ? ? ? ? ? >> ? ? ? (msg:"got five or more"; >> >>>>> ? ? ? ? ? ? ? ? content:"something"; flowint:something,isset; >> >>>>> ? ? ? ? ? ? ? ? flowint:something,+,1; flowint:something,>,5; >> >>>>> sid:11;) >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> This never works, I just have the first rule >> >>>>> ? ? ? ? ? ? ? ? fire once when the TCP session is terminated. >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> P.S.: As a side note the wiki should be updated >> >>>>> ? ? ? ? ? ? ? ? to include probably "sid"s for the rules, as >> >>>>> ? ? ? ? ? ? ? ? currently when I try to run the examples >> >>>>> ? ? ? ? ? ? ? ? >> suricata complains about duplicated rules. >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? >> Thanks, >> >>>>> ? ? ? ? ? ? ? ? >> >> >>>>> ? ? ? ? ? ? ? ? > >> >>>>> ? ? ? ? ? ? ? ? > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? This seems to work : >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >> >>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got one"; content:"something"; >> >>>>> ? ? ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; >> >>>>> ? ? ? ? ? ? ? ? noalert; sid:10; priority: 1;) >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >> >>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got more"; content:"something"; >> >>>>> ? ? ? ? ? ? ? ? flowint:something,isset; flowint:something,+,1; >> >>>>> ? ? ? ? ? ? ? ? noalert; sid:11; priority: 2;) >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >> >>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got too many"; content:"something"; >> >>>>> ? ? ? ? ? ? ? ? flowint:something,isset; flowint:something,>,2; >> >>>>> ? ? ? ? ? ? ? ? sid:12; priority: 3;) >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? ? ? _______________________________________________ >> >>>>> ? ? ? ? ? ? ? ? Oisf-users mailing list >> >>>>> ? ? ? ? ? ? ? ? Oisf-users at openinfosecfoundation.org >> >>>>> ? ? ? ? ? ? ? ? >> >>>>> >> >>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >>>>> >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? Hi Nikolay, >> >>>>> ? ? ? ? ? ? I think this is the way it is supposed to work. (last >> >>>>> ? ? ? ? ? ? example, by you). >> >>>>> >> >>>>> ? ? ? ? ? ? When you take out "noalert" form sid 11 - does it fire ? >> >>>>> >> >>>>> ? ? ? ? ? ? And are these the only rules that are loaded in terms >> >>>>> ? ? ? ? ? ? of flowint or you have others before that? >> >>>>> >> >>>>> ? ? ? ? ? ? thanks >> >>>>> >> >>>>> >> >>>>> >> >>>>> ? ? ? ? ? ? -- >> >>>>> ? ? ? ? ? ? Peter Manev >> >>>> >> >>>> >> >>>> ? ? ? ? ? ? Yes, It fires, the problem I have is that it doesn't >> >>>> ? ? ? ? ? ? fire for each occurence of "content". >> >>>> ? ? ? ? ? ? Is alert supposed to fire once per packet if it matches, >> >>>> ? ? ? ? ? ? or for each match in the stream? >> >>>> >> >>>> ? ? ? ? ? ? For example now I'm using these rules to catch if there >> >>>> ? ? ? ? ? ? are more than some defined amount of email addresses in >> >>>> ? ? ? ? ? ? a given stream : >> >>>> >> >>>> >> >>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >> >>>> ? ? ? ? ? ? ? ? ? ? (msg:"got one email addr"; content:"|40|"; >> >>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> >>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >> >>>> ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; sid:10; >> >>>> ? ? ? ? ? ? priority:3; noalert;) >> >>>> >> >>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >> >>>> ? ? ? ? ? ? ? ? ? ? (msg:"got more email addrs"; content:"|40|"; >> >>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> >>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >> >>>> ? ? ? ? ? ? flowint:something,isset; flowint:something,+,1; sid:11; >> >>>> ? ? ? ? ? ? priority:2; noalert;) >> >>>> >> >>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >> >>>> ? ? ? ? ? ? ? ? ? ? (msg:"Got too many email addrs!"; >> >>>> ? ? ? ? ? ? content:"|40|"; >> >>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >> >>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >> >>>> ? ? ? ? ? ? flowint:something,isset; flowint:something,>,10; sid:12; >> >>>> ? ? ? ? ? ? priority:1; classtype:policy-violation;) >> >>>> >> >>>> >> >>>> ? ? ? ? ? ? This for example works, but would not match for a simple >> >>>> ? ? ? ? ? ? plain text file with 10 email adresses, I need to have >> >>>> ? ? ? ? ? ? maybe 40-50 or more for this to match. >> >>>> ? ? ? ? ? ? Maybe I'm missing something? >> >>>> >> >>>> ? ? ? ? ? ? And yes, these are my only rules that I'm testing with. >> >>>> ? ? ? ? ? ? No other rules with or without flowint whatsoever. >> >>>> >> >>>> >> >>>> ? ? ? ? Hi , >> >>>> ? ? ? ? Just so I understand you correctly - you have a text file >> >>>> ? ? ? ? (in the stream) and in that text file you have 10 e-mail >> >>>> ? ? ? ? addresses and it wold not fire. correct ? >> >>>> >> >>>> >> >>>> ? ? ? ? thanks >> >>>> >> >>>> >> >>>> ? ? ? ? -- >> >>>> ? ? ? ? Peter Manev >> >>> >> >>> ? ? ? ? Exactly. >> >>> >> >>> ? ? ? ? For example if I try to fetch the file emails.txt via http >> >>> ? ? ? ? which has the following content : >> >>> >> >>> ? ? ? ? # cat emails.txt >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> >> >>> ? ? ? ? $ curl http://testserver/emails.txt >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? edin at email.com >> >>> ? ? ? ? $ >> >>> >> >>> ? ? ? ? And I also remove the "noalert" option from the rules, this >> >>> ? ? ? ? is what I get in fast.log : >> >>> >> >>> ? ? ? ? 02/11/2012-20:37:23.988271 ?[**] [1:10:0] got one email addr >> >>> ? ? ? ? [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >> >>> ? ? ? ? -> Y.Y.Y.Y:57923 >> >>> ? ? ? ? 02/11/2012-20:37:23.988271 ?[**] [1:11:0] got more email >> >>> ? ? ? ? addrs [**] [Classification: (null)] [Priority: 2] {TCP} >> >>> ? ? ? ? X.X.X.X:80 -> Y.Y.Y.Y:57923 >> >>> >> >>> >> >>> ? ? ? ? If I change the third rule to fire if the flowint var is more >> >>> ? ? ? ? than 1, it is being triggered. >> >>> >> >>> ? ? ? ? If I insert some random data between the email addresses in >> >>> ? ? ? ? the text file, then I get 4 maybe 5 matches. Doesn't it have >> >>> ? ? ? ? to match all 10 of them? >> >>> >> >>> >> >>> ? ? 1. What happens if you take out ?the PCRE expressions from all >> >>> ? ? the rules ? >> >>> ? ? 2. sid:12 - should not fire because you have >10 , and there are >> >>> ? ? exactly 10 e-mails in the file >> >>> ? ? 3. how big is the stream itself? i think it is below 2KB, correct? >> >>> ? ? 4. is the PCRE matching the e-mails, under the unix shell ? >> >>> ? ? 5. yes i think you should get more sid:11 alerts - but first lets >> >>> ? ? investigate the above 4. >> >>> >> >>> ? ? thanks >> >>> >> >>> ? ? -- >> >>> ? ? Peter Manev >> >> >> >> ? ? The file with only the 10 emails is 160 bytes. Even without pcre I >> >> ? ? get the same result : >> >> >> >> ? ? alert tcp $HOME_NET 80 -> any any \ >> >> ? ? ? ? ? ? (msg:"got one email addr"; content:"|40|"; \ >> >> ? ? ? ? ? ? flow:established,from_server; flowint:something,notset; >> >> ? ? flowint:something,=,1; sid:10; priority:3;) >> >> >> >> ? ? alert tcp $HOME_NET 80 -> any any \ >> >> ? ? ? ? ? ? (msg:"got more email addrs"; content:"|40|"; \ >> >> ? ? ? ? ? ? flow:established,from_server; flowint:something,isset; >> >> ? ? flowint:something,+,1; sid:11; priority:2;) >> >> >> >> ? ? alert tcp $HOME_NET 80 -> any any \ >> >> ? ? ? ? ? ? (msg:"Got too many email addrs!"; content:"|40|"; \ >> >> ? ? ? ? ? ? flow:established,from_server; flowint:something,isset; >> >> ? ? flowint:something,>,9; sid:12; priority:1; >> >> ? ? classtype:policy-violation;) >> >> >> >> >> >> ? ? alerts I get : >> >> >> >> ? ? 02/11/2012-21:23:14.567194 ?[**] [1:10:0] got one email addr [**] >> >> ? ? [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58158 >> >> ? ? 02/11/2012-21:23:14.567194 ?[**] [1:11:0] got more email addrs >> >> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58158 >> >> >> >> ? ? If I put some '#' symbols between the emails in the file so that >> >> ? ? it gets about 9K big and I fetch it I get these alerts : >> >> >> >> ? ? 02/11/2012-21:25:37.755214 ?[**] [1:10:0] got one email addr [**] >> >> ? ? [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58166 >> >> ? ? 02/11/2012-21:25:37.755214 ?[**] [1:11:0] got more email addrs >> >> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58166 >> >> ? ? 02/11/2012-21:25:37.761077 ?[**] [1:11:0] got more email addrs >> >> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58166 >> >> ? ? 02/11/2012-21:25:37.764451 ?[**] [1:11:0] got more email addrs >> >> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >> >> ? ? Y.Y.Y.Y:58166 >> >> >> >> >> >> >> >> Hi Nikolay, >> >> >> >> >> >> Can you please post this as a bug - please be detailed (as you were in >> >> your 2 previous e-mails). >> >> Personally i think here sid 11 is the problem , may be it does not >> >> count/increment correctly.... >> >> thanks >> >> >> >> >> >> -- >> >> Peter Manev >> > >> > Yes I will post this as a bug. But I've just found a much simpler case. >> > >> > Let's for example have only this rule in suricata : >> > >> > ? alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) >> > >> > Then on a monitored machine from the $HOME_NET range I do : >> > >> > ? echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 >> > >> > And on different host I do : >> > >> > ? nc testserver 6666 >> > >> > This gets the ten @ chars transferred, and I get only one alert. >> > But for example if I echo more @ chars, like 5000 or something, I get >> > 3-6 alerts. >> > I have to check what is actually the number of packets with payload, >> > probably the rule >> > is matched once per packet? But this could not explain that I get >> > different number of alerts on different runs. >> >> The behavior is by design. TCP data by default is inspected in the >> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected >> at once. > > >> >> Suricata will not try to find every possible match in a >> payload, but just one. > > That's good to know - clears out a few questions of mine.... > but then a PCRE (matching on 10 "@")?should match all of them - correct? > having in mind they are in the same "chunk". > If I have understood your question right, no! Pcre works just like content on the first match it finds. So alerts wise or match wise it should work the same as using content > >> >> >> The reason you get more alerts if you increase the payload >> significantly, is that the stream is inspected in chunks. The size of >> those chunks is determined by your stream toserver_chunk_size setting. >> >> -- >> --------------------------------------------- >> Victor Julien >> http://www.inliniac.net/ >> PGP: http://www.inliniac.net/victorjulien.asc >> --------------------------------------------- >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > -- > Peter Manev > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -- Anoop Saldanha From ndenev at gmail.com Wed Feb 15 00:34:12 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 07:34:12 +0200 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> Message-ID: <744B9CBB-5C60-427D-997F-859523B2A882@gmail.com> On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote: > On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev wrote: >> >> >> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien wrote: >>> >>> On 02/12/2012 08:15 AM, Nikolay Denev wrote: >>>> >>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: >>>> >>>>> >>>>> >>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev >>>> > wrote: >>>>> >>>>> >>>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev >>>>> > wrote: >>>>>> >>>>>> >>>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >>>>>>> > wrote: >>>>>>> >>>>>>> >>>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >>>>>>>> > wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>>>>> >>>>>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>>>>> > >>>>>>>> >> Hi all, >>>>>>>> >> >>>>>>>> >> It's probably stupid question and I'm missing >>>>>>>> something but I don't seem to be able >>>>>>>> >> to generate alert immediately when for example a >>>>>>>> given string is found inside a TCP stream. >>>>>>>> >> When the TCP connection closes, suricata >>>>>>>> immediately prints the alert in fast.log. >>>>>>>> >> How can I make the alert be generated >>>>>>>> immediately when the rule condition is matched? >>>>>>>> >> >>>>>>>> >> Also I don't know if its because of this I don't >>>>>>>> seem to be able to trigger the rule to match >>>>>>>> several times on the same stream, >>>>>>>> >> while I have the string that should fire the >>>>>>>> alert several times in the stream. >>>>>>>> >> >>>>>>>> >> Here's an example : >>>>>>>> >> >>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> >> (msg:"got one"; content:"something"; >>>>>>>> flowint:something,notset; flowint:something,=,1; >>>>>>>> sid:10;) >>>>>>>> >> >>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> >> (msg:"got five or more"; >>>>>>>> content:"something"; flowint:something,isset; >>>>>>>> flowint:something,+,1; flowint:something,>,5; >>>>>>>> sid:11;) >>>>>>>> >> >>>>>>>> >> This never works, I just have the first rule >>>>>>>> fire once when the TCP session is terminated. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> P.S.: As a side note the wiki should be updated >>>>>>>> to include probably "sid"s for the rules, as >>>>>>>> currently when I try to run the examples >>>>>>>> >> suricata complains about duplicated rules. >>>>>>>> >> >>>>>>>> >> Thanks, >>>>>>>> >> >>>>>>>> > >>>>>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>>>>> >>>>>>>> This seems to work : >>>>>>>> >>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> (msg:"got one"; content:"something"; >>>>>>>> flowint:something,notset; flowint:something,=,1; >>>>>>>> noalert; sid:10; priority: 1;) >>>>>>>> >>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> (msg:"got more"; content:"something"; >>>>>>>> flowint:something,isset; flowint:something,+,1; >>>>>>>> noalert; sid:11; priority: 2;) >>>>>>>> >>>>>>>> >>>>>>>> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>> (msg:"got too many"; content:"something"; >>>>>>>> flowint:something,isset; flowint:something,>,2; >>>>>>>> sid:12; priority: 3;) >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Oisf-users mailing list >>>>>>>> Oisf-users at openinfosecfoundation.org >>>>>>>> >>>>>>>> >>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi Nikolay, >>>>>>>> I think this is the way it is supposed to work. (last >>>>>>>> example, by you). >>>>>>>> >>>>>>>> When you take out "noalert" form sid 11 - does it fire ? >>>>>>>> >>>>>>>> And are these the only rules that are loaded in terms >>>>>>>> of flowint or you have others before that? >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Peter Manev >>>>>>> >>>>>>> >>>>>>> Yes, It fires, the problem I have is that it doesn't >>>>>>> fire for each occurence of "content". >>>>>>> Is alert supposed to fire once per packet if it matches, >>>>>>> or for each match in the stream? >>>>>>> >>>>>>> For example now I'm using these rules to catch if there >>>>>>> are more than some defined amount of email addresses in >>>>>>> a given stream : >>>>>>> >>>>>>> >>>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>>> (msg:"got one email addr"; content:"|40|"; >>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>> flow:established,from_server; >>>>>>> flowint:something,notset; flowint:something,=,1; sid:10; >>>>>>> priority:3; noalert;) >>>>>>> >>>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>>> (msg:"got more email addrs"; content:"|40|"; >>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>> flow:established,from_server; >>>>>>> flowint:something,isset; flowint:something,+,1; sid:11; >>>>>>> priority:2; noalert;) >>>>>>> >>>>>>> alert tcp $HOME_NET 80 -> any any \ >>>>>>> (msg:"Got too many email addrs!"; >>>>>>> content:"|40|"; >>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>> flow:established,from_server; >>>>>>> flowint:something,isset; flowint:something,>,10; sid:12; >>>>>>> priority:1; classtype:policy-violation;) >>>>>>> >>>>>>> >>>>>>> This for example works, but would not match for a simple >>>>>>> plain text file with 10 email adresses, I need to have >>>>>>> maybe 40-50 or more for this to match. >>>>>>> Maybe I'm missing something? >>>>>>> >>>>>>> And yes, these are my only rules that I'm testing with. >>>>>>> No other rules with or without flowint whatsoever. >>>>>>> >>>>>>> >>>>>>> Hi , >>>>>>> Just so I understand you correctly - you have a text file >>>>>>> (in the stream) and in that text file you have 10 e-mail >>>>>>> addresses and it wold not fire. correct ? >>>>>>> >>>>>>> >>>>>>> thanks >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Peter Manev >>>>>> >>>>>> Exactly. >>>>>> >>>>>> For example if I try to fetch the file emails.txt via http >>>>>> which has the following content : >>>>>> >>>>>> # cat emails.txt >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> >>>>>> $ curl http://testserver/emails.txt >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> edin at email.com >>>>>> $ >>>>>> >>>>>> And I also remove the "noalert" option from the rules, this >>>>>> is what I get in fast.log : >>>>>> >>>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr >>>>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >>>>>> -> Y.Y.Y.Y:57923 >>>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email >>>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} >>>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923 >>>>>> >>>>>> >>>>>> If I change the third rule to fire if the flowint var is more >>>>>> than 1, it is being triggered. >>>>>> >>>>>> If I insert some random data between the email addresses in >>>>>> the text file, then I get 4 maybe 5 matches. Doesn't it have >>>>>> to match all 10 of them? >>>>>> >>>>>> >>>>>> 1. What happens if you take out the PCRE expressions from all >>>>>> the rules ? >>>>>> 2. sid:12 - should not fire because you have >10 , and there are >>>>>> exactly 10 e-mails in the file >>>>>> 3. how big is the stream itself? i think it is below 2KB, correct? >>>>>> 4. is the PCRE matching the e-mails, under the unix shell ? >>>>>> 5. yes i think you should get more sid:11 alerts - but first lets >>>>>> investigate the above 4. >>>>>> >>>>>> thanks >>>>>> >>>>>> -- >>>>>> Peter Manev >>>>> >>>>> The file with only the 10 emails is 160 bytes. Even without pcre I >>>>> get the same result : >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"got one email addr"; content:"|40|"; \ >>>>> flow:established,from_server; flowint:something,notset; >>>>> flowint:something,=,1; sid:10; priority:3;) >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"got more email addrs"; content:"|40|"; \ >>>>> flow:established,from_server; flowint:something,isset; >>>>> flowint:something,+,1; sid:11; priority:2;) >>>>> >>>>> alert tcp $HOME_NET 80 -> any any \ >>>>> (msg:"Got too many email addrs!"; content:"|40|"; \ >>>>> flow:established,from_server; flowint:something,isset; >>>>> flowint:something,>,9; sid:12; priority:1; >>>>> classtype:policy-violation;) >>>>> >>>>> >>>>> alerts I get : >>>>> >>>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**] >>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58158 >>>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs >>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58158 >>>>> >>>>> If I put some '#' symbols between the emails in the file so that >>>>> it gets about 9K big and I fetch it I get these alerts : >>>>> >>>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**] >>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58166 >>>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs >>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58166 >>>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs >>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58166 >>>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs >>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>> Y.Y.Y.Y:58166 >>>>> >>>>> >>>>> >>>>> Hi Nikolay, >>>>> >>>>> >>>>> Can you please post this as a bug - please be detailed (as you were in >>>>> your 2 previous e-mails). >>>>> Personally i think here sid 11 is the problem , may be it does not >>>>> count/increment correctly.... >>>>> thanks >>>>> >>>>> >>>>> -- >>>>> Peter Manev >>>> >>>> Yes I will post this as a bug. But I've just found a much simpler case. >>>> >>>> Let's for example have only this rule in suricata : >>>> >>>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) >>>> >>>> Then on a monitored machine from the $HOME_NET range I do : >>>> >>>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 >>>> >>>> And on different host I do : >>>> >>>> nc testserver 6666 >>>> >>>> This gets the ten @ chars transferred, and I get only one alert. >>>> But for example if I echo more @ chars, like 5000 or something, I get >>>> 3-6 alerts. >>>> I have to check what is actually the number of packets with payload, >>>> probably the rule >>>> is matched once per packet? But this could not explain that I get >>>> different number of alerts on different runs. >>> >>> The behavior is by design. TCP data by default is inspected in the >>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected >>> at once. >> >> >>> >>> Suricata will not try to find every possible match in a >>> payload, but just one. >> >> That's good to know - clears out a few questions of mine.... >> but then a PCRE (matching on 10 "@") should match all of them - correct? >> having in mind they are in the same "chunk". >> > > If I have understood your question right, no! Pcre works just like > content on the first match it finds. So alerts wise or match wise it > should work the same as using content > So this means that there is no way to count the total number of occurrences of a given string or pattern in a flow, and alert if some predefined number is reached? i.e. no matter the number I will get one alert per chunk? Something like 'g' (global match) flag for pcre? This will definitely be very expensive, but looks interesting as feature. >> >>> >>> >>> The reason you get more alerts if you increase the payload >>> significantly, is that the stream is inspected in chunks. The size of >>> those chunks is determined by your stream toserver_chunk_size setting. >>> >>> -- >>> --------------------------------------------- >>> Victor Julien >>> http://www.inliniac.net/ >>> PGP: http://www.inliniac.net/victorjulien.asc >>> --------------------------------------------- >>> >>> _______________________________________________ >>> Oisf-users mailing list >>> Oisf-users at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> >> >> >> >> -- >> Peter Manev >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> > > > > -- > Anoop Saldanha > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From ndenev at gmail.com Wed Feb 15 00:42:42 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 07:42:42 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. Message-ID: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> Hi, I'm wondering how suricata decides which packet to capture and dump in the unified2 log file and which not to. I'm running Snorby to collect the alerts, and I've noticed that sometimes for a single rule, some of the alerts have the packet dump present and some not. Regards, Nikolay From anoopsaldanha at gmail.com Wed Feb 15 00:52:25 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Wed, 15 Feb 2012 11:22:25 +0530 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: <744B9CBB-5C60-427D-997F-859523B2A882@gmail.com> References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> <744B9CBB-5C60-427D-997F-859523B2A882@gmail.com> Message-ID: On Wed, Feb 15, 2012 at 11:04 AM, Nikolay Denev wrote: > > On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote: > >> On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev wrote: >>> >>> >>> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien wrote: >>>> >>>> On 02/12/2012 08:15 AM, Nikolay Denev wrote: >>>>> >>>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev >>>>> > wrote: >>>>>> >>>>>> >>>>>> ? ? On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> ? ? On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev >>>>>> ? ? > wrote: >>>>>>> >>>>>>> >>>>>>> ? ? ? ? On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev >>>>>>>> ? ? ? ? > wrote: >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? ? ? On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev >>>>>>>>> ? ? ? ? ? ? > wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote: >>>>>>>>> ? ? ? ? ? ? ? ? > >>>>>>>>> ? ? ? ? ? ? ? ? >> Hi all, >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> It's probably stupid question and I'm missing >>>>>>>>> ? ? ? ? ? ? ? ? something but I don't seem to be able >>>>>>>>> ? ? ? ? ? ? ? ? >> to generate alert immediately when for example a >>>>>>>>> ? ? ? ? ? ? ? ? given string is found inside a TCP stream. >>>>>>>>> ? ? ? ? ? ? ? ? >> When the TCP connection closes, suricata >>>>>>>>> ? ? ? ? ? ? ? ? immediately prints the alert in fast.log. >>>>>>>>> ? ? ? ? ? ? ? ? >> How can I make the alert be generated >>>>>>>>> ? ? ? ? ? ? ? ? immediately when the rule condition is matched? >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> Also I don't know if its because of this I don't >>>>>>>>> ? ? ? ? ? ? ? ? seem to be able to trigger the rule to match >>>>>>>>> ? ? ? ? ? ? ? ? several times on the same stream, >>>>>>>>> ? ? ? ? ? ? ? ? >> while I have the string that should fire the >>>>>>>>> ? ? ? ? ? ? ? ? alert several times in the stream. >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> Here's an example : >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> ? ? ? ? ? ? ? ? >> ? ? ? (msg:"got one"; content:"something"; >>>>>>>>> ? ? ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; >>>>>>>>> ? ? ? ? ? ? ? ? sid:10;) >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> ? ? ? ? ? ? ? ? >> ? ? ? (msg:"got five or more"; >>>>>>>>> ? ? ? ? ? ? ? ? content:"something"; flowint:something,isset; >>>>>>>>> ? ? ? ? ? ? ? ? flowint:something,+,1; flowint:something,>,5; >>>>>>>>> sid:11;) >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> This never works, I just have the first rule >>>>>>>>> ? ? ? ? ? ? ? ? fire once when the TCP session is terminated. >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> P.S.: As a side note the wiki should be updated >>>>>>>>> ? ? ? ? ? ? ? ? to include probably "sid"s for the rules, as >>>>>>>>> ? ? ? ? ? ? ? ? currently when I try to run the examples >>>>>>>>> ? ? ? ? ? ? ? ? >> suricata complains about duplicated rules. >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? >> Thanks, >>>>>>>>> ? ? ? ? ? ? ? ? >> >>>>>>>>> ? ? ? ? ? ? ? ? > >>>>>>>>> ? ? ? ? ? ? ? ? > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE. >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? This seems to work : >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got one"; content:"something"; >>>>>>>>> ? ? ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; >>>>>>>>> ? ? ? ? ? ? ? ? noalert; sid:10; priority: 1;) >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got more"; content:"something"; >>>>>>>>> ? ? ? ? ? ? ? ? flowint:something,isset; flowint:something,+,1; >>>>>>>>> ? ? ? ? ? ? ? ? noalert; sid:11; priority: 2;) >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? alert tcp $HOME_NET 6666 -> any any \ >>>>>>>>> ? ? ? ? ? ? ? ? ? ? ? ?(msg:"got too many"; content:"something"; >>>>>>>>> ? ? ? ? ? ? ? ? flowint:something,isset; flowint:something,>,2; >>>>>>>>> ? ? ? ? ? ? ? ? sid:12; priority: 3;) >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? ? ? _______________________________________________ >>>>>>>>> ? ? ? ? ? ? ? ? Oisf-users mailing list >>>>>>>>> ? ? ? ? ? ? ? ? Oisf-users at openinfosecfoundation.org >>>>>>>>> ? ? ? ? ? ? ? ? >>>>>>>>> >>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? Hi Nikolay, >>>>>>>>> ? ? ? ? ? ? I think this is the way it is supposed to work. (last >>>>>>>>> ? ? ? ? ? ? example, by you). >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? When you take out "noalert" form sid 11 - does it fire ? >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? And are these the only rules that are loaded in terms >>>>>>>>> ? ? ? ? ? ? of flowint or you have others before that? >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ? ? ? ? ? ? -- >>>>>>>>> ? ? ? ? ? ? Peter Manev >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? ? ? Yes, It fires, the problem I have is that it doesn't >>>>>>>> ? ? ? ? ? ? fire for each occurence of "content". >>>>>>>> ? ? ? ? ? ? Is alert supposed to fire once per packet if it matches, >>>>>>>> ? ? ? ? ? ? or for each match in the stream? >>>>>>>> >>>>>>>> ? ? ? ? ? ? For example now I'm using these rules to catch if there >>>>>>>> ? ? ? ? ? ? are more than some defined amount of email addresses in >>>>>>>> ? ? ? ? ? ? a given stream : >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? (msg:"got one email addr"; content:"|40|"; >>>>>>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >>>>>>>> ? ? ? ? ? ? flowint:something,notset; flowint:something,=,1; sid:10; >>>>>>>> ? ? ? ? ? ? priority:3; noalert;) >>>>>>>> >>>>>>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? (msg:"got more email addrs"; content:"|40|"; >>>>>>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >>>>>>>> ? ? ? ? ? ? flowint:something,isset; flowint:something,+,1; sid:11; >>>>>>>> ? ? ? ? ? ? priority:2; noalert;) >>>>>>>> >>>>>>>> ? ? ? ? ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? (msg:"Got too many email addrs!"; >>>>>>>> ? ? ? ? ? ? content:"|40|"; >>>>>>>> ? ? ? ? ? ? pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ >>>>>>>> ? ? ? ? ? ? ? ? ? ? flow:established,from_server; >>>>>>>> ? ? ? ? ? ? flowint:something,isset; flowint:something,>,10; sid:12; >>>>>>>> ? ? ? ? ? ? priority:1; classtype:policy-violation;) >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? ? ? This for example works, but would not match for a simple >>>>>>>> ? ? ? ? ? ? plain text file with 10 email adresses, I need to have >>>>>>>> ? ? ? ? ? ? maybe 40-50 or more for this to match. >>>>>>>> ? ? ? ? ? ? Maybe I'm missing something? >>>>>>>> >>>>>>>> ? ? ? ? ? ? And yes, these are my only rules that I'm testing with. >>>>>>>> ? ? ? ? ? ? No other rules with or without flowint whatsoever. >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? Hi , >>>>>>>> ? ? ? ? Just so I understand you correctly - you have a text file >>>>>>>> ? ? ? ? (in the stream) and in that text file you have 10 e-mail >>>>>>>> ? ? ? ? addresses and it wold not fire. correct ? >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? thanks >>>>>>>> >>>>>>>> >>>>>>>> ? ? ? ? -- >>>>>>>> ? ? ? ? Peter Manev >>>>>>> >>>>>>> ? ? ? ? Exactly. >>>>>>> >>>>>>> ? ? ? ? For example if I try to fetch the file emails.txt via http >>>>>>> ? ? ? ? which has the following content : >>>>>>> >>>>>>> ? ? ? ? # cat emails.txt >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> >>>>>>> ? ? ? ? $ curl http://testserver/emails.txt >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? edin at email.com >>>>>>> ? ? ? ? $ >>>>>>> >>>>>>> ? ? ? ? And I also remove the "noalert" option from the rules, this >>>>>>> ? ? ? ? is what I get in fast.log : >>>>>>> >>>>>>> ? ? ? ? 02/11/2012-20:37:23.988271 ?[**] [1:10:0] got one email addr >>>>>>> ? ? ? ? [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 >>>>>>> ? ? ? ? -> Y.Y.Y.Y:57923 >>>>>>> ? ? ? ? 02/11/2012-20:37:23.988271 ?[**] [1:11:0] got more email >>>>>>> ? ? ? ? addrs [**] [Classification: (null)] [Priority: 2] {TCP} >>>>>>> ? ? ? ? X.X.X.X:80 -> Y.Y.Y.Y:57923 >>>>>>> >>>>>>> >>>>>>> ? ? ? ? If I change the third rule to fire if the flowint var is more >>>>>>> ? ? ? ? than 1, it is being triggered. >>>>>>> >>>>>>> ? ? ? ? If I insert some random data between the email addresses in >>>>>>> ? ? ? ? the text file, then I get 4 maybe 5 matches. Doesn't it have >>>>>>> ? ? ? ? to match all 10 of them? >>>>>>> >>>>>>> >>>>>>> ? ? 1. What happens if you take out ?the PCRE expressions from all >>>>>>> ? ? the rules ? >>>>>>> ? ? 2. sid:12 - should not fire because you have >10 , and there are >>>>>>> ? ? exactly 10 e-mails in the file >>>>>>> ? ? 3. how big is the stream itself? i think it is below 2KB, correct? >>>>>>> ? ? 4. is the PCRE matching the e-mails, under the unix shell ? >>>>>>> ? ? 5. yes i think you should get more sid:11 alerts - but first lets >>>>>>> ? ? investigate the above 4. >>>>>>> >>>>>>> ? ? thanks >>>>>>> >>>>>>> ? ? -- >>>>>>> ? ? Peter Manev >>>>>> >>>>>> ? ? The file with only the 10 emails is 160 bytes. Even without pcre I >>>>>> ? ? get the same result : >>>>>> >>>>>> ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>> ? ? ? ? ? ? (msg:"got one email addr"; content:"|40|"; \ >>>>>> ? ? ? ? ? ? flow:established,from_server; flowint:something,notset; >>>>>> ? ? flowint:something,=,1; sid:10; priority:3;) >>>>>> >>>>>> ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>> ? ? ? ? ? ? (msg:"got more email addrs"; content:"|40|"; \ >>>>>> ? ? ? ? ? ? flow:established,from_server; flowint:something,isset; >>>>>> ? ? flowint:something,+,1; sid:11; priority:2;) >>>>>> >>>>>> ? ? alert tcp $HOME_NET 80 -> any any \ >>>>>> ? ? ? ? ? ? (msg:"Got too many email addrs!"; content:"|40|"; \ >>>>>> ? ? ? ? ? ? flow:established,from_server; flowint:something,isset; >>>>>> ? ? flowint:something,>,9; sid:12; priority:1; >>>>>> ? ? classtype:policy-violation;) >>>>>> >>>>>> >>>>>> ? ? alerts I get : >>>>>> >>>>>> ? ? 02/11/2012-21:23:14.567194 ?[**] [1:10:0] got one email addr [**] >>>>>> ? ? [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58158 >>>>>> ? ? 02/11/2012-21:23:14.567194 ?[**] [1:11:0] got more email addrs >>>>>> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58158 >>>>>> >>>>>> ? ? If I put some '#' symbols between the emails in the file so that >>>>>> ? ? it gets about 9K big and I fetch it I get these alerts : >>>>>> >>>>>> ? ? 02/11/2012-21:25:37.755214 ?[**] [1:10:0] got one email addr [**] >>>>>> ? ? [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58166 >>>>>> ? ? 02/11/2012-21:25:37.755214 ?[**] [1:11:0] got more email addrs >>>>>> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58166 >>>>>> ? ? 02/11/2012-21:25:37.761077 ?[**] [1:11:0] got more email addrs >>>>>> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58166 >>>>>> ? ? 02/11/2012-21:25:37.764451 ?[**] [1:11:0] got more email addrs >>>>>> ? ? [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> >>>>>> ? ? Y.Y.Y.Y:58166 >>>>>> >>>>>> >>>>>> >>>>>> Hi Nikolay, >>>>>> >>>>>> >>>>>> Can you please post this as a bug - please be detailed (as you were in >>>>>> your 2 previous e-mails). >>>>>> Personally i think here sid 11 is the problem , may be it does not >>>>>> count/increment correctly.... >>>>>> thanks >>>>>> >>>>>> >>>>>> -- >>>>>> Peter Manev >>>>> >>>>> Yes I will post this as a bug. But I've just found a much simpler case. >>>>> >>>>> Let's for example have only this rule in suricata : >>>>> >>>>> ? alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) >>>>> >>>>> Then on a monitored machine from the $HOME_NET range I do : >>>>> >>>>> ? echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 >>>>> >>>>> And on different host I do : >>>>> >>>>> ? nc testserver 6666 >>>>> >>>>> This gets the ten @ chars transferred, and I get only one alert. >>>>> But for example if I echo more @ chars, like 5000 or something, I get >>>>> 3-6 alerts. >>>>> I have to check what is actually the number of packets with payload, >>>>> probably the rule >>>>> is matched once per packet? But this could not explain that I get >>>>> different number of alerts on different runs. >>>> >>>> The behavior is by design. TCP data by default is inspected in the >>>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected >>>> at once. >>> >>> >>>> >>>> Suricata will not try to find every possible match in a >>>> payload, but just one. >>> >>> That's good to know - clears out a few questions of mine.... >>> but then a PCRE (matching on 10 "@") should match all of them - correct? >>> having in mind they are in the same "chunk". >>> >> >> If I have understood your question right, no! ?Pcre works just like >> content on the first match it finds. ?So alerts wise or match wise it >> should work the same as using content >> > > So this means that there is no way to count the total number of occurrences of a > given string or pattern in a flow, and alert if some predefined number is reached? > i.e. no matter the number I will get one alert per chunk? > > Something like 'g' (global match) flag for pcre? This will definitely be very expensive, but looks interesting as feature. > You won't be able to alert for every pattern in the flow, but you can alert once by counting the no of patterns present in the stream using pcre. You don't need 'g' or such feature to do so. Something like pcre:"/(kaboom.*){3}/"; should do the trick by alerting if the pattern kaboom is present in the string thrice. But I'd suggest avoiding pcre under most circumstances. >>> >>>> >>>> >>>> The reason you get more alerts if you increase the payload >>>> significantly, is that the stream is inspected in chunks. The size of >>>> those chunks is determined by your stream toserver_chunk_size setting. >>>> >>>> -- >>>> --------------------------------------------- >>>> Victor Julien >>>> http://www.inliniac.net/ >>>> PGP: http://www.inliniac.net/victorjulien.asc >>>> --------------------------------------------- >>>> >>>> _______________________________________________ >>>> Oisf-users mailing list >>>> Oisf-users at openinfosecfoundation.org >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> >>> >>> >>> >>> -- >>> Peter Manev >>> >>> _______________________________________________ >>> Oisf-users mailing list >>> Oisf-users at openinfosecfoundation.org >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> >> >> >> >> -- >> Anoop Saldanha >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -- Anoop Saldanha From victor at inliniac.net Wed Feb 15 03:07:52 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 15 Feb 2012 09:07:52 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> Message-ID: <4F3B67D8.7060806@inliniac.net> On 02/15/2012 06:42 AM, Nikolay Denev wrote: > Hi, > > I'm wondering how suricata decides which packet to capture and dump in the unified2 log file and which not to. > > I'm running Snorby to collect the alerts, and I've noticed that sometimes for a single rule, some of the alerts have > the packet dump present and some not. That is odd. There should always be a packet. Is this happening with specific rules and / or traffic? -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From anoopsaldanha at gmail.com Wed Feb 15 03:08:29 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Wed, 15 Feb 2012 13:38:29 +0530 Subject: [Oisf-users] New MPM available In-Reply-To: <4F3A2DBA.9060009@inliniac.net> References: <4F3A2DBA.9060009@inliniac.net> Message-ID: On Tue, Feb 14, 2012 at 3:17 PM, Victor Julien wrote: > On 02/14/2012 10:22 AM, Anoop Saldanha wrote: >> Hello all, >> >> We have a new MPM available in our codebase - "ac-bs". ?This provides >> compression that's pretty close to ac-gfbs, while performing better >> than ac-gfbs. >> >> To use this mpm, set >> >> "mpm-algo: ac-bs" in the conf file. >> >> Would appreciate performance numbers with both >> >> "sgh-mpm-context:full" >> and >> "sgh-mpm-context:single" >> >> To give an explanation on what "sgh-mpm-context" and the params "full" >> and "single" mean, these refer to how we set up mpm contexts. >> "single" indicates that we use a single context for all the patterns >> in the engine. ?"full" indicates that we split the patterns into many >> mpm contexts, one mpm context per signature group head(sgh). >> >> To use "full" with a sufficiently decent ruleset(say > 10k rules with >> a decent no of patterns) would require a lot of memory, running into a >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case >> of "ac". ?"single" solves this with a single context and hence the >> smaller memory footprint for the engine. >> >> If the machine has sufficient memory, "full" is suggested as it >> provides much better performance than "single", albeit at the cost of >> increased memory consumption. ?More of a available_memory vs >> performance scenario. >> >> Looking forward to some performance/memory feedback/benchmarks with >> this mpm from the community. > > So far from what I have seen, in a default et ruleset with the default > suricata.yaml, "ac" is faster than "ac-bs". > > It would be interesting to set the "detect-engine.profile" to high with > "ac-bs", as that settings increases the number of rule groups (sgh). > It will make a difference only if we use sgh-mpm-context:full. Keen to see how ac-bs performs with "full" sgh-mpm-context, against other mpms. -- Anoop Saldanha From christophe at vandeplas.com Wed Feb 15 04:11:24 2012 From: christophe at vandeplas.com (Christophe Vandeplas) Date: Wed, 15 Feb 2012 10:11:24 +0100 Subject: [Oisf-users] Suricata VLAN Message-ID: Hello, I have a situation where a switch is acting 'originally' with traffic mirroring. The mirrored traffic in inbound direction is in the native vlan, and the outbound is in a tagged vlan. I wonder how Suricata handles these flows. Will it be able to reconstruct the TCP sessions correctly? Even if the traffic is not in the same VLAN? What would be the impact if it doesn't reconstruct the traffic? I'm certain that some things will still work, but I'm not certain about the real impact. Thanks for the advice. Christophe From eric at regit.org Wed Feb 15 04:15:56 2012 From: eric at regit.org (Eric Leblond) Date: Wed, 15 Feb 2012 10:15:56 +0100 Subject: [Oisf-users] Suricata VLAN In-Reply-To: References: Message-ID: <1329297356.4451.39.camel@ice-age.regit.org> Hello, Le mercredi 15 f?vrier 2012 ? 10:11 +0100, Christophe Vandeplas a ?crit : > Hello, > > I have a situation where a switch is acting 'originally' with traffic mirroring. > > The mirrored traffic in inbound direction is in the native vlan, and > the outbound is in a tagged vlan. > > I wonder how Suricata handles these flows. > Will it be able to reconstruct the TCP sessions correctly? Even if the > traffic is not in the same VLAN? > > What would be the impact if it doesn't reconstruct the traffic? > I'm certain that some things will still work, but I'm not certain > about the real impact. You may want to use a BPF expression to only select the packet from one of the VLAN. For example, "not vlan XX" should select only the incoming packets. This could avoid issue with seeing two times each packet. To provide a BPF, just add it at the end of suricata command line. BR, -- Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: This is a digitally signed message part Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/bd5fe180/attachment.bin From ndenev at gmail.com Wed Feb 15 04:26:29 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 11:26:29 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <4F3B67D8.7060806@inliniac.net> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> Message-ID: <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> On Feb 15, 2012, at 10:07 AM, Victor Julien wrote: > On 02/15/2012 06:42 AM, Nikolay Denev wrote: >> Hi, >> >> I'm wondering how suricata decides which packet to capture and dump in the unified2 log file and which not to. >> >> I'm running Snorby to collect the alerts, and I've noticed that sometimes for a single rule, some of the alerts have >> the packet dump present and some not. > > That is odd. There should always be a packet. Is this happening with > specific rules and / or traffic? > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users I've checked now. Some of the alerts without packet dump are packets with only headers and no payload, for example syn packets from RBN listed IPs. Which should be normal. But I have also alert from this rule: alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ETPRO TROJAN Win32/Chir.B at mm User-Agent (KPeerUpdater)"; flow:to_server,established; content:"User-Agent|3a| KPeerUpdater|0d 0a|"; http_header; reference:url,www.threatexpert.com/report.aspx?md5=5ca614132b183b2812cb69112879237f; reference:url,www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Worm%3AWin32%2FChir.B%40mm; classtype:trojan-activity; sid:2803871; rev:2;) And in snorby I see no packet dump, and packet len is 40? I can also look in the unified2.alert file to make sure it's not snorby problem. (if I can find some tool to check it :) ) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/1742f8d1/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: snorby-no-packet-dump.png Type: image/png Size: 82904 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/1742f8d1/snorby-no-packet-dump-0001.png From christophe at vandeplas.com Wed Feb 15 04:28:00 2012 From: christophe at vandeplas.com (Christophe Vandeplas) Date: Wed, 15 Feb 2012 10:28:00 +0100 Subject: [Oisf-users] Suricata VLAN In-Reply-To: <1329297356.4451.39.camel@ice-age.regit.org> References: <1329297356.4451.39.camel@ice-age.regit.org> Message-ID: Hello Eric, I think you misunderstood the situation. (or I am misunderstanding your answer) We are mirroring one physical port of the switch. That port is configured as an access port, in vlan X. (so no VLAN trunk) Our sniffing port seems to send out the traffic from the mirrored physical port, but the inbound traffic contains a VLAN tag, and the outbound traffic is in the native VLAN. In other words , a TCP handshake is: 10.0.0.1 -> 2.2.2.2 SYN native vlan 2.2.2.2 -> SYN/ACK tagged vlan X 10.0.0.1 -> 2.2.2.2 ACK native vlan. It seems to be a "feature" of this switch. (how it handles traffic internally in the switch) So the thing is that I don't want to filter out the VLAN traffic, but I'd like to know the impact on Suricata. Will it be able to rebuild the TCP sessions, will it have impact on the IDS rules? ... Thanks On Wed, Feb 15, 2012 at 10:15 AM, Eric Leblond wrote: > Hello, > > Le mercredi 15 f?vrier 2012 ? 10:11 +0100, Christophe Vandeplas a > ?crit : >> Hello, >> >> I have a situation where a switch is acting 'originally' with traffic mirroring. >> >> The mirrored traffic in inbound direction is in the native vlan, and >> the outbound is in a tagged vlan. >> >> I wonder how Suricata handles these flows. >> Will it be able to reconstruct the TCP sessions correctly? Even if the >> traffic is not in the same VLAN? >> >> What would be the impact if it doesn't reconstruct the traffic? >> I'm certain that some things will still work, but I'm not certain >> about the real impact. > > You may want to use a BPF expression to only select the packet from one > of the VLAN. For example, "not vlan XX" should select only the incoming > packets. This could avoid issue with seeing two times each packet. > > To provide a BPF, just add it at the end of suricata command line. > > BR, > -- > Eric From c.d.wakelin at reading.ac.uk Wed Feb 15 04:46:29 2012 From: c.d.wakelin at reading.ac.uk (Chris Wakelin) Date: Wed, 15 Feb 2012 09:46:29 +0000 Subject: [Oisf-users] Suricata VLAN In-Reply-To: References: <1329297356.4451.39.camel@ice-age.regit.org> Message-ID: <4F3B7EF5.5070100@reading.ac.uk> This is a similar situation to our Extreme border switches (which we're not monitoring at the moment; we're monitoring the core to firewall instead). It didn't seem to cause any problems in Suricata. We're using PF_RING with cluster-per-flow and Will Metcalf pointed out that it uses the VLAN id as part of its header hashing and suggested we modify the hash to leave the VLAN id out. You do need to make sure that Suricata's default-packet-size is appropriate to take into account the VLAN headers; we set it manually to 1522. Best Wishes, Chris On 15/02/2012 09:28, Christophe Vandeplas wrote: > Hello Eric, > > I think you misunderstood the situation. (or I am misunderstanding your answer) > > We are mirroring one physical port of the switch. > That port is configured as an access port, in vlan X. (so no VLAN trunk) > > Our sniffing port seems to send out the traffic from the mirrored physical port, > but the inbound traffic contains a VLAN tag, > and the outbound traffic is in the native VLAN. > > In other words , a TCP handshake is: > 10.0.0.1 -> 2.2.2.2 SYN native vlan > 2.2.2.2 -> SYN/ACK tagged vlan X > 10.0.0.1 -> 2.2.2.2 ACK native vlan. > > It seems to be a "feature" of this switch. (how it handles traffic > internally in the switch) > > So the thing is that I don't want to filter out the VLAN traffic, > but I'd like to know the impact on Suricata. Will it be able to > rebuild the TCP sessions, will it have impact on the IDS rules? ... > > Thanks > > > > > On Wed, Feb 15, 2012 at 10:15 AM, Eric Leblond wrote: >> Hello, >> >> Le mercredi 15 f?vrier 2012 ? 10:11 +0100, Christophe Vandeplas a >> ?crit : >>> Hello, >>> >>> I have a situation where a switch is acting 'originally' with traffic mirroring. >>> >>> The mirrored traffic in inbound direction is in the native vlan, and >>> the outbound is in a tagged vlan. >>> >>> I wonder how Suricata handles these flows. >>> Will it be able to reconstruct the TCP sessions correctly? Even if the >>> traffic is not in the same VLAN? >>> >>> What would be the impact if it doesn't reconstruct the traffic? >>> I'm certain that some things will still work, but I'm not certain >>> about the real impact. >> >> You may want to use a BPF expression to only select the packet from one >> of the VLAN. For example, "not vlan XX" should select only the incoming >> packets. This could avoid issue with seeing two times each packet. >> >> To provide a BPF, just add it at the end of suricata command line. >> >> BR, >> -- >> Eric > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin at reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094 From victor at inliniac.net Wed Feb 15 04:56:48 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 15 Feb 2012 10:56:48 +0100 Subject: [Oisf-users] Suricata VLAN In-Reply-To: <4F3B7EF5.5070100@reading.ac.uk> References: <1329297356.4451.39.camel@ice-age.regit.org> <4F3B7EF5.5070100@reading.ac.uk> Message-ID: <4F3B8160.4090104@inliniac.net> On 02/15/2012 10:46 AM, Chris Wakelin wrote: > You do need to make sure that Suricata's default-packet-size is > appropriate to take into account the VLAN headers; we set it manually to > 1522. "Mis setting" this only has a performance impact, not an accuracy impact. Bigger packets than the default-packet-size are still handled, but using a slightly slower code path. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From petermanev at gmail.com Wed Feb 15 05:39:51 2012 From: petermanev at gmail.com (Peter Manev) Date: Wed, 15 Feb 2012 11:39:51 +0100 Subject: [Oisf-users] real time alert on tcp stream and flowint In-Reply-To: References: <20318A00-0EF5-428C-BC0C-F2B2BEFF9EF1@gmail.com> <0B9B5165-F9F6-4894-A4B3-B04F82D6FF4B@gmail.com> <783F2D06-6F26-4277-81FA-EC9E59ABBE75@gmail.com> <96929C1B-FEEB-4EC2-B064-7574C5B184BA@gmail.com> <4F3A27B3.8020608@inliniac.net> <744B9CBB-5C60-427D-997F-859523B2A882@gmail.com> Message-ID: On Wed, Feb 15, 2012 at 6:52 AM, Anoop Saldanha wrote: > On Wed, Feb 15, 2012 at 11:04 AM, Nikolay Denev wrote: > > > > On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote: > > > >> On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev > wrote: > >>> > >>> > >>> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien > wrote: > >>>> > >>>> On 02/12/2012 08:15 AM, Nikolay Denev wrote: > >>>>> > >>>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote: > >>>>> > >>>>>> > >>>>>> > >>>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev >>>>>> > wrote: > >>>>>> > >>>>>> > >>>>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote: > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev < > ndenev at gmail.com > >>>>>>> > wrote: > >>>>>>> > >>>>>>> > >>>>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev > >>>>>>>> > wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote: > >>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev > >>>>>>>>> > wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote: > >>>>>>>>> > >>>>>>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev > wrote: > >>>>>>>>> > > >>>>>>>>> >> Hi all, > >>>>>>>>> >> > >>>>>>>>> >> It's probably stupid question and I'm missing > >>>>>>>>> something but I don't seem to be able > >>>>>>>>> >> to generate alert immediately when for > example a > >>>>>>>>> given string is found inside a TCP stream. > >>>>>>>>> >> When the TCP connection closes, suricata > >>>>>>>>> immediately prints the alert in fast.log. > >>>>>>>>> >> How can I make the alert be generated > >>>>>>>>> immediately when the rule condition is matched? > >>>>>>>>> >> > >>>>>>>>> >> Also I don't know if its because of this I > don't > >>>>>>>>> seem to be able to trigger the rule to match > >>>>>>>>> several times on the same stream, > >>>>>>>>> >> while I have the string that should fire the > >>>>>>>>> alert several times in the stream. > >>>>>>>>> >> > >>>>>>>>> >> Here's an example : > >>>>>>>>> >> > >>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \ > >>>>>>>>> >> (msg:"got one"; content:"something"; > >>>>>>>>> flowint:something,notset; flowint:something,=,1; > >>>>>>>>> sid:10;) > >>>>>>>>> >> > >>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \ > >>>>>>>>> >> (msg:"got five or more"; > >>>>>>>>> content:"something"; flowint:something,isset; > >>>>>>>>> flowint:something,+,1; flowint:something,>,5; > >>>>>>>>> sid:11;) > >>>>>>>>> >> > >>>>>>>>> >> This never works, I just have the first rule > >>>>>>>>> fire once when the TCP session is terminated. > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> P.S.: As a side note the wiki should be > updated > >>>>>>>>> to include probably "sid"s for the rules, as > >>>>>>>>> currently when I try to run the examples > >>>>>>>>> >> suricata complains about duplicated rules. > >>>>>>>>> >> > >>>>>>>>> >> Thanks, > >>>>>>>>> >> > >>>>>>>>> > > >>>>>>>>> > I'm running 1.2.1 RELEASE on > FreeBSD-9.0-STABLE. > >>>>>>>>> > >>>>>>>>> This seems to work : > >>>>>>>>> > >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>>>>>> (msg:"got one"; content:"something"; > >>>>>>>>> flowint:something,notset; flowint:something,=,1; > >>>>>>>>> noalert; sid:10; priority: 1;) > >>>>>>>>> > >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>>>>>> (msg:"got more"; content:"something"; > >>>>>>>>> flowint:something,isset; flowint:something,+,1; > >>>>>>>>> noalert; sid:11; priority: 2;) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \ > >>>>>>>>> (msg:"got too many"; content:"something"; > >>>>>>>>> flowint:something,isset; flowint:something,>,2; > >>>>>>>>> sid:12; priority: 3;) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> Oisf-users mailing list > >>>>>>>>> Oisf-users at openinfosecfoundation.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Hi Nikolay, > >>>>>>>>> I think this is the way it is supposed to work. (last > >>>>>>>>> example, by you). > >>>>>>>>> > >>>>>>>>> When you take out "noalert" form sid 11 - does it > fire ? > >>>>>>>>> > >>>>>>>>> And are these the only rules that are loaded in terms > >>>>>>>>> of flowint or you have others before that? > >>>>>>>>> > >>>>>>>>> thanks > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Peter Manev > >>>>>>>> > >>>>>>>> > >>>>>>>> Yes, It fires, the problem I have is that it doesn't > >>>>>>>> fire for each occurence of "content". > >>>>>>>> Is alert supposed to fire once per packet if it > matches, > >>>>>>>> or for each match in the stream? > >>>>>>>> > >>>>>>>> For example now I'm using these rules to catch if > there > >>>>>>>> are more than some defined amount of email addresses > in > >>>>>>>> a given stream : > >>>>>>>> > >>>>>>>> > >>>>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>>>> (msg:"got one email addr"; content:"|40|"; > >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>>>>>> flow:established,from_server; > >>>>>>>> flowint:something,notset; flowint:something,=,1; > sid:10; > >>>>>>>> priority:3; noalert;) > >>>>>>>> > >>>>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>>>> (msg:"got more email addrs"; content:"|40|"; > >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>>>>>> flow:established,from_server; > >>>>>>>> flowint:something,isset; flowint:something,+,1; > sid:11; > >>>>>>>> priority:2; noalert;) > >>>>>>>> > >>>>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>>>> (msg:"Got too many email addrs!"; > >>>>>>>> content:"|40|"; > >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \ > >>>>>>>> flow:established,from_server; > >>>>>>>> flowint:something,isset; flowint:something,>,10; > sid:12; > >>>>>>>> priority:1; classtype:policy-violation;) > >>>>>>>> > >>>>>>>> > >>>>>>>> This for example works, but would not match for a > simple > >>>>>>>> plain text file with 10 email adresses, I need to have > >>>>>>>> maybe 40-50 or more for this to match. > >>>>>>>> Maybe I'm missing something? > >>>>>>>> > >>>>>>>> And yes, these are my only rules that I'm testing > with. > >>>>>>>> No other rules with or without flowint whatsoever. > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi , > >>>>>>>> Just so I understand you correctly - you have a text file > >>>>>>>> (in the stream) and in that text file you have 10 e-mail > >>>>>>>> addresses and it wold not fire. correct ? > >>>>>>>> > >>>>>>>> > >>>>>>>> thanks > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Peter Manev > >>>>>>> > >>>>>>> Exactly. > >>>>>>> > >>>>>>> For example if I try to fetch the file emails.txt via http > >>>>>>> which has the following content : > >>>>>>> > >>>>>>> # cat emails.txt > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> > >>>>>>> $ curl http://testserver/emails.txt > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> edin at email.com > >>>>>>> $ > >>>>>>> > >>>>>>> And I also remove the "noalert" option from the rules, this > >>>>>>> is what I get in fast.log : > >>>>>>> > >>>>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email > addr > >>>>>>> [**] [Classification: (null)] [Priority: 3] {TCP} > X.X.X.X:80 > >>>>>>> -> Y.Y.Y.Y:57923 > >>>>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email > >>>>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP} > >>>>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923 > >>>>>>> > >>>>>>> > >>>>>>> If I change the third rule to fire if the flowint var is > more > >>>>>>> than 1, it is being triggered. > >>>>>>> > >>>>>>> If I insert some random data between the email addresses in > >>>>>>> the text file, then I get 4 maybe 5 matches. Doesn't it > have > >>>>>>> to match all 10 of them? > >>>>>>> > >>>>>>> > >>>>>>> 1. What happens if you take out the PCRE expressions from all > >>>>>>> the rules ? > >>>>>>> 2. sid:12 - should not fire because you have >10 , and there > are > >>>>>>> exactly 10 e-mails in the file > >>>>>>> 3. how big is the stream itself? i think it is below 2KB, > correct? > >>>>>>> 4. is the PCRE matching the e-mails, under the unix shell ? > >>>>>>> 5. yes i think you should get more sid:11 alerts - but first > lets > >>>>>>> investigate the above 4. > >>>>>>> > >>>>>>> thanks > >>>>>>> > >>>>>>> -- > >>>>>>> Peter Manev > >>>>>> > >>>>>> The file with only the 10 emails is 160 bytes. Even without > pcre I > >>>>>> get the same result : > >>>>>> > >>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>> (msg:"got one email addr"; content:"|40|"; \ > >>>>>> flow:established,from_server; flowint:something,notset; > >>>>>> flowint:something,=,1; sid:10; priority:3;) > >>>>>> > >>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>> (msg:"got more email addrs"; content:"|40|"; \ > >>>>>> flow:established,from_server; flowint:something,isset; > >>>>>> flowint:something,+,1; sid:11; priority:2;) > >>>>>> > >>>>>> alert tcp $HOME_NET 80 -> any any \ > >>>>>> (msg:"Got too many email addrs!"; content:"|40|"; \ > >>>>>> flow:established,from_server; flowint:something,isset; > >>>>>> flowint:something,>,9; sid:12; priority:1; > >>>>>> classtype:policy-violation;) > >>>>>> > >>>>>> > >>>>>> alerts I get : > >>>>>> > >>>>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr > [**] > >>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58158 > >>>>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs > >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58158 > >>>>>> > >>>>>> If I put some '#' symbols between the emails in the file so that > >>>>>> it gets about 9K big and I fetch it I get these alerts : > >>>>>> > >>>>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr > [**] > >>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58166 > >>>>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs > >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58166 > >>>>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs > >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58166 > >>>>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs > >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 -> > >>>>>> Y.Y.Y.Y:58166 > >>>>>> > >>>>>> > >>>>>> > >>>>>> Hi Nikolay, > >>>>>> > >>>>>> > >>>>>> Can you please post this as a bug - please be detailed (as you were > in > >>>>>> your 2 previous e-mails). > >>>>>> Personally i think here sid 11 is the problem , may be it does not > >>>>>> count/increment correctly.... > >>>>>> thanks > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Peter Manev > >>>>> > >>>>> Yes I will post this as a bug. But I've just found a much simpler > case. > >>>>> > >>>>> Let's for example have only this rule in suricata : > >>>>> > >>>>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";) > >>>>> > >>>>> Then on a monitored machine from the $HOME_NET range I do : > >>>>> > >>>>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666 > >>>>> > >>>>> And on different host I do : > >>>>> > >>>>> nc testserver 6666 > >>>>> > >>>>> This gets the ten @ chars transferred, and I get only one alert. > >>>>> But for example if I echo more @ chars, like 5000 or something, I get > >>>>> 3-6 alerts. > >>>>> I have to check what is actually the number of packets with payload, > >>>>> probably the rule > >>>>> is matched once per packet? But this could not explain that I get > >>>>> different number of alerts on different runs. > >>>> > >>>> The behavior is by design. TCP data by default is inspected in the > >>>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is > inspected > >>>> at once. > >>> > >>> > >>>> > >>>> Suricata will not try to find every possible match in a > >>>> payload, but just one. > >>> > >>> That's good to know - clears out a few questions of mine.... > >>> but then a PCRE (matching on 10 "@") should match all of them - > correct? > >>> having in mind they are in the same "chunk". > >>> > >> > >> If I have understood your question right, no! Pcre works just like > >> content on the first match it finds. So alerts wise or match wise it > >> should work the same as using content > >> > > > > So this means that there is no way to count the total number of > occurrences of a > > given string or pattern in a flow, and alert if some predefined number > is reached? > > i.e. no matter the number I will get one alert per chunk? > > > > Something like 'g' (global match) flag for pcre? This will definitely be > very expensive, but looks interesting as feature. > > > > You won't be able to alert for every pattern in the flow, but you can > alert once by counting the no of patterns present in the stream using > pcre. You don't need 'g' or such feature to do so. > > Something like > > pcre:"/(kaboom.*){3}/"; > That's what i had in mind ... > > should do the trick by alerting if the pattern kaboom is present in > the string thrice. > > But I'd suggest avoiding pcre under most circumstances. > > >>> > >>>> > >>>> > >>>> The reason you get more alerts if you increase the payload > >>>> significantly, is that the stream is inspected in chunks. The size of > >>>> those chunks is determined by your stream toserver_chunk_size setting. > >>>> > >>>> -- > >>>> --------------------------------------------- > >>>> Victor Julien > >>>> http://www.inliniac.net/ > >>>> PGP: http://www.inliniac.net/victorjulien.asc > >>>> --------------------------------------------- > >>>> > >>>> _______________________________________________ > >>>> Oisf-users mailing list > >>>> Oisf-users at openinfosecfoundation.org > >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >>> > >>> > >>> > >>> > >>> -- > >>> Peter Manev > >>> > >>> _______________________________________________ > >>> Oisf-users mailing list > >>> Oisf-users at openinfosecfoundation.org > >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > >>> > >> > >> > >> > >> -- > >> Anoop Saldanha > >> _______________________________________________ > >> Oisf-users mailing list > >> Oisf-users at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > -- > Anoop Saldanha > -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/1e430fb4/attachment-0001.html From victor at inliniac.net Wed Feb 15 05:53:39 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 15 Feb 2012 11:53:39 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> Message-ID: <4F3B8EB3.9030204@inliniac.net> On 02/15/2012 10:26 AM, Nikolay Denev wrote: > > On Feb 15, 2012, at 10:07 AM, Victor Julien wrote: > >> On 02/15/2012 06:42 AM, Nikolay Denev wrote: >>> Hi, >>> >>> I'm wondering how suricata decides which packet to capture and dump >>> in the unified2 log file and which not to. >>> >>> I'm running Snorby to collect the alerts, and I've noticed that >>> sometimes for a single rule, some of the alerts have >>> the packet dump present and some not. >> >> That is odd. There should always be a packet. Is this happening with >> specific rules and / or traffic? >> > I've checked now. Some of the alerts without packet dump are packets > with only headers and no payload, > for example syn packets from RBN listed IPs. Which should be normal. But > I have also alert from this rule: > > alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ETPRO TROJAN > Win32/Chir.B at mm User-Agent (KPeerUpdater)"; flow:to_server,established; > content:"User-Agent|3a| KPeerUpdater|0d 0a|"; http_header; > reference:url,www.threatexpert.com/report.aspx?md5=5ca614132b183b2812cb69112879237f > ; > reference:url,www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Worm%3AWin32%2FChir.B%40mm > ; > classtype:trojan-activity; sid:2803871; rev:2;) > > And in snorby I see no packet dump, and packet len is 40? Might be the ACK packet triggering the reassembly. Still should have logged the packet I think. Can you enable the alert-debug.log in Suricata for a while? When you see this issue again, see what it logs. Btw, in your screen shot the SEQ and ACK values are the same. That seems unusual as well. > > I can also look in the unified2.alert file to make sure it's not snorby > problem. (if I can find some tool to check it :) ) > Let's focus on what Suricata does right now. We've had issues with unified2 in the past. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From victor at inliniac.net Wed Feb 15 05:57:11 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 15 Feb 2012 11:57:11 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <4F3B8EB3.9030204@inliniac.net> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> Message-ID: <4F3B8F87.2050308@inliniac.net> On 02/15/2012 11:53 AM, Victor Julien wrote: > Might be the ACK packet triggering the reassembly. Still should have > logged the packet I think. Still should have logged the *stream* is what I meant to say here. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From ndenev at gmail.com Wed Feb 15 05:59:05 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 12:59:05 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <4F3B8EB3.9030204@inliniac.net> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> Message-ID: On Feb 15, 2012, at 12:53 PM, Victor Julien wrote: > On 02/15/2012 10:26 AM, Nikolay Denev wrote: >> >> On Feb 15, 2012, at 10:07 AM, Victor Julien wrote: >> >>> On 02/15/2012 06:42 AM, Nikolay Denev wrote: >>>> Hi, >>>> >>>> I'm wondering how suricata decides which packet to capture and dump >>>> in the unified2 log file and which not to. >>>> >>>> I'm running Snorby to collect the alerts, and I've noticed that >>>> sometimes for a single rule, some of the alerts have >>>> the packet dump present and some not. >>> >>> That is odd. There should always be a packet. Is this happening with >>> specific rules and / or traffic? >>> >> I've checked now. Some of the alerts without packet dump are packets >> with only headers and no payload, >> for example syn packets from RBN listed IPs. Which should be normal. But >> I have also alert from this rule: >> >> alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ETPRO TROJAN >> Win32/Chir.B at mm User-Agent (KPeerUpdater)"; flow:to_server,established; >> content:"User-Agent|3a| KPeerUpdater|0d 0a|"; http_header; >> reference:url,www.threatexpert.com/report.aspx?md5=5ca614132b183b2812cb69112879237f >> ; >> reference:url,www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Worm%3AWin32%2FChir.B%40mm >> ; >> classtype:trojan-activity; sid:2803871; rev:2;) >> >> And in snorby I see no packet dump, and packet len is 40? > > Might be the ACK packet triggering the reassembly. Still should have > logged the packet I think. > > Can you enable the alert-debug.log in Suricata for a while? When you see > this issue again, see what it logs. > > Btw, in your screen shot the SEQ and ACK values are the same. That seems > unusual as well. > > >> >> I can also look in the unified2.alert file to make sure it's not snorby >> problem. (if I can find some tool to check it :) ) >> > > Let's focus on what Suricata does right now. We've had issues with > unified2 in the past. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > Ok, I've enabled alert-debug.log now. I'll let you know when I have something. From ndenev at gmail.com Wed Feb 15 06:28:47 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 13:28:47 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <4F3B8EB3.9030204@inliniac.net> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> Message-ID: <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> On Feb 15, 2012, at 12:53 PM, Victor Julien wrote: > On 02/15/2012 10:26 AM, Nikolay Denev wrote: >> >> On Feb 15, 2012, at 10:07 AM, Victor Julien wrote: >> >>> On 02/15/2012 06:42 AM, Nikolay Denev wrote: >>>> Hi, >>>> >>>> I'm wondering how suricata decides which packet to capture and dump >>>> in the unified2 log file and which not to. >>>> >>>> I'm running Snorby to collect the alerts, and I've noticed that >>>> sometimes for a single rule, some of the alerts have >>>> the packet dump present and some not. >>> >>> That is odd. There should always be a packet. Is this happening with >>> specific rules and / or traffic? >>> >> I've checked now. Some of the alerts without packet dump are packets >> with only headers and no payload, >> for example syn packets from RBN listed IPs. Which should be normal. But >> I have also alert from this rule: >> >> alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ETPRO TROJAN >> Win32/Chir.B at mm User-Agent (KPeerUpdater)"; flow:to_server,established; >> content:"User-Agent|3a| KPeerUpdater|0d 0a|"; http_header; >> reference:url,www.threatexpert.com/report.aspx?md5=5ca614132b183b2812cb69112879237f >> ; >> reference:url,www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Worm%3AWin32%2FChir.B%40mm >> ; >> classtype:trojan-activity; sid:2803871; rev:2;) >> >> And in snorby I see no packet dump, and packet len is 40? > > Might be the ACK packet triggering the reassembly. Still should have > logged the packet I think. > > Can you enable the alert-debug.log in Suricata for a while? When you see > this issue again, see what it logs. > > Btw, in your screen shot the SEQ and ACK values are the same. That seems > unusual as well. > > >> >> I can also look in the unified2.alert file to make sure it's not snorby >> problem. (if I can find some tool to check it :) ) >> > > Let's focus on what Suricata does right now. We've had issues with > unified2 in the past. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > Here's one such alert, but there is packet data in the alert-debug file : (also packet len differs, maybe snorby issue?) +================ TIME: 02/15/2012-13:18:15.459170 SRC IP: X.X.X.X DST IP: Y.Y.Y.Y PROTO: 6 SRC PORT: 55192 DST PORT: 80 TCP SEQ: 1360766462 TCP ACK: 1891794325 FLOW: to_server: TRUE, to_client: FALSE FLOW Start TS: 02/15/2012-13:18:15.017736 FLOW IPONLY SET: TOSERVER: TRUE, TOCLIENT: TRUE FLOW ACTION: DROP: FALSE, PASS FALSE FLOW NOINSPECTION: PACKET: FALSE, PAYLOAD: FALSE, APP_LAYER: FALSE FLOW APP_LAYER: DETECTED: TRUE, PROTO 1 PACKET LEN: 68 PACKET: 0000 02 04 96 37 53 8D F0 DE F1 95 DF F7 81 00 00 00 ...7S... ........ 0010 81 00 00 70 08 00 45 00 00 28 1E 3B 40 00 80 06 ...p..E. .(.;@... 0020 AF 37 0A 81 15 24 6B 14 A2 A4 D7 98 00 50 51 1B .7...$k. .....PQ. 0030 A5 FE 70 C2 7D 95 50 10 01 00 C4 1C 00 00 00 00 ..p.}.P. ........ 0040 00 00 00 00 .... ALERT CNT: 1 ALERT MSG [00]: ETPRO POLICY dl.dropbox Download ALERT GID [00]: 1 ALERT SID [00]: 2804233 ALERT REV [00]: 3 ALERT CLASS [00]: Potential Corporate Privacy Violation ALERT PRIO [00]: 1 ALERT FOUND IN [00]: OTHER -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/3d681666/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: snorby.png Type: image/png Size: 83758 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/3d681666/snorby-0001.png From petermanev at gmail.com Wed Feb 15 06:52:44 2012 From: petermanev at gmail.com (Peter Manev) Date: Wed, 15 Feb 2012 12:52:44 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> Message-ID: Just from observation - "PACKET LEN: 68" in debug alert but in Snorby it says "40" - so it does seem there is a bit of discrepancy .... If you use pcap.log(ing) in yaml , does this packet indeed have 68 or 40 length ? -- Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/f70ac047/attachment.html From ndenev at gmail.com Wed Feb 15 08:29:54 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 15:29:54 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> Message-ID: <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> On Feb 15, 2012, at 1:52 PM, Peter Manev wrote: > > Just from observation - > "PACKET LEN: 68" in debug alert > but in Snorby it says "40" - so it does seem there is a bit of discrepancy .... > If you use pcap.log(ing) in yaml , does this packet indeed have 68 or 40 length ? > > -- > Peter Manev I've just turned on pcap-log in suricata.yaml. From ndenev at gmail.com Wed Feb 15 09:31:56 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Wed, 15 Feb 2012 16:31:56 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> Message-ID: On Feb 15, 2012, at 3:29 PM, Nikolay Denev wrote: > > On Feb 15, 2012, at 1:52 PM, Peter Manev wrote: > >> >> Just from observation - >> "PACKET LEN: 68" in debug alert >> but in Snorby it says "40" - so it does seem there is a bit of discrepancy .... >> If you use pcap.log(ing) in yaml , does this packet indeed have 68 or 40 length ? >> >> -- >> Peter Manev > > I've just turned on pcap-log in suricata.yaml. > Ok here's another one. The rule is : alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE Alexa Spyware Reporting"; flow:established,to_server; uricontent:"/data?"; nocase; uricontent:"cli="; nocase; uricontent:"&dat="; nocase; uricontent:"&ver="; nocase; uricontent:"&uid="; nocase; reference:url,doc.emergingthreats.net/bin/view/Main/2003219; classtype:trojan-activity; sid:2003219; rev:4;) +================ TIME: 02/15/2012-16:02:56.567244 SRC IP: X.X.X.X DST IP: Y.Y.Y.Y PROTO: 6 SRC PORT: 58761 DST PORT: 80 TCP SEQ: 3317584075 TCP ACK: 2654953614 FLOW: to_server: TRUE, to_client: FALSE FLOW Start TS: 02/15/2012-16:02:56.295055 FLOW IPONLY SET: TOSERVER: TRUE, TOCLIENT: TRUE FLOW ACTION: DROP: FALSE, PASS FALSE FLOW NOINSPECTION: PACKET: FALSE, PAYLOAD: FALSE, APP_LAYER: TRUE FLOW APP_LAYER: DETECTED: TRUE, PROTO 1 PACKET LEN: 68 PACKET: 0000 02 04 96 37 53 8D F0 DE F1 75 DD AE 81 00 00 00 ...7S... .u...... 0010 81 00 00 6C 08 00 45 00 00 28 24 DD 40 00 80 06 ...l..E. .($. at ... 0020 CF F9 0A 81 0D 47 4B 65 A2 CC E5 89 00 50 C5 BE .....GKe .....P.. 0030 50 CB 9E 3F 60 8E 50 10 3F 05 6F A4 00 00 00 00 P..?`.P. ?.o..... 0040 00 00 00 00 .... ALERT CNT: 1 ALERT MSG [00]: ET MALWARE Alexa Spyware Reporting ALERT GID [00]: 1 ALERT SID [00]: 2003219 ALERT REV [00]: 4 ALERT CLASS [00]: A Network Trojan was Detected ALERT PRIO [00]: 1 ALERT FOUND IN [00]: OTHER +================ And this is from the pcap log : 16:02:56.295055 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [S], seq 3317583354, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0 16:02:56.425664 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [.], ack 2654949527, win 16425, length 0 16:02:56.425473 IP Y.Y.Y.Y.80 > X.X.X.X.58761: Flags [S.], seq 2654949526, ack 3317583355, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 2], length 0 16:02:56.426276 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [P.], seq 1:721, ack 1, win 16425, length 720 16:02:56.563356 IP truncated-ip - 4 bytes missing! Y.Y.Y.Y.80 > X.X.X.X.58761: Flags [.], seq 1:1461, ack 721, win 1820, length 1460 16:02:56.563365 IP truncated-ip - 4 bytes missing! Y.Y.Y.Y.80 > X.X.X.X.58761: Flags [.], seq 1461:2921, ack 721, win 1820, length 1460 16:02:56.563927 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [.], ack 2921, win 16425, length 0 16:02:56.564533 IP Y.Y.Y.Y.80 > X.X.X.X.58761: Flags [FP.], seq 2921:4087, ack 721, win 1820, length 1166 16:02:56.567872 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [R.], seq 721, ack 4088, win 0, length 0 16:02:56.567244 IP X.X.X.X.58761 > Y.Y.Y.Y.80: Flags [.], ack 4088, win 16133, length 0 Notice the truncated-ip packets. My interface is Intel 10G card with MTU 9000 and suricata is set : default-packet-size: 1522 The switch has the port set like this "Jumbo: Enabled, MTU= 9216" Also, just for info, this is extreme networks switch, that is mirroring the packets in one direction with VLAN tag, and and untagged in the other. Just like the recent thred in oisf-users@ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/793e676d/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: snorby-alexa.png Type: image/png Size: 83164 bytes Desc: not available Url : http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/793e676d/snorby-alexa-0001.png From petermanev at gmail.com Wed Feb 15 10:03:32 2012 From: petermanev at gmail.com (Peter Manev) Date: Wed, 15 Feb 2012 16:03:32 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> Message-ID: Ok, Just a couple of suggestions: 1. Make the MTU on the suricata box equal the MTU on the switch port where it is connected to. 2. The interface that Suricata listens on (ex. eth0) , does it have all the VLANs untagged there? Or some are tagged and some untagged? Because if not - that might a problem. and if you could check that these two have any different effect ? ... thanks -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/f4ac14d2/attachment.html From c.d.wakelin at reading.ac.uk Wed Feb 15 16:20:00 2012 From: c.d.wakelin at reading.ac.uk (Chris Wakelin) Date: Wed, 15 Feb 2012 21:20:00 +0000 Subject: [Oisf-users] New MPM available In-Reply-To: References: <4F3A2DBA.9060009@inliniac.net> Message-ID: <4F3C2180.70608@reading.ac.uk> On 15/02/12 08:08, Anoop Saldanha wrote: > On Tue, Feb 14, 2012 at 3:17 PM, Victor Julien wrote: >> On 02/14/2012 10:22 AM, Anoop Saldanha wrote: >>> Hello all, >>> >>> We have a new MPM available in our codebase - "ac-bs". This provides >>> compression that's pretty close to ac-gfbs, while performing better >>> than ac-gfbs. >>> >>> To use this mpm, set >>> >>> "mpm-algo: ac-bs" in the conf file. >>> >>> Would appreciate performance numbers with both >>> >>> "sgh-mpm-context:full" >>> and >>> "sgh-mpm-context:single" >>> > Keen to see how ac-bs performs with "full" sgh-mpm-context, against other mpms. Here's a quick test against a 1.2GB 1.7m pcap with ~4k rules, 13 alerts, 10k URLs: ac/context=full/profile=medium/runmode=single: 30.8s ~4.5GB max mem ac-bs/ : 46.5s ~3GB max mem ac-gfbs/ : 53s ~3GB max mem Much the same with profile=high (slightly less time for ac-bs and ac-gfbs and more for ac). I guess we need a bigger ruleset to see a difference. ac/context=full/profile=medium/runmode=autofp (8 detect threads, CPU affinity set): 13s ~5GB max mem ac-bs/: 16.1s ~3.2GB max mem ac-gfbs/: 18.1s ~3.2GB max mem so slightly more memory and much faster! The upshot seems to be that it's somewhere between ac and ac-gfbs for performance whilst using the same memory as ac-gfbs. Best Wishes, Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin at reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094 From ndenev at gmail.com Thu Feb 16 08:40:20 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 16 Feb 2012 15:40:20 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> Message-ID: On Feb 15, 2012, at 5:03 PM, Peter Manev wrote: > > Ok, > Just a couple of suggestions: > 1. Make the MTU on the suricata box equal the MTU on the switch port where it is connected to. I don't think this is an issue, as all of the other ports are not jumbo frames enabled, and I don't have frames bigger than 1522 bytes. > 2. The interface that Suricata listens on (ex. eth0) , does it have all the VLANs untagged there? Or some are tagged and some untagged? Because if not - that might a problem. > I was wrong, there are some untagged packets, but they are mirrored LACP and LLDP frames. All the IP traffic from the mirrored ports is QinQ tagged, with the outer tag with VLAN 0 and the inner with the actual vlan being mirrored (all the ports that I mirror have only VLAN tagged traffic) Suricata seems to handle this OK, probably ignores the vlan tag? > and if you could check that these two have any different effect ? ... > > thanks > > -- > Regards, > Peter Manev > From petermanev at gmail.com Thu Feb 16 08:51:31 2012 From: petermanev at gmail.com (Peter Manev) Date: Thu, 16 Feb 2012 14:51:31 +0100 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> Message-ID: On Thu, Feb 16, 2012 at 2:40 PM, Nikolay Denev wrote: > > On Feb 15, 2012, at 5:03 PM, Peter Manev wrote: > > > > > Ok, > > Just a couple of suggestions: > > 1. Make the MTU on the suricata box equal the MTU on the switch port > where it is connected to. > > I don't think this is an issue, as all of the other ports are not jumbo > frames enabled, and I don't have frames bigger than 1522 bytes. > > Ok, I thought you have frames bigger than that... > > 2. The interface that Suricata listens on (ex. eth0) , does it have all > the VLANs untagged there? Or some are tagged and some untagged? Because if > not - that might a problem. > > > > I was wrong, there are some untagged packets, but they are mirrored LACP > and LLDP frames. > All the IP traffic from the mirrored ports is QinQ tagged, with the outer > tag with VLAN 0 and the inner with the actual vlan being mirrored (all the > ports that I mirror have only VLAN tagged traffic) > Suricata seems to handle this OK, probably ignores the vlan tag? > > > In that case the only discrepancy I see is the reported length of the packet in Suricata and Snorby - to me it looks like Suri reports it correctly (debug log), but Snorby does not. I can not be sure because from what I saw as a screen shot of Snorby , the packet has the same SEQ and ACK number - kind of strange, and the SEQ or ACK number there does not much the one in debug log.... can you please confirm that? > > and if you could check that these two have any different effect ? ... > > > > thanks > > > > -- > > Regards, > > Peter Manev > > > > -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120216/145ff2d7/attachment.html From ndenev at gmail.com Thu Feb 16 09:10:26 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 16 Feb 2012 16:10:26 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> Message-ID: <6F8D4DEC-6254-4F66-B827-256413EF67DC@gmail.com> On Feb 16, 2012, at 3:51 PM, Peter Manev wrote: > > > On Thu, Feb 16, 2012 at 2:40 PM, Nikolay Denev wrote: > > On Feb 15, 2012, at 5:03 PM, Peter Manev wrote: > > > > > Ok, > > Just a couple of suggestions: > > 1. Make the MTU on the suricata box equal the MTU on the switch port where it is connected to. > > I don't think this is an issue, as all of the other ports are not jumbo frames enabled, and I don't have frames bigger than 1522 bytes. > > > Ok, I thought you have frames bigger than that... > > 2. The interface that Suricata listens on (ex. eth0) , does it have all the VLANs untagged there? Or some are tagged and some untagged? Because if not - that might a problem. > > > > I was wrong, there are some untagged packets, but they are mirrored LACP and LLDP frames. > All the IP traffic from the mirrored ports is QinQ tagged, with the outer tag with VLAN 0 and the inner with the actual vlan being mirrored (all the ports that I mirror have only VLAN tagged traffic) > Suricata seems to handle this OK, probably ignores the vlan tag? > > > > In that case the only discrepancy I see is the reported length of the packet in Suricata and Snorby - to me it looks like Suri reports it correctly (debug log), but Snorby does not. I can not be sure because from what I saw as a screen shot of Snorby , the packet has the same SEQ and ACK number - kind of strange, and the SEQ or ACK number there does not much the one in debug log.... can you please confirm that? > Yep, this is what I'm seeing too. I have to probably see what exactly is in the unified2 log. Cause after suricata logs it, then it is touched by barnyard2 and then imported into the snorby mysql db, then snorby displays the alert, so there are other places that can break things. Speaking of this, does anybody know a tool to display unified2 logs? I've found a perl module but it seems to not properly display the packet dump. > > and if you could check that these two have any different effect ? ... > > > > thanks > > > > -- > > Regards, > > Peter Manev > > > > > > > -- > Regards, > Peter Manev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120216/2583d868/attachment.html From ndenev at gmail.com Thu Feb 16 09:16:32 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 16 Feb 2012 16:16:32 +0200 Subject: [Oisf-users] Config options discrepancies between suricata.yaml and ConfNodeLookupChildValue() in the source. Message-ID: Hi, I've noticed that there are some discrepancies regarding the config options that are in the default suricata.yaml file, that I guess most people use as a starting point to modify for their needs. For example I've tried to set the pcap logging to use only 10 files and rotate them, and I noticed that this didn't work. I've found that the source uses "ConfNodeLookupChildValue(conf, "max-files");" to get the number of files, but suricata.yaml has "max_files", so this option is not parsed. I see other similar mixups of dashes and underscores like : /usr/local/etc/suricata/suricata.yaml: use_stream_depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets ./log-pcap.c: use_stream_depth = ConfNodeLookupChildValue(conf, "use-stream-depth"); ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil-base-dir"); ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil_base_dir"); ts-format and ts_format? I can try to prepare a patch for these when I have free time, if it's clear what should be the convention: underscore or dash. P.S. running suricata-1.2.1 release From victor at inliniac.net Thu Feb 16 11:30:52 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 16 Feb 2012 17:30:52 +0100 Subject: [Oisf-users] Config options discrepancies between suricata.yaml and ConfNodeLookupChildValue() in the source. In-Reply-To: References: Message-ID: <4F3D2F3C.30308@inliniac.net> On 02/16/2012 03:16 PM, Nikolay Denev wrote: > Hi, > > I've noticed that there are some discrepancies regarding the config options that are in the default suricata.yaml file, that > I guess most people use as a starting point to modify for their needs. For example I've tried to set the pcap logging to use only 10 files and rotate them, > and I noticed that this didn't work. I've found that the source uses "ConfNodeLookupChildValue(conf, "max-files");" to get the number of files, but suricata.yaml > has "max_files", so this option is not parsed. > I see other similar mixups of dashes and underscores like : > > /usr/local/etc/suricata/suricata.yaml: use_stream_depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets > ./log-pcap.c: use_stream_depth = ConfNodeLookupChildValue(conf, "use-stream-depth"); > > ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil-base-dir"); > ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil_base_dir"); > > > ts-format and ts_format? > > I can try to prepare a patch for these when I have free time, if it's clear what should be the convention: underscore or dash. Yeah it's inconsistent. The goal is to have the dash approach everywhere. Complicating things is that for every setting we convert, I want the old way to continue to work. So backwards compatibility, like with the sguil-base-dir. Still interested in helping out? :) -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From ndenev at gmail.com Thu Feb 16 11:44:25 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Thu, 16 Feb 2012 18:44:25 +0200 Subject: [Oisf-users] Config options discrepancies between suricata.yaml and ConfNodeLookupChildValue() in the source. In-Reply-To: <4F3D2F3C.30308@inliniac.net> References: <4F3D2F3C.30308@inliniac.net> Message-ID: <01E16982-FAEC-4F05-ACFE-2AF87674AE9B@gmail.com> On Feb 16, 2012, at 6:30 PM, Victor Julien wrote: > On 02/16/2012 03:16 PM, Nikolay Denev wrote: >> Hi, >> >> I've noticed that there are some discrepancies regarding the config options that are in the default suricata.yaml file, that >> I guess most people use as a starting point to modify for their needs. For example I've tried to set the pcap logging to use only 10 files and rotate them, >> and I noticed that this didn't work. I've found that the source uses "ConfNodeLookupChildValue(conf, "max-files");" to get the number of files, but suricata.yaml >> has "max_files", so this option is not parsed. >> I see other similar mixups of dashes and underscores like : >> >> /usr/local/etc/suricata/suricata.yaml: use_stream_depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets >> ./log-pcap.c: use_stream_depth = ConfNodeLookupChildValue(conf, "use-stream-depth"); >> >> ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil-base-dir"); >> ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil_base_dir"); >> >> >> ts-format and ts_format? >> >> I can try to prepare a patch for these when I have free time, if it's clear what should be the convention: underscore or dash. > > Yeah it's inconsistent. The goal is to have the dash approach > everywhere. Complicating things is that for every setting we convert, I > want the old way to continue to work. So backwards compatibility, like > with the sguil-base-dir. > > Still interested in helping out? :) > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- Hmm :) Does converting all underscores to dashes in the ConfNodeLookup* functions (emitting warning about the uncerscore versions being deprecated) and using only dashes internally makes sense? This way even "sguil_base-dir" would work as a side effect, but I doubt it would break anything. From victor at inliniac.net Thu Feb 16 11:46:20 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 16 Feb 2012 17:46:20 +0100 Subject: [Oisf-users] Config options discrepancies between suricata.yaml and ConfNodeLookupChildValue() in the source. In-Reply-To: <01E16982-FAEC-4F05-ACFE-2AF87674AE9B@gmail.com> References: <4F3D2F3C.30308@inliniac.net> <01E16982-FAEC-4F05-ACFE-2AF87674AE9B@gmail.com> Message-ID: <4F3D32DC.9090007@inliniac.net> On 02/16/2012 05:44 PM, Nikolay Denev wrote: > > On Feb 16, 2012, at 6:30 PM, Victor Julien wrote: > >> On 02/16/2012 03:16 PM, Nikolay Denev wrote: >>> Hi, >>> >>> I've noticed that there are some discrepancies regarding the config options that are in the default suricata.yaml file, that >>> I guess most people use as a starting point to modify for their needs. For example I've tried to set the pcap logging to use only 10 files and rotate them, >>> and I noticed that this didn't work. I've found that the source uses "ConfNodeLookupChildValue(conf, "max-files");" to get the number of files, but suricata.yaml >>> has "max_files", so this option is not parsed. >>> I see other similar mixups of dashes and underscores like : >>> >>> /usr/local/etc/suricata/suricata.yaml: use_stream_depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets >>> ./log-pcap.c: use_stream_depth = ConfNodeLookupChildValue(conf, "use-stream-depth"); >>> >>> ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil-base-dir"); >>> ./log-pcap.c: s_dir = ConfNodeLookupChildValue(conf, "sguil_base_dir"); >>> >>> >>> ts-format and ts_format? >>> >>> I can try to prepare a patch for these when I have free time, if it's clear what should be the convention: underscore or dash. >> >> Yeah it's inconsistent. The goal is to have the dash approach >> everywhere. Complicating things is that for every setting we convert, I >> want the old way to continue to work. So backwards compatibility, like >> with the sguil-base-dir. >> >> Still interested in helping out? :) >> >> -- >> --------------------------------------------- >> Victor Julien >> http://www.inliniac.net/ >> PGP: http://www.inliniac.net/victorjulien.asc >> --------------------------------------------- > > Hmm :) > Does converting all underscores to dashes in the ConfNodeLookup* functions (emitting warning about the uncerscore versions being deprecated) and using only dashes internally makes sense? > This way even "sguil_base-dir" would work as a side effect, but I doubt it would break anything. > Thats actually a pretty good idea, as long as we only do it for the option name, not the value. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From anoopsaldanha at gmail.com Fri Feb 17 10:29:07 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Fri, 17 Feb 2012 20:59:07 +0530 Subject: [Oisf-users] New MPM available In-Reply-To: <4F3C2180.70608@reading.ac.uk> References: <4F3A2DBA.9060009@inliniac.net> <4F3C2180.70608@reading.ac.uk> Message-ID: On Thu, Feb 16, 2012 at 2:50 AM, Chris Wakelin wrote: > On 15/02/12 08:08, Anoop Saldanha wrote: >> On Tue, Feb 14, 2012 at 3:17 PM, Victor Julien wrote: >>> On 02/14/2012 10:22 AM, Anoop Saldanha wrote: >>>> Hello all, >>>> >>>> We have a new MPM available in our codebase - "ac-bs". ?This provides >>>> compression that's pretty close to ac-gfbs, while performing better >>>> than ac-gfbs. >>>> >>>> To use this mpm, set >>>> >>>> "mpm-algo: ac-bs" in the conf file. >>>> >>>> Would appreciate performance numbers with both >>>> >>>> "sgh-mpm-context:full" >>>> and >>>> "sgh-mpm-context:single" >>>> > > > >> Keen to see how ac-bs performs with "full" sgh-mpm-context, against other mpms. > > Here's a quick test against a 1.2GB 1.7m pcap with ~4k rules, 13 alerts, > 10k URLs: > > ac/context=full/profile=medium/runmode=single: 30.8s ~4.5GB max mem > ac-bs/ : 46.5s ~3GB max mem > ac-gfbs/ : 53s ~3GB max mem > > Much the same with profile=high (slightly less time for ac-bs and > ac-gfbs and more for ac). I guess we need a bigger ruleset to see a > difference. > > ac/context=full/profile=medium/runmode=autofp (8 detect threads, CPU > affinity set): 13s ~5GB max mem > ac-bs/: 16.1s ~3.2GB max mem > ac-gfbs/: 18.1s ~3.2GB max mem > > so slightly more memory and much faster! > > The upshot seems to be that it's somewhere between ac and ac-gfbs for > performance whilst using the same memory as ac-gfbs. > > Best Wishes, > Chris > > -- Thanks for the nos. The new one in full mode probably looks like an alternative to ac/single, if we have enough memory to run the engine with ac-bs/full, but not ac/full -- Anoop Saldanha From victor at inliniac.net Fri Feb 17 11:38:31 2012 From: victor at inliniac.net (Victor Julien) Date: Fri, 17 Feb 2012 17:38:31 +0100 Subject: [Oisf-users] New MPM available In-Reply-To: References: <4F3A2DBA.9060009@inliniac.net> <4F3C2180.70608@reading.ac.uk> Message-ID: <4F3E8287.2060105@inliniac.net> On 02/17/2012 04:29 PM, Anoop Saldanha wrote: > On Thu, Feb 16, 2012 at 2:50 AM, Chris Wakelin > wrote: >> On 15/02/12 08:08, Anoop Saldanha wrote: >>> On Tue, Feb 14, 2012 at 3:17 PM, Victor Julien wrote: >>>> On 02/14/2012 10:22 AM, Anoop Saldanha wrote: >>>>> Hello all, >>>>> >>>>> We have a new MPM available in our codebase - "ac-bs". This provides >>>>> compression that's pretty close to ac-gfbs, while performing better >>>>> than ac-gfbs. >>>>> >>>>> To use this mpm, set >>>>> >>>>> "mpm-algo: ac-bs" in the conf file. >>>>> >>>>> Would appreciate performance numbers with both >>>>> >>>>> "sgh-mpm-context:full" >>>>> and >>>>> "sgh-mpm-context:single" >>>>> >> >> >> >>> Keen to see how ac-bs performs with "full" sgh-mpm-context, against other mpms. >> >> Here's a quick test against a 1.2GB 1.7m pcap with ~4k rules, 13 alerts, >> 10k URLs: >> >> ac/context=full/profile=medium/runmode=single: 30.8s ~4.5GB max mem >> ac-bs/ : 46.5s ~3GB max mem >> ac-gfbs/ : 53s ~3GB max mem >> >> Much the same with profile=high (slightly less time for ac-bs and >> ac-gfbs and more for ac). I guess we need a bigger ruleset to see a >> difference. >> >> ac/context=full/profile=medium/runmode=autofp (8 detect threads, CPU >> affinity set): 13s ~5GB max mem >> ac-bs/: 16.1s ~3.2GB max mem >> ac-gfbs/: 18.1s ~3.2GB max mem >> >> so slightly more memory and much faster! >> >> The upshot seems to be that it's somewhere between ac and ac-gfbs for >> performance whilst using the same memory as ac-gfbs. >> >> Best Wishes, >> Chris >> >> -- > > Thanks for the nos. > > The new one in full mode probably looks like an alternative to > ac/single, if we have enough memory to run the engine with ac-bs/full, > but not ac/full > I wonder what the performance & memory usage would be of (algo/ctx/profile): ac/single/medium vs ac-bs/full/high If I find some time I'll test it myself. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From victor at inliniac.net Fri Feb 17 11:57:38 2012 From: victor at inliniac.net (Victor Julien) Date: Fri, 17 Feb 2012 17:57:38 +0100 Subject: [Oisf-users] New MPM available In-Reply-To: <4F3E8287.2060105@inliniac.net> References: <4F3A2DBA.9060009@inliniac.net> <4F3C2180.70608@reading.ac.uk> <4F3E8287.2060105@inliniac.net> Message-ID: <4F3E8702.8080402@inliniac.net> On 02/17/2012 05:38 PM, Victor Julien wrote: > On 02/17/2012 04:29 PM, Anoop Saldanha wrote: >> On Thu, Feb 16, 2012 at 2:50 AM, Chris Wakelin >> wrote: >>> On 15/02/12 08:08, Anoop Saldanha wrote: >>>> On Tue, Feb 14, 2012 at 3:17 PM, Victor Julien wrote: >>>>> On 02/14/2012 10:22 AM, Anoop Saldanha wrote: >>>>>> Hello all, >>>>>> >>>>>> We have a new MPM available in our codebase - "ac-bs". This provides >>>>>> compression that's pretty close to ac-gfbs, while performing better >>>>>> than ac-gfbs. >>>>>> >>>>>> To use this mpm, set >>>>>> >>>>>> "mpm-algo: ac-bs" in the conf file. >>>>>> >>>>>> Would appreciate performance numbers with both >>>>>> >>>>>> "sgh-mpm-context:full" >>>>>> and >>>>>> "sgh-mpm-context:single" >>>>>> >>> >>> >>> >>>> Keen to see how ac-bs performs with "full" sgh-mpm-context, against other mpms. >>> >>> Here's a quick test against a 1.2GB 1.7m pcap with ~4k rules, 13 alerts, >>> 10k URLs: >>> >>> ac/context=full/profile=medium/runmode=single: 30.8s ~4.5GB max mem >>> ac-bs/ : 46.5s ~3GB max mem >>> ac-gfbs/ : 53s ~3GB max mem >>> >>> Much the same with profile=high (slightly less time for ac-bs and >>> ac-gfbs and more for ac). I guess we need a bigger ruleset to see a >>> difference. >>> >>> ac/context=full/profile=medium/runmode=autofp (8 detect threads, CPU >>> affinity set): 13s ~5GB max mem >>> ac-bs/: 16.1s ~3.2GB max mem >>> ac-gfbs/: 18.1s ~3.2GB max mem >>> >>> so slightly more memory and much faster! >>> >>> The upshot seems to be that it's somewhere between ac and ac-gfbs for >>> performance whilst using the same memory as ac-gfbs. >>> >>> Best Wishes, >>> Chris >>> >>> -- >> >> Thanks for the nos. >> >> The new one in full mode probably looks like an alternative to >> ac/single, if we have enough memory to run the engine with ac-bs/full, >> but not ac/full >> > > I wonder what the performance & memory usage would be of (algo/ctx/profile): > > ac/single/medium > vs > ac-bs/full/high > > If I find some time I'll test it myself. > Okay, bad idea :) I killed ac-bs/full/high with emerging-all.rules at 12GB memory usage. ac-bs/full/medium used 4780MB but ran slower than ac/single/medium (830MB memory). -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From decanio.tom at gmail.com Sun Feb 19 17:49:14 2012 From: decanio.tom at gmail.com (Tom DeCanio) Date: Sun, 19 Feb 2012 14:49:14 -0800 Subject: [Oisf-users] New MPM available In-Reply-To: References: Message-ID: I just brought this up on the Tilera (tilegx). Haven't benchmarked it yet, but the tables do look much smaller than those produced by ac. Seems like this should improve performance here. When I get my benchmarking setup back I'll gather some new numbers. Tom On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha wrote: > Hello all, > > We have a new MPM available in our codebase - "ac-bs". This provides > compression that's pretty close to ac-gfbs, while performing better > than ac-gfbs. > > To use this mpm, set > > "mpm-algo: ac-bs" in the conf file. > > Would appreciate performance numbers with both > > "sgh-mpm-context:full" > and > "sgh-mpm-context:single" > > To give an explanation on what "sgh-mpm-context" and the params "full" > and "single" mean, these refer to how we set up mpm contexts. > "single" indicates that we use a single context for all the patterns > in the engine. "full" indicates that we split the patterns into many > mpm contexts, one mpm context per signature group head(sgh). > > To use "full" with a sufficiently decent ruleset(say > 10k rules with > a decent no of patterns) would require a lot of memory, running into a > couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case > of "ac". "single" solves this with a single context and hence the > smaller memory footprint for the engine. > > If the machine has sufficient memory, "full" is suggested as it > provides much better performance than "single", albeit at the cost of > increased memory consumption. More of a available_memory vs > performance scenario. > > Looking forward to some performance/memory feedback/benchmarks with > this mpm from the community. > > *mpm - multi pattern matcher > *sgh - signature group head > > -- > Anoop Saldanha > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120219/fa5a7ce4/attachment.html From anoopsaldanha at gmail.com Mon Feb 20 00:27:21 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Mon, 20 Feb 2012 10:57:21 +0530 Subject: [Oisf-users] New MPM available In-Reply-To: References: Message-ID: as a reference, these are the table sizes on my box with "ac-bs" for all the mpm contexts used by the engine, for a 18k ruleset * in bytes "ac-bs" 24348 38486 118900 47736 4716 4648804 558 15874 266202 6838 696 692 3982784 10756976 On Mon, Feb 20, 2012 at 4:19 AM, Tom DeCanio wrote: > I just brought this up on the Tilera (tilegx).? Haven't benchmarked it yet, > but the tables do look much smaller than those produced by ac.? Seems like > this should improve performance here.? When I get my benchmarking setup back > I'll gather some new numbers. > > Tom > > On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha > wrote: >> >> Hello all, >> >> We have a new MPM available in our codebase - "ac-bs". ?This provides >> compression that's pretty close to ac-gfbs, while performing better >> than ac-gfbs. >> >> To use this mpm, set >> >> "mpm-algo: ac-bs" in the conf file. >> >> Would appreciate performance numbers with both >> >> "sgh-mpm-context:full" >> and >> "sgh-mpm-context:single" >> >> To give an explanation on what "sgh-mpm-context" and the params "full" >> and "single" mean, these refer to how we set up mpm contexts. >> "single" indicates that we use a single context for all the patterns >> in the engine. ?"full" indicates that we split the patterns into many >> mpm contexts, one mpm context per signature group head(sgh). >> >> To use "full" with a sufficiently decent ruleset(say > 10k rules with >> a decent no of patterns) would require a lot of memory, running into a >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case >> of "ac". ?"single" solves this with a single context and hence the >> smaller memory footprint for the engine. >> >> If the machine has sufficient memory, "full" is suggested as it >> provides much better performance than "single", albeit at the cost of >> increased memory consumption. ?More of a available_memory vs >> performance scenario. >> >> Looking forward to some performance/memory feedback/benchmarks with >> this mpm from the community. >> >> *mpm - multi pattern matcher >> *sgh - signature group head >> >> -- >> Anoop Saldanha >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > -- Anoop Saldanha From ndenev at gmail.com Mon Feb 20 10:35:12 2012 From: ndenev at gmail.com (Nikolay Denev) Date: Mon, 20 Feb 2012 17:35:12 +0200 Subject: [Oisf-users] Packet capture dump in unified2 logs. In-Reply-To: <6F8D4DEC-6254-4F66-B827-256413EF67DC@gmail.com> References: <998BB910-E58C-49D9-8D9C-A03D9113726C@gmail.com> <4F3B67D8.7060806@inliniac.net> <9AB456BF-E2D2-450D-8906-CCA3739BBB4E@gmail.com> <4F3B8EB3.9030204@inliniac.net> <029E0FE7-551A-4861-9898-D6A85BCF2F7A@gmail.com> <046648BA-C9EA-4BD9-85FD-59769ECF6854@gmail.com> <6F8D4DEC-6254-4F66-B827-256413EF67DC@gmail.com> Message-ID: <39C29B43-C9D8-457D-9452-4A69FB5C8127@gmail.com> On Feb 16, 2012, at 4:10 PM, Nikolay Denev wrote: > > On Feb 16, 2012, at 3:51 PM, Peter Manev wrote: > >> >> >> On Thu, Feb 16, 2012 at 2:40 PM, Nikolay Denev wrote: >> >> On Feb 15, 2012, at 5:03 PM, Peter Manev wrote: >> >> > >> > Ok, >> > Just a couple of suggestions: >> > 1. Make the MTU on the suricata box equal the MTU on the switch port where it is connected to. >> >> I don't think this is an issue, as all of the other ports are not jumbo frames enabled, and I don't have frames bigger than 1522 bytes. >> >> >> Ok, I thought you have frames bigger than that... >> > 2. The interface that Suricata listens on (ex. eth0) , does it have all the VLANs untagged there? Or some are tagged and some untagged? Because if not - that might a problem. >> > >> >> I was wrong, there are some untagged packets, but they are mirrored LACP and LLDP frames. >> All the IP traffic from the mirrored ports is QinQ tagged, with the outer tag with VLAN 0 and the inner with the actual vlan being mirrored (all the ports that I mirror have only VLAN tagged traffic) >> Suricata seems to handle this OK, probably ignores the vlan tag? >> >> >> >> In that case the only discrepancy I see is the reported length of the packet in Suricata and Snorby - to me it looks like Suri reports it correctly (debug log), but Snorby does not. I can not be sure because from what I saw as a screen shot of Snorby , the packet has the same SEQ and ACK number - kind of strange, and the SEQ or ACK number there does not much the one in debug log.... can you please confirm that? >> > > Yep, this is what I'm seeing too. I have to probably see what exactly is in the unified2 log. Cause after suricata logs it, then it is touched by barnyard2 and then imported into the snorby mysql db, then snorby displays the alert, so there are other places that can break things. > > Speaking of this, does anybody know a tool to display unified2 logs? I've found a perl module but it seems to not properly display the packet dump. > >> > and if you could check that these two have any different effect ? ... >> > >> > thanks >> > >> > -- >> > Regards, >> > Peter Manev >> > >> >> >> >> >> -- >> Regards, >> Peter Manev >> > Here are a few more alerts without payload : +================ TIME: 02/20/2012-16:04:34.004598 SRC IP: X.X.X.X DST IP: Y.Y.Y.Y PROTO: 6 SRC PORT: 56946 DST PORT: 80 TCP SEQ: 3549381042 TCP ACK: 3077078161 FLOW: to_server: TRUE, to_client: FALSE FLOW Start TS: 02/20/2012-15:03:00.216065 FLOW IPONLY SET: TOSERVER: TRUE, TOCLIENT: TRUE FLOW ACTION: DROP: FALSE, PASS FALSE FLOW NOINSPECTION: PACKET: FALSE, PAYLOAD: FALSE, APP_LAYER: FALSE FLOW APP_LAYER: DETECTED: TRUE, PROTO 1 PACKET LEN: 0 PACKET: ALERT CNT: 1 ALERT MSG [00]: ET CURRENT_EVENTS HTTP Request to a *.co.cc domain ALERT GID [00]: 1 ALERT SID [00]: 2011374 ALERT REV [00]: 5 ALERT CLASS [00]: Potentially Bad Traffic ALERT PRIO [00]: 2 ALERT FOUND IN [00]: OTHER +================ Here's the rule that triggered them : alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET CURRENT_EVENTS HTTP Request to a *.co.cc domain"; flow: to_server,established; content:".co.cc|0D 0A|"; http_header; classtype:bad-unknown; sid:2011374; rev:5;) Snorby is not even able to parse the SRC IP and DST IP from the alert. I've also valid alerts from this rule, with packet and stream dump. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120220/f4b66b29/attachment.html From decanio.tom at gmail.com Mon Feb 20 14:24:11 2012 From: decanio.tom at gmail.com (Tom DeCanio) Date: Mon, 20 Feb 2012 11:24:11 -0800 Subject: [Oisf-users] New MPM available In-Reply-To: References: Message-ID: <1329765851.6968.17.camel@deceng-i7> How were you getting the byte counts? I put back a bit of code to dump state counts an no more. ##############Delta Table (state count 970)############## ##############Delta Table (state count 540)############## ##############Delta Table (state count 1908)############## ##############Delta Table (state count 1908)############## ##############Delta Table (state count 302)############## ##############Delta Table (state count 15263)############## ##############Delta Table (state count 9)############## ##############Delta Table (state count 686)############## ##############Delta Table (state count 6002)############## ##############Delta Table (state count 469)############## ##############Delta Table (state count 45)############## ##############Delta Table (state count 17218)############## ##############Delta Table (state count 7285)############## Some testing indicates that "ac-bs" isn't as fast as the old "ac" on tie Tilera for the ruleset I've been using. Tom On Mon, 2012-02-20 at 10:57 +0530, Anoop Saldanha wrote: > as a reference, these are the table sizes on my box with "ac-bs" for > all the mpm contexts used by the engine, for a 18k ruleset > > * in bytes > > "ac-bs" > 24348 > 38486 > 118900 > 47736 > 4716 > 4648804 > 558 > 15874 > 266202 > 6838 > 696 > 692 > 3982784 > 10756976 > > On Mon, Feb 20, 2012 at 4:19 AM, Tom DeCanio wrote: > > I just brought this up on the Tilera (tilegx). Haven't benchmarked it yet, > > but the tables do look much smaller than those produced by ac. Seems like > > this should improve performance here. When I get my benchmarking setup back > > I'll gather some new numbers. > > > > Tom > > > > On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha > > wrote: > >> > >> Hello all, > >> > >> We have a new MPM available in our codebase - "ac-bs". This provides > >> compression that's pretty close to ac-gfbs, while performing better > >> than ac-gfbs. > >> > >> To use this mpm, set > >> > >> "mpm-algo: ac-bs" in the conf file. > >> > >> Would appreciate performance numbers with both > >> > >> "sgh-mpm-context:full" > >> and > >> "sgh-mpm-context:single" > >> > >> To give an explanation on what "sgh-mpm-context" and the params "full" > >> and "single" mean, these refer to how we set up mpm contexts. > >> "single" indicates that we use a single context for all the patterns > >> in the engine. "full" indicates that we split the patterns into many > >> mpm contexts, one mpm context per signature group head(sgh). > >> > >> To use "full" with a sufficiently decent ruleset(say > 10k rules with > >> a decent no of patterns) would require a lot of memory, running into a > >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case > >> of "ac". "single" solves this with a single context and hence the > >> smaller memory footprint for the engine. > >> > >> If the machine has sufficient memory, "full" is suggested as it > >> provides much better performance than "single", albeit at the cost of > >> increased memory consumption. More of a available_memory vs > >> performance scenario. > >> > >> Looking forward to some performance/memory feedback/benchmarks with > >> this mpm from the community. > >> > >> *mpm - multi pattern matcher > >> *sgh - signature group head > >> > >> -- > >> Anoop Saldanha > >> > >> _______________________________________________ > >> Oisf-users mailing list > >> Oisf-users at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > > From anoopsaldanha at gmail.com Tue Feb 21 02:31:40 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Tue, 21 Feb 2012 13:01:40 +0530 Subject: [Oisf-users] New MPM available In-Reply-To: <1329765851.6968.17.camel@deceng-i7> References: <1329765851.6968.17.camel@deceng-i7> Message-ID: On Tue, Feb 21, 2012 at 12:54 AM, Tom DeCanio wrote: > How were you getting the byte counts? ?I put back a bit of code to dump > state counts an no more. > diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c index 9e08a23..ef9f139 100644 --- a/src/util-mpm-ac-bs.c +++ b/src/util-mpm-ac-bs.c @@ -972,6 +972,7 @@ static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx) exit(EXIT_FAILURE); } memset(ctx->state_table_mod, 0, size); + printf("size: %d\n", size); mpm_ctx->memory_cnt++; mpm_ctx->memory_size += size; > ##############Delta Table (state count 970)############## > ##############Delta Table (state count 540)############## > ##############Delta Table (state count 1908)############## > ##############Delta Table (state count 1908)############## > ##############Delta Table (state count 302)############## > ##############Delta Table (state count 15263)############## > ##############Delta Table (state count 9)############## > ##############Delta Table (state count 686)############## > ##############Delta Table (state count 6002)############## > ##############Delta Table (state count 469)############## > ##############Delta Table (state count 45)############## > ##############Delta Table (state count 17218)############## > ##############Delta Table (state count 7285)############## > > Some testing indicates that "ac-bs" isn't as fast as the old "ac" on tie > Tilera for the ruleset I've been using. > What's the perf difference? Probably with some optimizations like the cache prefetching for next buffer byte you had done earlier? > Tom > > On Mon, 2012-02-20 at 10:57 +0530, Anoop Saldanha wrote: >> as a reference, these are the table sizes on my box with "ac-bs" for >> all the mpm contexts used by the engine, for a 18k ruleset >> >> * in bytes >> >> "ac-bs" >> 24348 >> 38486 >> 118900 >> 47736 >> 4716 >> 4648804 >> 558 >> 15874 >> 266202 >> 6838 >> 696 >> 692 >> 3982784 >> 10756976 >> >> On Mon, Feb 20, 2012 at 4:19 AM, Tom DeCanio wrote: >> > I just brought this up on the Tilera (tilegx). ?Haven't benchmarked it yet, >> > but the tables do look much smaller than those produced by ac. ?Seems like >> > this should improve performance here. ?When I get my benchmarking setup back >> > I'll gather some new numbers. >> > >> > Tom >> > >> > On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha >> > wrote: >> >> >> >> Hello all, >> >> >> >> We have a new MPM available in our codebase - "ac-bs". ?This provides >> >> compression that's pretty close to ac-gfbs, while performing better >> >> than ac-gfbs. >> >> >> >> To use this mpm, set >> >> >> >> "mpm-algo: ac-bs" in the conf file. >> >> >> >> Would appreciate performance numbers with both >> >> >> >> "sgh-mpm-context:full" >> >> and >> >> "sgh-mpm-context:single" >> >> >> >> To give an explanation on what "sgh-mpm-context" and the params "full" >> >> and "single" mean, these refer to how we set up mpm contexts. >> >> "single" indicates that we use a single context for all the patterns >> >> in the engine. ?"full" indicates that we split the patterns into many >> >> mpm contexts, one mpm context per signature group head(sgh). >> >> >> >> To use "full" with a sufficiently decent ruleset(say > 10k rules with >> >> a decent no of patterns) would require a lot of memory, running into a >> >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case >> >> of "ac". ?"single" solves this with a single context and hence the >> >> smaller memory footprint for the engine. >> >> >> >> If the machine has sufficient memory, "full" is suggested as it >> >> provides much better performance than "single", albeit at the cost of >> >> increased memory consumption. ?More of a available_memory vs >> >> performance scenario. >> >> >> >> Looking forward to some performance/memory feedback/benchmarks with >> >> this mpm from the community. >> >> >> >> *mpm - multi pattern matcher >> >> *sgh - signature group head >> >> >> >> -- >> >> Anoop Saldanha >> >> >> >> _______________________________________________ >> >> Oisf-users mailing list >> >> Oisf-users at openinfosecfoundation.org >> >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> > >> > >> >> >> > > -- Anoop Saldanha From tingw.liu at gmail.com Wed Feb 22 04:57:30 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Wed, 22 Feb 2012 17:57:30 +0800 Subject: [Oisf-users] I want to get a valid library for suricate! Message-ID: I want to get a valid newlest library for suricate. Who can tell me how to do that? Thanks ! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120222/3ff545f1/attachment.html From victor at inliniac.net Wed Feb 22 04:59:11 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 22 Feb 2012 10:59:11 +0100 Subject: [Oisf-users] I want to get a valid library for suricate! In-Reply-To: References: Message-ID: <4F44BC6F.7000602@inliniac.net> On 02/22/2012 10:57 AM, tingwei liu wrote: > I want to get a valid newlest library for suricate. Who can tell me how > to do that? What library do you mean? -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From tingw.liu at gmail.com Wed Feb 22 05:01:36 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Wed, 22 Feb 2012 18:01:36 +0800 Subject: [Oisf-users] I want to get a valid library for suricate! In-Reply-To: <4F44BC6F.7000602@inliniac.net> References: <4F44BC6F.7000602@inliniac.net> Message-ID: On Wed, Feb 22, 2012 at 5:59 PM, Victor Julien wrote: > On 02/22/2012 10:57 AM, tingwei liu wrote: > > I want to get a valid newlest library for suricate. Who can tell me how > > to do that? > > What library do you mean? > > I mean rules for suricate. Thanks for your answer! > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120222/ae351d54/attachment.html From victor at inliniac.net Wed Feb 22 05:06:18 2012 From: victor at inliniac.net (Victor Julien) Date: Wed, 22 Feb 2012 11:06:18 +0100 Subject: [Oisf-users] I want to get a valid library for suricate! In-Reply-To: References: <4F44BC6F.7000602@inliniac.net> Message-ID: <4F44BE1A.5010801@inliniac.net> On 02/22/2012 11:01 AM, tingwei liu wrote: > > > On Wed, Feb 22, 2012 at 5:59 PM, Victor Julien > wrote: > > On 02/22/2012 10:57 AM, tingwei liu wrote: > > I want to get a valid newlest library for suricate. Who can tell > me how > > to do that? > > What library do you mean? > > I mean rules for suricate. > Thanks for your answer! I recommend using a update manager like Oinkmaster or Pulledpork. A guide for using Oinkmaster can be found here: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Rule_Management_with_Oinkmaster Hope this helps, Victor > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From tingw.liu at gmail.com Wed Feb 22 05:11:12 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Wed, 22 Feb 2012 18:11:12 +0800 Subject: [Oisf-users] I want to get a valid library for suricate! In-Reply-To: <4F44BE1A.5010801@inliniac.net> References: <4F44BC6F.7000602@inliniac.net> <4F44BE1A.5010801@inliniac.net> Message-ID: On Wed, Feb 22, 2012 at 6:06 PM, Victor Julien wrote: > On 02/22/2012 11:01 AM, tingwei liu wrote: > > > > > > On Wed, Feb 22, 2012 at 5:59 PM, Victor Julien > > wrote: > > > > On 02/22/2012 10:57 AM, tingwei liu wrote: > > > I want to get a valid newlest library for suricate. Who can tell > > me how > > > to do that? > > > > What library do you mean? > > > > I mean rules for suricate. > > Thanks for your answer! > > I recommend using a update manager like Oinkmaster or Pulledpork. A > guide for using Oinkmaster can be found here: > > > https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Rule_Management_with_Oinkmaster Just like this! Thank you very much! > > > Hope this helps, > Victor > > > > -- > > --------------------------------------------- > > Victor Julien > > http://www.inliniac.net/ > > PGP: http://www.inliniac.net/victorjulien.asc > > --------------------------------------------- > > > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120222/1b07a149/attachment.html From mc8647 at mclink.it Wed Feb 22 18:38:39 2012 From: mc8647 at mclink.it (mc8647) Date: Thu, 23 Feb 2012 00:38:39 +0100 Subject: [Oisf-users] where are my missing packets ? Message-ID: <4F457C7F.8040401@mclink.it> Hi to everybody, first time poster. I must admit that I've been following suricata for only one week. I'm reading mailing lists, testing, reading blogs and again testing... modifying configuration, modifying rules, and again reading and testing... Now I need to ask the experts for some help. My server is a double cpu with 6 cores each = 12 core (24 if I enable hyperthreading), 12 GB ram, using last ubuntu with kernel 3.0 64 bit. I can double (or triplicate) the ram if needed. I compiled suricata from the 1.2.1 tar source file. I don't have PF_RING enabled since the broadcom PF_RING aware drivers doesn't easily compile under 3.0 kernel (I was not able to compile them...) I want to setup as a IDS and activate some rules to check for malicious traffic in order to locate malware infected workstations. I have a mirror port that gives me about 200/400 mbit of lan traffic on a 1 gbit port of the server. No special setting was done on linux network kernel parameters. From day 1 I noticed that we are losing packets. If I stop suricata with ctrl-c I get a message stating about 25% packets missed. I changed several parameters, the first was max-pending-packets I set to 500 then to 5000 and now to 50000. I also raised memory available for various buffers. I also tried some threading settings. This evening I read that the packets missed percentage printed at ctrl-c are from the pcap library, but if I run tcpdump together with suricata I see the packets in tcpdump output but they don't show up in suricata http.log... When suricata starts http.log logs entries really fast but then it gets slower and slower probably due to the missed packets. top reports a range of 9-25% for each cpu, with a total of about 230% on the process. I'd like to ask you a lot of questions but I know it is not possible in a single message :-) So just to start I'd like to know which metrics should I monitor in stat.log, in top (swap ? process size ?) in order to understand where these packets miss the road... would it be good to test the setup using replicable traffic, like a pcap file ? Try some other optimization method with 3.0 kernel (cpu affinity, memory, af_packet, network kernel parameters....) or is it better to wipe the disks, install a 2.6.39 kernel and install PF_RING ? Or is the hardware not powerfull enough anyway ? Thanks Francesco From c.d.wakelin at reading.ac.uk Wed Feb 22 18:52:37 2012 From: c.d.wakelin at reading.ac.uk (Chris Wakelin) Date: Wed, 22 Feb 2012 23:52:37 +0000 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F457C7F.8040401@mclink.it> References: <4F457C7F.8040401@mclink.it> Message-ID: <4F457FC5.4010906@reading.ac.uk> Your machine sounds powerful enough. What size ruleset are you using? Which runmode? I'd certainly look at using autofp for pcap (not that I use it - I'm using PF_RING). You might get some benefit in PF_RING even with non-PF_RING ethernet drivers. The PF_RING developers haven't got any Broadcom cards to test alas, so they've not been maintaining the PF_RING drivers. Which chipset do you have? I've actually got some "bnx2" cards in my boxes, but I'm not using them for capture. Another option might be to get an Intel card for capturing; the e1000e 1GB ones are very cheap. I'm running roughly 4K rules on a machine with two quad-core processors, 16GB RAM, e1000e card and PF_RING with runmode=workers. One of the Broadcom interfaces is used for management so I don't cut the branch I'm sitting on when I use PF_RING in non-transparent mode! Best Wishes, Chris On 22/02/12 23:38, mc8647 wrote: > Hi to everybody, first time poster. > > I must admit that I've been following suricata for only one week. I'm > reading mailing lists, testing, reading blogs and again testing... > modifying configuration, modifying rules, and again reading and testing... > > Now I need to ask the experts for some help. > > My server is a double cpu with 6 cores each = 12 core (24 if I enable > hyperthreading), 12 GB ram, using last ubuntu with kernel 3.0 64 bit. I > can double (or triplicate) the ram if needed. I compiled suricata from > the 1.2.1 tar source file. > I don't have PF_RING enabled since the broadcom PF_RING aware drivers > doesn't easily compile under 3.0 kernel (I was not able to compile them...) > > I want to setup as a IDS and activate some rules to check for malicious > traffic in order to locate malware infected workstations. > > I have a mirror port that gives me about 200/400 mbit of lan traffic on > a 1 gbit port of the server. No special setting was done on linux > network kernel parameters. > > From day 1 I noticed that we are losing packets. If I stop suricata > with ctrl-c I get a message stating about 25% packets missed. I changed > several parameters, the first was max-pending-packets I set to 500 then > to 5000 and now to 50000. I also raised memory available for various > buffers. I also tried some threading settings. > This evening I read that the packets missed percentage printed at ctrl-c > are from the pcap library, but if I run tcpdump together with suricata I > see the packets in tcpdump output but they don't show up in suricata > http.log... > When suricata starts http.log logs entries really fast but then it gets > slower and slower probably due to the missed packets. > > top reports a range of 9-25% for each cpu, with a total of about 230% on > the process. > > > I'd like to ask you a lot of questions but I know it is not possible in > a single message :-) > > So just to start I'd like to know which metrics should I monitor in > stat.log, in top (swap ? process size ?) in order to understand where > these packets miss the road... would it be good to test the setup using > replicable traffic, like a pcap file ? > > Try some other optimization method with 3.0 kernel (cpu affinity, > memory, af_packet, network kernel parameters....) or is it better to > wipe the disks, install a 2.6.39 kernel and install PF_RING ? Or is the > hardware not powerfull enough anyway ? > > Thanks > Francesco > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin at reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094 From mc8647 at mclink.it Wed Feb 22 19:15:15 2012 From: mc8647 at mclink.it (mc8647) Date: Thu, 23 Feb 2012 01:15:15 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F457FC5.4010906@reading.ac.uk> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> Message-ID: <4F458513.1040409@mclink.it> Thanks for reply. The server is a HP DL360G7, it has 4 onboard lan ports... We are testing a proprietary IDS with another mirror port on a twin server (they are identically configured hardware). This proprietary IDS runs inside a esx4 VM with 8 cpu and it has no missing packets! So with less CPUs, less ram, and with esx overhead it is able to not lose packets. I think it is linux based with highly personlized setup, for example it supports just 3 hardware servers and esx VMs. "If I stop suricata with ctrl-c I get a message stating about 25% packets missed." should have been If I stop suricata with ctrl-c I get a message stating that from 3 to about 25% packets were missed depending on the run. Francesco From r.fulton at auckland.ac.nz Wed Feb 22 20:02:15 2012 From: r.fulton at auckland.ac.nz (Russell Fulton) Date: Thu, 23 Feb 2012 14:02:15 +1300 Subject: [Oisf-users] looking for debian packages... Message-ID: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> Hi I am running debian squeeze on my sensors. The suricata package for squeeze is at 1.0.1. I see sid has 1.2.1? Does anyone have an up to date .deb that I can use on 6.0.4 (squeeze)? Anyone running the sid package under squeeze? Russell From thantry at gmail.com Wed Feb 22 20:07:28 2012 From: thantry at gmail.com (Hariharan Thantry) Date: Wed, 22 Feb 2012 17:07:28 -0800 Subject: [Oisf-users] New MPM available In-Reply-To: References: <1329765851.6968.17.camel@deceng-i7> Message-ID: Hi folks, Just curious. What exactly are the b2g, b3g Multi-pattern-match algorithms? I know Wu-Manber, Knuth-Morris-Pratt & Aho-Corasick, but couldn't figure what algorithm b2g (or b3g) implemented... Thanks, Hari On Mon, Feb 20, 2012 at 11:31 PM, Anoop Saldanha wrote: > On Tue, Feb 21, 2012 at 12:54 AM, Tom DeCanio wrote: >> How were you getting the byte counts? ?I put back a bit of code to dump >> state counts an no more. >> > > diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c > index 9e08a23..ef9f139 100644 > --- a/src/util-mpm-ac-bs.c > +++ b/src/util-mpm-ac-bs.c > @@ -972,6 +972,7 @@ static inline void > SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx) > ? ? ? ? ? ? exit(EXIT_FAILURE); > ? ? ? ? } > ? ? ? ? memset(ctx->state_table_mod, 0, size); > + ? ? ? ?printf("size: %d\n", size); > > ? ? ? ? mpm_ctx->memory_cnt++; > ? ? ? ? mpm_ctx->memory_size += size; > > >> ##############Delta Table (state count 970)############## >> ##############Delta Table (state count 540)############## >> ##############Delta Table (state count 1908)############## >> ##############Delta Table (state count 1908)############## >> ##############Delta Table (state count 302)############## >> ##############Delta Table (state count 15263)############## >> ##############Delta Table (state count 9)############## >> ##############Delta Table (state count 686)############## >> ##############Delta Table (state count 6002)############## >> ##############Delta Table (state count 469)############## >> ##############Delta Table (state count 45)############## >> ##############Delta Table (state count 17218)############## >> ##############Delta Table (state count 7285)############## >> >> Some testing indicates that "ac-bs" isn't as fast as the old "ac" on tie >> Tilera for the ruleset I've been using. >> > > What's the perf difference? > > Probably with some optimizations like the cache prefetching for next > buffer byte you had done earlier? > >> Tom >> >> On Mon, 2012-02-20 at 10:57 +0530, Anoop Saldanha wrote: >>> as a reference, these are the table sizes on my box with "ac-bs" for >>> all the mpm contexts used by the engine, for a 18k ruleset >>> >>> * in bytes >>> >>> "ac-bs" >>> 24348 >>> 38486 >>> 118900 >>> 47736 >>> 4716 >>> 4648804 >>> 558 >>> 15874 >>> 266202 >>> 6838 >>> 696 >>> 692 >>> 3982784 >>> 10756976 >>> >>> On Mon, Feb 20, 2012 at 4:19 AM, Tom DeCanio wrote: >>> > I just brought this up on the Tilera (tilegx). ?Haven't benchmarked it yet, >>> > but the tables do look much smaller than those produced by ac. ?Seems like >>> > this should improve performance here. ?When I get my benchmarking setup back >>> > I'll gather some new numbers. >>> > >>> > Tom >>> > >>> > On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha >>> > wrote: >>> >> >>> >> Hello all, >>> >> >>> >> We have a new MPM available in our codebase - "ac-bs". ?This provides >>> >> compression that's pretty close to ac-gfbs, while performing better >>> >> than ac-gfbs. >>> >> >>> >> To use this mpm, set >>> >> >>> >> "mpm-algo: ac-bs" in the conf file. >>> >> >>> >> Would appreciate performance numbers with both >>> >> >>> >> "sgh-mpm-context:full" >>> >> and >>> >> "sgh-mpm-context:single" >>> >> >>> >> To give an explanation on what "sgh-mpm-context" and the params "full" >>> >> and "single" mean, these refer to how we set up mpm contexts. >>> >> "single" indicates that we use a single context for all the patterns >>> >> in the engine. ?"full" indicates that we split the patterns into many >>> >> mpm contexts, one mpm context per signature group head(sgh). >>> >> >>> >> To use "full" with a sufficiently decent ruleset(say > 10k rules with >>> >> a decent no of patterns) would require a lot of memory, running into a >>> >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case >>> >> of "ac". ?"single" solves this with a single context and hence the >>> >> smaller memory footprint for the engine. >>> >> >>> >> If the machine has sufficient memory, "full" is suggested as it >>> >> provides much better performance than "single", albeit at the cost of >>> >> increased memory consumption. ?More of a available_memory vs >>> >> performance scenario. >>> >> >>> >> Looking forward to some performance/memory feedback/benchmarks with >>> >> this mpm from the community. >>> >> >>> >> *mpm - multi pattern matcher >>> >> *sgh - signature group head >>> >> >>> >> -- >>> >> Anoop Saldanha >>> >> >>> >> _______________________________________________ >>> >> Oisf-users mailing list >>> >> Oisf-users at openinfosecfoundation.org >>> >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> > >>> > >>> >>> >>> >> >> > > > > -- > Anoop Saldanha > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From tingw.liu at gmail.com Wed Feb 22 20:33:40 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Thu, 23 Feb 2012 09:33:40 +0800 Subject: [Oisf-users] Make the log to sql. Message-ID: How can I make the log to database? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/ccfe9d74/attachment.html From r.fulton at auckland.ac.nz Wed Feb 22 20:48:30 2012 From: r.fulton at auckland.ac.nz (Russell Fulton) Date: Thu, 23 Feb 2012 14:48:30 +1300 Subject: [Oisf-users] Make the log to sql. In-Reply-To: References: Message-ID: <57B8E00F-3B49-4701-B7A5-28EF5EF07837@auckland.ac.nz> one way is to log out to unified2 and use barnyard2 with a database plugin. Russell On 23/02/2012, at 2:33 PM, tingwei liu wrote: > How can I make the log to database? > > Thanks! _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From mcholste at gmail.com Wed Feb 22 21:16:11 2012 From: mcholste at gmail.com (Martin Holste) Date: Wed, 22 Feb 2012 20:16:11 -0600 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F458513.1040409@mclink.it> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> Message-ID: The biggest performance boost you can get is to run with the pattern matcher as "ac" and all of the settings on "high" in the tuning. This will use a lot of RAM--you may not have enough to run all of the rules you want. I highly suggest adding as much RAM as possible, running ac with autofp, and use PF_RING with or without the proper Broadcom driver. In the stats file, look at the tcp.segment_memcap_drop and tcp.ssn_memcap_drop. If you see drops there, you need to up the buffers even more for memcap, etc. Regarding comparison to another IDS: Suricata may be doing a lot more work than the other setup. Remember that it is actually deconstructing every HTTP session before it even gets to the pattern matching. This is powerful stuff, and it costs CPU time. Also, keep in mind the number of rules being run when making comparisons. One good baseline for a sanity check is to disable all of the rules and run Suricata for a bit. Make sure that it isn't dropping packets just doing stream reassembly and HTTP analysis. Once you've verified it's not dropping there, then you know that tweaking the number of rules and/or the pattern matching settings will provide a benefit. That server should definitely be able to handle 400 Mb/sec, one way or another. On Wed, Feb 22, 2012 at 6:15 PM, mc8647 wrote: > Thanks for reply. > > The server is a HP DL360G7, it has 4 onboard lan ports... > > We are testing a proprietary IDS with another mirror port on a twin > server (they are identically configured hardware). > > This proprietary IDS runs inside a esx4 VM with 8 cpu and it has no > missing packets! > > So with less CPUs, less ram, and with esx overhead it is able to not > lose packets. I think it is linux based with highly personlized setup, > for example it supports just 3 hardware servers and esx VMs. > > > "If I stop suricata with ctrl-c I get a message stating about 25% > packets missed." should have been > > If I stop suricata with ctrl-c I get a message stating that from 3 to about 25% packets were missed depending on the run. > > Francesco > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From tingw.liu at gmail.com Wed Feb 22 21:50:59 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Thu, 23 Feb 2012 10:50:59 +0800 Subject: [Oisf-users] Make the log to sql. In-Reply-To: <57B8E00F-3B49-4701-B7A5-28EF5EF07837@auckland.ac.nz> References: <57B8E00F-3B49-4701-B7A5-28EF5EF07837@auckland.ac.nz> Message-ID: On Thu, Feb 23, 2012 at 9:48 AM, Russell Fulton wrote: > one way is to log out to unified2 and use barnyard2 with a database plugin. > > I will try. Thanks! > Russell > > On 23/02/2012, at 2:33 PM, tingwei liu wrote: > > > How can I make the log to database? > > > > Thanks! _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/9c44a648/attachment.html From anoopsaldanha at gmail.com Wed Feb 22 22:36:11 2012 From: anoopsaldanha at gmail.com (Anoop Saldanha) Date: Thu, 23 Feb 2012 09:06:11 +0530 Subject: [Oisf-users] New MPM available In-Reply-To: References: <1329765851.6968.17.camel@deceng-i7> Message-ID: On Thu, Feb 23, 2012 at 6:37 AM, Hariharan Thantry wrote: > Hi folks, > > Just curious. What exactly are the b2g, b3g Multi-pattern-match > algorithms? I know Wu-Manber, Knuth-Morris-Pratt & Aho-Corasick, but > couldn't figure what algorithm b2g (or b3g) implemented... > q-gram versions of bndmq > Thanks, > Hari > > On Mon, Feb 20, 2012 at 11:31 PM, Anoop Saldanha > wrote: >> On Tue, Feb 21, 2012 at 12:54 AM, Tom DeCanio wrote: >>> How were you getting the byte counts? ?I put back a bit of code to dump >>> state counts an no more. >>> >> >> diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c >> index 9e08a23..ef9f139 100644 >> --- a/src/util-mpm-ac-bs.c >> +++ b/src/util-mpm-ac-bs.c >> @@ -972,6 +972,7 @@ static inline void >> SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx) >> ? ? ? ? ? ? exit(EXIT_FAILURE); >> ? ? ? ? } >> ? ? ? ? memset(ctx->state_table_mod, 0, size); >> + ? ? ? ?printf("size: %d\n", size); >> >> ? ? ? ? mpm_ctx->memory_cnt++; >> ? ? ? ? mpm_ctx->memory_size += size; >> >> >>> ##############Delta Table (state count 970)############## >>> ##############Delta Table (state count 540)############## >>> ##############Delta Table (state count 1908)############## >>> ##############Delta Table (state count 1908)############## >>> ##############Delta Table (state count 302)############## >>> ##############Delta Table (state count 15263)############## >>> ##############Delta Table (state count 9)############## >>> ##############Delta Table (state count 686)############## >>> ##############Delta Table (state count 6002)############## >>> ##############Delta Table (state count 469)############## >>> ##############Delta Table (state count 45)############## >>> ##############Delta Table (state count 17218)############## >>> ##############Delta Table (state count 7285)############## >>> >>> Some testing indicates that "ac-bs" isn't as fast as the old "ac" on tie >>> Tilera for the ruleset I've been using. >>> >> >> What's the perf difference? >> >> Probably with some optimizations like the cache prefetching for next >> buffer byte you had done earlier? >> >>> Tom >>> >>> On Mon, 2012-02-20 at 10:57 +0530, Anoop Saldanha wrote: >>>> as a reference, these are the table sizes on my box with "ac-bs" for >>>> all the mpm contexts used by the engine, for a 18k ruleset >>>> >>>> * in bytes >>>> >>>> "ac-bs" >>>> 24348 >>>> 38486 >>>> 118900 >>>> 47736 >>>> 4716 >>>> 4648804 >>>> 558 >>>> 15874 >>>> 266202 >>>> 6838 >>>> 696 >>>> 692 >>>> 3982784 >>>> 10756976 >>>> >>>> On Mon, Feb 20, 2012 at 4:19 AM, Tom DeCanio wrote: >>>> > I just brought this up on the Tilera (tilegx). ?Haven't benchmarked it yet, >>>> > but the tables do look much smaller than those produced by ac. ?Seems like >>>> > this should improve performance here. ?When I get my benchmarking setup back >>>> > I'll gather some new numbers. >>>> > >>>> > Tom >>>> > >>>> > On Tue, Feb 14, 2012 at 1:22 AM, Anoop Saldanha >>>> > wrote: >>>> >> >>>> >> Hello all, >>>> >> >>>> >> We have a new MPM available in our codebase - "ac-bs". ?This provides >>>> >> compression that's pretty close to ac-gfbs, while performing better >>>> >> than ac-gfbs. >>>> >> >>>> >> To use this mpm, set >>>> >> >>>> >> "mpm-algo: ac-bs" in the conf file. >>>> >> >>>> >> Would appreciate performance numbers with both >>>> >> >>>> >> "sgh-mpm-context:full" >>>> >> and >>>> >> "sgh-mpm-context:single" >>>> >> >>>> >> To give an explanation on what "sgh-mpm-context" and the params "full" >>>> >> and "single" mean, these refer to how we set up mpm contexts. >>>> >> "single" indicates that we use a single context for all the patterns >>>> >> in the engine. ?"full" indicates that we split the patterns into many >>>> >> mpm contexts, one mpm context per signature group head(sgh). >>>> >> >>>> >> To use "full" with a sufficiently decent ruleset(say > 10k rules with >>>> >> a decent no of patterns) would require a lot of memory, running into a >>>> >> couple of gigs for ac-gfbs or ac-bs or b2gc, or tens of gigs in case >>>> >> of "ac". ?"single" solves this with a single context and hence the >>>> >> smaller memory footprint for the engine. >>>> >> >>>> >> If the machine has sufficient memory, "full" is suggested as it >>>> >> provides much better performance than "single", albeit at the cost of >>>> >> increased memory consumption. ?More of a available_memory vs >>>> >> performance scenario. >>>> >> >>>> >> Looking forward to some performance/memory feedback/benchmarks with >>>> >> this mpm from the community. >>>> >> >>>> >> *mpm - multi pattern matcher >>>> >> *sgh - signature group head >>>> >> >>>> >> -- >>>> >> Anoop Saldanha >>>> >> >>>> >> _______________________________________________ >>>> >> Oisf-users mailing list >>>> >> Oisf-users at openinfosecfoundation.org >>>> >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>> > >>>> > >>>> >>>> >>>> >>> >>> >> >> >> >> -- >> Anoop Saldanha >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users -- Anoop Saldanha From victor at inliniac.net Thu Feb 23 02:42:50 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 08:42:50 +0100 Subject: [Oisf-users] looking for debian packages... In-Reply-To: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> References: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> Message-ID: <4F45EDFA.6010409@inliniac.net> On 02/23/2012 02:02 AM, Russell Fulton wrote: > Hi > > I am running debian squeeze on my sensors. The suricata package for squeeze is at 1.0.1. I see sid has 1.2.1? > > Does anyone have an up to date .deb that I can use on 6.0.4 (squeeze)? > > Anyone running the sid package under squeeze? Debian testing has 1.2.1 as well. You might want to try pulling the testing/sid debian src package and rebuilding it on your squeeze box. I think you have a good chance it will work. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From tingw.liu at gmail.com Thu Feb 23 02:45:22 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Thu, 23 Feb 2012 15:45:22 +0800 Subject: [Oisf-users] suricate with pfring Message-ID: I have installed pfring-5.1.2. I have write test.c like this: int main() { return pfring_open(); } And "gcc test.c -lpfring" is work . But suricate configure with --enable-pfring return "library is not found" Fedora15_64 system. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/dc3c880a/attachment.html From victor at inliniac.net Thu Feb 23 02:48:57 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 08:48:57 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> Message-ID: <4F45EF69.3050405@inliniac.net> Enabling hyper threading is also recommended. It's not magic, but it will gain you some. Btw, can you share a record of the stats.log after Suricata has been running for some time? Cheers, Victor On 02/23/2012 03:16 AM, Martin Holste wrote: > The biggest performance boost you can get is to run with the pattern > matcher as "ac" and all of the settings on "high" in the tuning. This > will use a lot of RAM--you may not have enough to run all of the rules > you want. I highly suggest adding as much RAM as possible, running ac > with autofp, and use PF_RING with or without the proper Broadcom > driver. > > In the stats file, look at the tcp.segment_memcap_drop and > tcp.ssn_memcap_drop. If you see drops there, you need to up the > buffers even more for memcap, etc. > > Regarding comparison to another IDS: Suricata may be doing a lot more > work than the other setup. Remember that it is actually > deconstructing every HTTP session before it even gets to the pattern > matching. This is powerful stuff, and it costs CPU time. Also, keep > in mind the number of rules being run when making comparisons. > > One good baseline for a sanity check is to disable all of the rules > and run Suricata for a bit. Make sure that it isn't dropping packets > just doing stream reassembly and HTTP analysis. Once you've verified > it's not dropping there, then you know that tweaking the number of > rules and/or the pattern matching settings will provide a benefit. > That server should definitely be able to handle 400 Mb/sec, one way or > another. > > On Wed, Feb 22, 2012 at 6:15 PM, mc8647 wrote: >> Thanks for reply. >> >> The server is a HP DL360G7, it has 4 onboard lan ports... >> >> We are testing a proprietary IDS with another mirror port on a twin >> server (they are identically configured hardware). >> >> This proprietary IDS runs inside a esx4 VM with 8 cpu and it has no >> missing packets! >> >> So with less CPUs, less ram, and with esx overhead it is able to not >> lose packets. I think it is linux based with highly personlized setup, >> for example it supports just 3 hardware servers and esx VMs. >> >> >> "If I stop suricata with ctrl-c I get a message stating about 25% >> packets missed." should have been >> >> If I stop suricata with ctrl-c I get a message stating that from 3 to about 25% packets were missed depending on the run. >> >> Francesco >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From tingw.liu at gmail.com Thu Feb 23 03:10:59 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Thu, 23 Feb 2012 16:10:59 +0800 Subject: [Oisf-users] suricate with pfring In-Reply-To: References: Message-ID: magic.h is in /usr/include/linux/, but it can not find it. Why? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/01192438/attachment.html From victor at inliniac.net Thu Feb 23 03:13:08 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 09:13:08 +0100 Subject: [Oisf-users] suricate with pfring In-Reply-To: References: Message-ID: <4F45F514.3090705@inliniac.net> On 02/23/2012 09:10 AM, tingwei liu wrote: > magic.h is in /usr/include/linux/, but it can not find it. Why? You need the libmagic development files. Debian: libmagic-dev Fedora: file-devel -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From tingw.liu at gmail.com Thu Feb 23 03:18:23 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Thu, 23 Feb 2012 16:18:23 +0800 Subject: [Oisf-users] suricate with pfring In-Reply-To: <4F45F514.3090705@inliniac.net> References: <4F45F514.3090705@inliniac.net> Message-ID: On Thu, Feb 23, 2012 at 4:13 PM, Victor Julien wrote: > On 02/23/2012 09:10 AM, tingwei liu wrote: > > magic.h is in /usr/include/linux/, but it can not find it. Why? > > You need the libmagic development files. > > Debian: libmagic-dev > Fedora: file-devel I tryied yum -y install libmagic-dev on fedora 15,but it doesn't work. yum -y install file-devel is OK Thanks a lot . > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/023dc785/attachment.html From petermanev at gmail.com Thu Feb 23 03:20:19 2012 From: petermanev at gmail.com (Peter Manev) Date: Thu, 23 Feb 2012 09:20:19 +0100 Subject: [Oisf-users] looking for debian packages... In-Reply-To: <4F45EDFA.6010409@inliniac.net> References: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> <4F45EDFA.6010409@inliniac.net> Message-ID: On Thu, Feb 23, 2012 at 8:42 AM, Victor Julien wrote: > On 02/23/2012 02:02 AM, Russell Fulton wrote: > > Hi > > > > I am running debian squeeze on my sensors. The suricata package for > squeeze is at 1.0.1. I see sid has 1.2.1? > > > > Does anyone have an up to date .deb that I can use on 6.0.4 (squeeze)? > > > > Anyone running the sid package under squeeze? > > Debian testing has 1.2.1 as well. You might want to try pulling the > testing/sid debian src package and rebuilding it on your squeeze box. I > think you have a good chance it will work. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > If you need some help creating/generating the latest pkg (out of src), please let me know. thanks -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/a1ea0645/attachment.html From petermanev at gmail.com Thu Feb 23 03:29:58 2012 From: petermanev at gmail.com (Peter Manev) Date: Thu, 23 Feb 2012 09:29:58 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F45EF69.3050405@inliniac.net> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> Message-ID: On Thu, Feb 23, 2012 at 8:48 AM, Victor Julien wrote: > Enabling hyper threading is also recommended. It's not magic, but it > will gain you some. > > Btw, can you share a record of the stats.log after Suricata has been > running for some time? > > Cheers, > Victor > > On 02/23/2012 03:16 AM, Martin Holste wrote: > > The biggest performance boost you can get is to run with the pattern > > matcher as "ac" and all of the settings on "high" in the tuning. This > > will use a lot of RAM--you may not have enough to run all of the rules > > you want. I highly suggest adding as much RAM as possible, running ac > > with autofp, and use PF_RING with or without the proper Broadcom > > driver. > > > > In the stats file, look at the tcp.segment_memcap_drop and > > tcp.ssn_memcap_drop. If you see drops there, you need to up the > > buffers even more for memcap, etc. > > > > Regarding comparison to another IDS: Suricata may be doing a lot more > > work than the other setup. Remember that it is actually > > deconstructing every HTTP session before it even gets to the pattern > > matching. This is powerful stuff, and it costs CPU time. Also, keep > > in mind the number of rules being run when making comparisons. > > > > One good baseline for a sanity check is to disable all of the rules > > and run Suricata for a bit. Make sure that it isn't dropping packets > > just doing stream reassembly and HTTP analysis. Once you've verified > > it's not dropping there, then you know that tweaking the number of > > rules and/or the pattern matching settings will provide a benefit. > > That server should definitely be able to handle 400 Mb/sec, one way or > > another. > > > > On Wed, Feb 22, 2012 at 6:15 PM, mc8647 wrote: > >> Thanks for reply. > >> > >> The server is a HP DL360G7, it has 4 onboard lan ports... > >> > >> We are testing a proprietary IDS with another mirror port on a twin > >> server (they are identically configured hardware). > >> > >> This proprietary IDS runs inside a esx4 VM with 8 cpu and it has no > >> missing packets! > >> > >> So with less CPUs, less ram, and with esx overhead it is able to not > >> lose packets. I think it is linux based with highly personlized setup, > >> for example it supports just 3 hardware servers and esx VMs. > >> > >> > >> "If I stop suricata with ctrl-c I get a message stating about 25% > >> packets missed." should have been > >> > >> If I stop suricata with ctrl-c I get a message stating that from 3 to > about 25% packets were missed depending on the run. > >> > >> Francesco > >> > >> _______________________________________________ > >> Oisf-users mailing list > >> Oisf-users at openinfosecfoundation.org > >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > _______________________________________________ > > Oisf-users mailing list > > Oisf-users at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > > > > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > I agree with Martin - up the buffers. BTW - if you load Suricata 1.2.1 (on an empty interface, no traffic) - how much mem is taken for 4K rules? thanks -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/5bc3f04e/attachment-0001.html From mc8647 at mclink.it Thu Feb 23 03:41:50 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 09:41:50 +0100 Subject: [Oisf-users] Fwd: where are my missing packets ? Message-ID: An embedded message was scrubbed... From: "Travel Factory S.r.l." Subject: Re: [Oisf-users] where are my missing packets ? Date: Thu, 23 Feb 2012 09:33:26 +0100 Size: 5247 Url: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/f9579e7c/attachment.mht From mc8647 at mclink.it Thu Feb 23 03:42:08 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 09:42:08 +0100 Subject: [Oisf-users] Fwd: where are my missing packets ? Message-ID: An embedded message was scrubbed... From: "Travel Factory S.r.l." Subject: Re: [Oisf-users] where are my missing packets ? Date: Thu, 23 Feb 2012 09:39:08 +0100 Size: 1654 Url: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/4ccfe307/attachment.mht From mc8647 at mclink.it Thu Feb 23 03:44:17 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 09:44:17 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: Message-ID: With all settings set to "high" the top reported values are: 13308 root 20 0 1398m 328m 2708 S 1 2.7 0:02.27 suricata From victor at inliniac.net Thu Feb 23 03:44:42 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 09:44:42 +0100 Subject: [Oisf-users] Fwd: where are my missing packets ? In-Reply-To: References: Message-ID: <4F45FC7A.7060002@inliniac.net> On 02/23/2012 09:42 AM, Travel Factory S.r.l. wrote: > 23/2/2012 -- 09:34:48 - - 1945 signatures processed. 2 are > IP-only rules, 559 are inspecting packet payload, 1500 inspect > application layer, 0 are decoder event only What ruleset are you running btw? ET? VRT? If ET, are you running the suricata version? -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From victor at inliniac.net Thu Feb 23 03:47:49 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 09:47:49 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> Message-ID: <4F45FD35.5050108@inliniac.net> On 02/23/2012 09:33 AM, Travel Factory S.r.l. wrote: > > > I was collecting some infos.... suricata has been running all the night > long, and now, with a 130 mbit load a > tail -f stats.log | grep tcp.segment_memcap_drop > reports > > tcp.segment_memcap_drop | Decode & Stream | 32054746 > tcp.segment_memcap_drop | Decode & Stream | 32067757 > tcp.segment_memcap_drop | Decode & Stream | 32086127 > tcp.segment_memcap_drop | Decode & Stream | 32103102 > tcp.segment_memcap_drop | Decode & Stream | 32124890 > tcp.segment_memcap_drop | Decode & Stream | 32148578 > tcp.segment_memcap_drop | Decode & Stream | 32171766 > tcp.segment_memcap_drop | Decode & Stream | 32189165 > tcp.segment_memcap_drop | Decode & Stream | 32211397 > tcp.segment_memcap_drop | Decode & Stream | 32233739 > tcp.segment_memcap_drop | Decode & Stream | 32262092 > tcp.segment_memcap_drop | Decode & Stream | 32277511 > tcp.segment_memcap_drop | Decode & Stream | 32295917 > tcp.segment_memcap_drop | Decode & Stream | 32319345 > tcp.segment_memcap_drop | Decode & Stream | 32338257 > tcp.segment_memcap_drop | Decode & Stream | 32357508 To make these go away, increase your stream.reassembly.memcap value. I think you have it set to 512mb or so: tcp.reassembly_memuse | Decode & Stream | 536870870 You could try doubling it. > > A stat.log record log is the following: > ------------------------------------------------------------------- > Date: 2/23/2012 -- 09:31:32 (uptime: 0d, 16h 10m 01s) > ------------------------------------------------------------------- > Counter | TM Name | Value > ------------------------------------------------------------------- > detect.alert | Detect | 18 > decoder.pkts | Decode & Stream | 514484781 > decoder.bytes | Decode & Stream | 475566946964 > decoder.ipv4 | Decode & Stream | 513403747 > decoder.ipv6 | Decode & Stream | 1899 > decoder.ethernet | Decode & Stream | 514484781 > decoder.raw | Decode & Stream | 0 > decoder.sll | Decode & Stream | 0 > decoder.tcp | Decode & Stream | 196568162 > decoder.udp | Decode & Stream | 285486352 > decoder.sctp | Decode & Stream | 0 > decoder.icmpv4 | Decode & Stream | 596837 > decoder.icmpv6 | Decode & Stream | 209 > decoder.ppp | Decode & Stream | 0 > decoder.pppoe | Decode & Stream | 0 > decoder.gre | Decode & Stream | 0 > decoder.vlan | Decode & Stream | 0 > decoder.avg_pkt_size | Decode & Stream | 924.355714 > decoder.max_pkt_size | Decode & Stream | 1518 > defrag.ipv4.fragments | Decode & Stream | 1483782 > defrag.ipv4.reassembled | Decode & Stream | 34346 > defrag.ipv4.timeouts | Decode & Stream | 0 > defrag.ipv6.fragments | Decode & Stream | 0 > defrag.ipv6.reassembled | Decode & Stream | 0 > defrag.ipv6.timeouts | Decode & Stream | 0 > tcp.sessions | Decode & Stream | 2265299 > tcp.ssn_memcap_drop | Decode & Stream | 0 > tcp.pseudo | Decode & Stream | 176610 > tcp.invalid_checksum | Decode & Stream | 0 > tcp.no_flow | Decode & Stream | 0 > tcp.reused_ssn | Decode & Stream | 318 > tcp.memuse | Decode & Stream | 34023072.000000 > tcp.syn | Decode & Stream | 2429325 > tcp.synack | Decode & Stream | 2213982 > tcp.rst | Decode & Stream | 257041 > tcp.segment_memcap_drop | Decode & Stream | 32621673 > tcp.stream_depth_reached | Decode & Stream | 0 > tcp.reassembly_memuse | Decode & Stream | 536870870.000000 > tcp.reassembly_gap | Decode & Stream | 86271 > flow_mgr.closed_pruned | FlowManagerThread | 1983794 > flow_mgr.new_pruned | FlowManagerThread | 601775 > flow_mgr.est_pruned | FlowManagerThread | 843364 > flow.memuse | FlowManagerThread | 29185432.000000 > flow.emerg_mode_entered | FlowManagerThread | 0 > flow.emerg_mode_over | FlowManagerThread | 0 Nothing out of the ordinary here, although udp is high vs tcp. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From mc8647 at mclink.it Thu Feb 23 03:52:38 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 09:52:38 +0100 Subject: [Oisf-users] Fwd: where are my missing packets ? In-Reply-To: <4F45FC7A.7060002@inliniac.net> References: <4F45FC7A.7060002@inliniac.net> Message-ID: rule-files: - file-identify.rules - cdd.rules <-- these are 5 rules I wrote, to see if the traffic flows... - exploit.rules - policy.rules - smtp.rules - specific-threats.rules - web-client.rules - emerging-web_client.rules My goal is to replicate this test: http://www.aldeid.com/wiki/Suricata-vs-snort/Test-cases/Client-side-attacks but since I have a lot of missing packets... I'm now remming all rules except cdd.rules and try again. From petermanev at gmail.com Thu Feb 23 03:49:45 2012 From: petermanev at gmail.com (Peter Manev) Date: Thu, 23 Feb 2012 09:49:45 +0100 Subject: [Oisf-users] Fwd: where are my missing packets ? In-Reply-To: <4F45FC7A.7060002@inliniac.net> References: <4F45FC7A.7060002@inliniac.net> Message-ID: On Thu, Feb 23, 2012 at 9:44 AM, Victor Julien wrote: > On 02/23/2012 09:42 AM, Travel Factory S.r.l. wrote: > > 23/2/2012 -- 09:34:48 - - 1945 signatures processed. 2 are > > IP-only rules, 559 are inspecting packet payload, 1500 inspect > > application layer, 0 are decoder event only > > What ruleset are you running btw? ET? VRT? If ET, are you running the > suricata version? > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > Just as a reminder if you have check sum offloading enabled on your network card ...make sure you disable the checksum - no in yaml....to be on the safe side..... -- Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/9f6509c5/attachment-0001.html From mc8647 at mclink.it Thu Feb 23 04:45:23 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 10:45:23 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F45FD35.5050108@inliniac.net> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> Message-ID: > To make these go away, increase your stream.reassembly.memcap value. >I > think you have it set to 512mb or so: > > tcp.reassembly_memuse | Decode & Stream | 536870870 > > You could try doubling it. I chenged the values, made some tests that failed then I pasted here the values so that you could check... I then realized that I - I don't remember why - set inline: yes stream: memcap: 640mb checksum_validation: no # reject wrong csums inline: yes # no inline mode reassembly: memcap: 2048mb depth: 50mb # reassemble 1mb into a stream toserver_chunk_size: 2560 toclient_chunk_size: 2560 I then set inline: no and I now have tcp.segment_memcap_drop | Detect | 0 with inline: yes I had this in stats.log after about 1:30: tcp.segment_memcap_drop | Detect | 0 tcp.reassembly_memuse | Detect | 38468978961.000000 --- tcp.segment_memcap_drop | Detect | 17583 tcp.reassembly_memuse | Detect | 38654704962.000000 --- tcp.segment_memcap_drop | Detect | 29346 tcp.reassembly_memuse | Detect | 38654704962.000000 When tcp.reassembly_memuse topped at 38654704962 suricata started to lose packets. Now, with inline: no, after 10 minutes I have: tcp.segment_memcap_drop | Detect | 0 tcp.reassembly_memuse | Detect | 15080209344.000000 growing slowing... So, it seems that I'm actually not losing packets... I will it run until memuse values reaches 38..... From petermanev at gmail.com Thu Feb 23 04:58:59 2012 From: petermanev at gmail.com (Peter Manev) Date: Thu, 23 Feb 2012 10:58:59 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> Message-ID: > > reassembly: > memcap: 2048mb > I like that :) > -- > Regards, Peter Manev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120223/1c5cb356/attachment.html From victor at inliniac.net Thu Feb 23 05:18:33 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 11:18:33 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> Message-ID: <4F461279.6010608@inliniac.net> On 02/23/2012 10:45 AM, Travel Factory S.r.l. wrote: > depth: 50mb # reassemble 1mb into a stream Any particular reason for this setting? This means large transfers, like big downloads, will be tracked much longer than normal. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From mc8647 at mclink.it Thu Feb 23 05:49:28 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 11:49:28 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F461279.6010608@inliniac.net> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> Message-ID: On Thu, 23 Feb 2012 11:18:33 +0100 Victor Julien wrote: > On 02/23/2012 10:45 AM, Travel Factory S.r.l. wrote: >> depth: 50mb # reassemble 1mb into a stream > > Any particular reason for this setting? This means large transfers, >like > big downloads, will be tracked much longer than normal. No, actually I raised every parameter regarding memory. I should read again the suricata.yaml parameters description. Should I lower it ? Anyhow, as expected, after 35:00, tcp.segment_memcap_drop | Detect | 0 tcp.reassembly_memuse | Detect | 38506791088.000000 tcp.segment_memcap_drop | Detect | 0 tcp.reassembly_memuse | Detect | 38596590000.000000 tcp.segment_memcap_drop | Detect | 157 tcp.reassembly_memuse | Detect | 38654700066.000000 tcp.segment_memcap_drop | Detect | 6057 tcp.reassembly_memuse | Detect | 38654705250.000000 tcp.segment_memcap_drop | Detect | 13473 The only rule file active has these 2 rules: alert tcp any any -> any any (msg:"FILE-IDENTIFY PDF file magic detection"; flow:to_client,established; file_data; content:"%PDF-"; fast_pattern; nocase; flowbits:set,file.pdf; classtype:misc-activity; sid:2049499999; rev:3;) alert ip [10.my.ip.address] any -> any any (msg:"FRANK traffic"; threshold: type limit, track by_src, seconds 60, count 1; sid:2405998999; rev:277;) The second rule is triggered and I see one message every 60 seconds, the first rule is not triggered when I do traffic from my pc but I see it in the log when traffic is made from other workstations... is the second rule masking the first ??? Or am I still losing packets ??? From mc8647 at mclink.it Thu Feb 23 07:24:58 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 13:24:58 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> Message-ID: loaded 2 new disks, installed a dual port card based on 82571EB (e1000e driver), installing a 2.6.38 kernel ubuntu.... From victor at inliniac.net Thu Feb 23 08:18:16 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 14:18:16 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> Message-ID: <4F463C98.7030802@inliniac.net> On 02/23/2012 11:49 AM, Travel Factory S.r.l. wrote: > On Thu, 23 Feb 2012 11:18:33 +0100 > Victor Julien wrote: >> On 02/23/2012 10:45 AM, Travel Factory S.r.l. wrote: >>> depth: 50mb # reassemble 1mb into a stream >> >> Any particular reason for this setting? This means large transfers, like >> big downloads, will be tracked much longer than normal. > > No, actually I raised every parameter regarding memory. I should read > again the suricata.yaml parameters description. > Should I lower it ? Might help, ya. > > Anyhow, as expected, after 35:00, > > tcp.segment_memcap_drop | Detect | 0 > tcp.reassembly_memuse | Detect | 38506791088.000000 > tcp.segment_memcap_drop | Detect | 0 > tcp.reassembly_memuse | Detect | 38596590000.000000 > tcp.segment_memcap_drop | Detect | 157 > tcp.reassembly_memuse | Detect | 38654700066.000000 > tcp.segment_memcap_drop | Detect | 6057 > tcp.reassembly_memuse | Detect | 38654705250.000000 > tcp.segment_memcap_drop | Detect | 13473 You might want to lower the flow time outs for TCP in your yaml file. > > > The only rule file active has these 2 rules: > > alert tcp any any -> any any (msg:"FILE-IDENTIFY PDF file magic > detection"; flow:to_client,established; file_data; content:"%PDF-"; > fast_pattern; nocase; flowbits:set,file.pdf; classtype:misc-activity; > sid:2049499999; rev:3;) > > alert ip [10.my.ip.address] any -> any any (msg:"FRANK traffic"; > threshold: type limit, track by_src, seconds 60, count 1; > sid:2405998999; rev:277;) > > > The second rule is triggered and I see one message every 60 seconds, the > first rule is not triggered when I do traffic from my pc but I see it in > the log when traffic is made from other workstations... is the second > rule masking the first ??? Or am I still losing packets ??? > Might be caused by bad checksums. Try disabling stream.checksum_validation. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From r.fulton at auckland.ac.nz Thu Feb 23 09:05:30 2012 From: r.fulton at auckland.ac.nz (Russell Fulton) Date: Fri, 24 Feb 2012 03:05:30 +1300 Subject: [Oisf-users] looking for debian packages... In-Reply-To: <4F45EDFA.6010409@inliniac.net> References: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> <4F45EDFA.6010409@inliniac.net> Message-ID: Thanks Victor, after talking to our linux admins we have decided to move the box to 'testing' so this is now a non-issue. Russell On 23/02/2012, at 8:42 PM, Victor Julien wrote: > On 02/23/2012 02:02 AM, Russell Fulton wrote: >> Hi >> >> I am running debian squeeze on my sensors. The suricata package for squeeze is at 1.0.1. I see sid has 1.2.1? >> >> Does anyone have an up to date .deb that I can use on 6.0.4 (squeeze)? >> >> Anyone running the sid package under squeeze? > > Debian testing has 1.2.1 as well. You might want to try pulling the > testing/sid debian src package and rebuilding it on your squeeze box. I > think you have a good chance it will work. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From victor at inliniac.net Thu Feb 23 09:10:04 2012 From: victor at inliniac.net (Victor Julien) Date: Thu, 23 Feb 2012 15:10:04 +0100 Subject: [Oisf-users] looking for debian packages... In-Reply-To: References: <3C4F7AC6-BADD-4A69-9765-5A28E20D190C@auckland.ac.nz> <4F45EDFA.6010409@inliniac.net> Message-ID: <4F4648BC.5030409@inliniac.net> Great choice. A debian dev once told me: "no debian developer runs debian stable" :) On 02/23/2012 03:05 PM, Russell Fulton wrote: > Thanks Victor, after talking to our linux admins we have decided to move the box to 'testing' so this is now a non-issue. > > Russell > > On 23/02/2012, at 8:42 PM, Victor Julien wrote: > >> On 02/23/2012 02:02 AM, Russell Fulton wrote: >>> Hi >>> >>> I am running debian squeeze on my sensors. The suricata package for squeeze is at 1.0.1. I see sid has 1.2.1? >>> >>> Does anyone have an up to date .deb that I can use on 6.0.4 (squeeze)? >>> >>> Anyone running the sid package under squeeze? >> >> Debian testing has 1.2.1 as well. You might want to try pulling the >> testing/sid debian src package and rebuilding it on your squeeze box. I >> think you have a good chance it will work. >> >> -- >> --------------------------------------------- >> Victor Julien >> http://www.inliniac.net/ >> PGP: http://www.inliniac.net/victorjulien.asc >> --------------------------------------------- >> >> _______________________________________________ >> Oisf-users mailing list >> Oisf-users at openinfosecfoundation.org >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- From mcholste at gmail.com Thu Feb 23 10:18:24 2012 From: mcholste at gmail.com (Martin Holste) Date: Thu, 23 Feb 2012 09:18:24 -0600 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: <4F463C98.7030802@inliniac.net> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> <4F463C98.7030802@inliniac.net> Message-ID: Some key settings I'm using on 750 Mb/sec with similar hardware but more RAM: # Set context to full (you might not have enough RAM for this) detect-engine: - profile: high - custom-values: toclient_src_groups: 2 toclient_dst_groups: 2 toclient_sp_groups: 2 toclient_dp_groups: 3 toserver_src_groups: 2 toserver_dst_groups: 4 toserver_sp_groups: 2 toserver_dp_groups: 25 - sgh-mpm-context: full - inspection-recursion-limit: 3000 # Run ac (you might not have enough RAM for this) mpm-algo: ac # Larger than default memcap and prune_flows flow: memcap: 3294967295 hash_size: 108435456 prealloc: 10000 emergency_recovery: 40 prune_flows: 500 # Use fewer detect threads threading: detect_thread_ratio: .5 # Aggressive flow timeouts flow-timeouts: default: new: 1 # 30 established: 10 #300 closed: 0 emergency_new: 1 #10 emergency_established: 1 #100 emergency_closed: 0 tcp: new: 1 #60 established: 10 #3600 closed: 0 #120 emergency_new: 1 #10 emergency_established: 5 #1 #300 emergency_closed: 20 udp: new: 1 #30 established: 1 #300 emergency_new: 1 #10 emergency_established: 1 #100 icmp: new: 1 #30 established: 1 #300 emergency_new: 1 #10 emergency_established: 1 #100 # Larger stream buffer stream: memcap: 3294967295 checksum_validation: no # reject wrong csums inline: no # no inline mode reassembly: memcap: 4294967295 depth: 1048576 # reassemble 1mb into a stream toserver_chunk_size: 2560 toclient_chunk_size: 2560 On Thu, Feb 23, 2012 at 7:18 AM, Victor Julien wrote: > On 02/23/2012 11:49 AM, Travel Factory S.r.l. wrote: >> On Thu, 23 Feb 2012 11:18:33 +0100 >> ?Victor Julien wrote: >>> On 02/23/2012 10:45 AM, Travel Factory S.r.l. wrote: >>>> ? ? depth: 50mb ? ? ? ? ? ? ? ? ?# reassemble 1mb into a stream >>> >>> Any particular reason for this setting? This means large transfers, like >>> big downloads, will be tracked much longer than normal. >> >> No, actually I raised every parameter regarding memory. I should read >> again the suricata.yaml parameters description. >> Should I lower it ? > > Might help, ya. > >> >> Anyhow, as expected, after 35:00, >> >> tcp.segment_memcap_drop ? | Detect ? ? ? ? ? ? ? ? ? ?| 0 >> tcp.reassembly_memuse ? ? | Detect ? ? ? ? ? ? ? ? ? ?| 38506791088.000000 >> tcp.segment_memcap_drop ? | Detect ? ? ? ? ? ? ? ? ? ?| 0 >> tcp.reassembly_memuse ? ? | Detect ? ? ? ? ? ? ? ? ? ?| 38596590000.000000 >> tcp.segment_memcap_drop ? | Detect ? ? ? ? ? ? ? ? ? ?| 157 >> tcp.reassembly_memuse ? ? | Detect ? ? ? ? ? ? ? ? ? ?| 38654700066.000000 >> tcp.segment_memcap_drop ? | Detect ? ? ? ? ? ? ? ? ? ?| 6057 >> tcp.reassembly_memuse ? ? | Detect ? ? ? ? ? ? ? ? ? ?| 38654705250.000000 >> tcp.segment_memcap_drop ? | Detect ? ? ? ? ? ? ? ? ? ?| 13473 > > You might want to lower the flow time outs for TCP in your yaml file. > >> >> >> The only rule file active has these 2 rules: >> >> alert tcp any any -> any any (msg:"FILE-IDENTIFY PDF file magic >> detection"; flow:to_client,established; file_data; content:"%PDF-"; >> fast_pattern; nocase; flowbits:set,file.pdf; classtype:misc-activity; >> sid:2049499999; rev:3;) >> >> alert ip [10.my.ip.address] any -> any any (msg:"FRANK traffic"; >> threshold: type limit, track by_src, seconds 60, count 1; >> sid:2405998999; rev:277;) >> >> >> The second rule is triggered and I see one message every 60 seconds, the >> first rule is not triggered when I do traffic from my pc but I see it in >> the log when traffic is made from other workstations... is the second >> rule masking the first ??? Or am I still losing packets ??? >> > > Might be caused by bad checksums. Try disabling stream.checksum_validation. > > -- > --------------------------------------------- > Victor Julien > http://www.inliniac.net/ > PGP: http://www.inliniac.net/victorjulien.asc > --------------------------------------------- > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From mc8647 at mclink.it Thu Feb 23 10:57:33 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 16:57:33 +0100 Subject: [Oisf-users] My missing packets are back ! In-Reply-To: <4F463C98.7030802@inliniac.net> References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> <4F463C98.7030802@inliniac.net> Message-ID: Now I have my missing packets back. New linux with 2.6.38 kernel, suricata 1.2.1, e1000e lan card, starting from clean yaml file with all rules removed but my own... Started suricata with: suricata -c /etc/suricata/suricata.yaml --af-packet=eth4 --runmode=autofp stats.log reports: Date: 2/23/2012 -- 16:51:35 (uptime: 0d, 01h 11m 04s) tcp.memuse | Detect | 237072960.000000 tcp.segment_memcap_drop | Detect | 0 tcp.reassembly_memuse | Detect | 12458622240.000000 flow.memuse | FlowManagerThread | 27841936.000000 The reassembly_memuse counter keeps adding, sometimes it lowers a bit... Changes applied are (please tell me which one are not important) -#max-pending-packets: 50 +max-pending-packets: 5000 -#default-packet-size: 1514 +default-packet-size: 15140 - cluster-type: cluster_round_robin + cluster-type: cluster_flow stream: - memcap: 32mb + memcap: 2048mb reassembly: - memcap: 64mb + memcap: 2048mb (and rules removed) I also run the tuning network kernel parameters I found in the mailing list.. Now, after more than one hour, I have 0 packet missed in suricata and all my test rule are triggered correctly ! Tomorrow I will try the PF_RING road... Thank everybody for your help. Francesco From mc8647 at mclink.it Thu Feb 23 11:02:10 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Thu, 23 Feb 2012 17:02:10 +0100 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> <4F463C98.7030802@inliniac.net> Message-ID: On Thu, 23 Feb 2012 09:18:24 -0600 Martin Holste wrote: > Some key settings I'm using on 750 Mb/sec with similar hardware but >more RAM: Thank you for your configuration file. I will review it tomorrow. As you can see in other message I "solved" the issues. I changed several components so I'm not sure why it now works, but I'm quite sure there were two main problems: - hardware, with checksum - software, too low STREAM.memcap value From mcholste at gmail.com Thu Feb 23 11:20:24 2012 From: mcholste at gmail.com (Martin Holste) Date: Thu, 23 Feb 2012 10:20:24 -0600 Subject: [Oisf-users] where are my missing packets ? In-Reply-To: References: <4F457C7F.8040401@mclink.it> <4F457FC5.4010906@reading.ac.uk> <4F458513.1040409@mclink.it> <4F45EF69.3050405@inliniac.net> <4F45FD35.5050108@inliniac.net> <4F461279.6010608@inliniac.net> <4F463C98.7030802@inliniac.net> Message-ID: > As you can see in other message I "solved" the issues. I saw that, congratulations! > I changed > several components so I'm not sure why it now works, but I'm quite > sure there were two main problems: > - hardware, with checksum > - software, too low STREAM.memcap value Yep, the memcap is the biggest one by far. As you get further in your testing, be sure to check out some of the advanced features Suricata offers like the newly-introduced file extraction as well as the incredibly valuable HTTP log. In production, the HTTP log to pair up with IDS alerts is critical, though there are other ways of getting that data. From tingw.liu at gmail.com Fri Feb 24 02:10:33 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Fri, 24 Feb 2012 15:10:33 +0800 Subject: [Oisf-users] PF_RING core dump Message-ID: My suricata is well with the normal mode. It core dumped while with the pfring mode. Just like: suricata -c /etc/suricata/suricata.yaml --pfring=eth0 PF_RING version is 5.2.1 suricata version is 1.2.1 I have debug it by gdb: (gdb) bt #0 0x0000003cfc13661d in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x0000003291a27c73 in pfring_mod_recv (ring=0x7ffff0000940, buffer=0x1cd13d8, buffer_len=1516, hdr=0x7ffff6b89500, wait_for_incoming_packet=1 '\001') at pfring_mod.c:422 #2 0x0000003291a25e9d in pfring_recv (ring=0x7ffff0000940, buffer=0x1cd13d8, buffer_len=1516, hdr=0x7ffff6b89500, wait_for_incoming_packet=1 '\001') at pfring.c:335 #3 0x0000000000420208 in ReceivePfringLoop () #4 0x00000000006f0867 in TmThreadsSlotPktAcqLoop () #5 0x0000003cfc407cd1 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003cfc0dfd3d in clone () from /lib64/libc.so.6 (gdb) The bug like in pfring, but other program depend this pfring is ok. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120224/c8b0b21e/attachment.html From mc8647 at mclink.it Mon Feb 27 17:17:49 2012 From: mc8647 at mclink.it (mc8647) Date: Mon, 27 Feb 2012 23:17:49 +0100 Subject: [Oisf-users] Suricata logs perfectly... and now ?! Message-ID: <4F4C010D.4010607@mclink.it> Ok, now Suricata runs and collect massive amounts of logs.... Thanks you to everybody for your help. I started to implement suricata because in January I found several strange connections in proxy logs. So I started to trace them down and we found Zeus installed on several PCs. More PCs were infected daily, with the av unable to stop them. So I collected all the samples I could, sent to av company etc etc, blocked the C&C/DROP IP at the firewall, cleaned the PCs... When I could trace the infections in proxy logs I found the same log lines: an advertising circuit, a couple of PDF downloads, .jar files (org.class, net.class, com.class...) and finally the payload... Seconds after the payload, a https connection to a hungarian IP (also hosting adult sites) started. The IP address is not listed in any "bot", "cc", malware-related list. Also the domain used were not blacklisted.... Installing Suricata I was expecting to find a lot more infected PCs. I enabled only the "malware"/"botnet" related rules and I found several PCs with: ET MALWARE dialno Dialer User-Agent (dialno) ET MALWARE Suspicious Mozilla User-Agent - Likely Fake (Mozilla/4.0) ET MALWARE AskSearch Toolbar Spyware User-Agent (AskTBar) ET MALWARE Fun Web Products Agent Traffic ET MALWARE Simbar Spyware User-Agent Detected ET MALWARE Casalemedia Spyware I also found several DROP/C&C log lines (but I don't have them with me) but several of them were FP and infact updating ET rules some of the IPs were not listed anymore... Now I'd like to ask about my results: - is this a classic outcome of a Suricata network log ? I was expecting way more infected PCs ! Or are the "free" rules not enough to catch in the wild malware ? - are these toolbar/spyware really really malware ? About the logs data: - how do you handle all this massive log data ? - how do you analyze it ? ok, I have tons of other questions but the message is already too long, so I can summarize with the message title: Suricata now works... and now what to do with all this data ? Thanks Francesco From mcholste at gmail.com Mon Feb 27 17:46:27 2012 From: mcholste at gmail.com (Martin Holste) Date: Mon, 27 Feb 2012 16:46:27 -0600 Subject: [Oisf-users] Suricata logs perfectly... and now ?! In-Reply-To: <4F4C010D.4010607@mclink.it> References: <4F4C010D.4010607@mclink.it> Message-ID: First off, you'll want to make sure that in addition to turning on MALWARE you have TROJAN on as well--those are the bulk of the Zeus and other sigs. You will also want some of the POLICY sigs on, especially the executable download sig. As for the massive amount of log data: That's something of a specialty of mine. I started the ELSA project to cope with exactly the situation you describe. Check it out here: http://code.google.com/p/enterprise-log-search-and-archive/ . Especially of note, ELSA will handle the HTTP logs that Suricata will create, which really helps when you're reviewing alerts. On Mon, Feb 27, 2012 at 4:17 PM, mc8647 wrote: > > Ok, now Suricata runs and collect massive amounts of logs.... Thanks you > to everybody for your help. > > I started to implement suricata because in January I found several > strange connections in proxy logs. So I started to trace them down and > we found Zeus installed on several PCs. More PCs were infected daily, > with the av unable to stop them. > So I collected all the samples I could, sent to av company etc etc, > blocked the C&C/DROP IP at the firewall, cleaned the PCs... > > When I could trace the infections in proxy logs I found the same log > lines: an advertising circuit, a couple of PDF downloads, .jar files > (org.class, net.class, com.class...) and finally the payload... Seconds > after the payload, a https connection to a hungarian IP (also hosting > adult sites) started. The IP address is not listed in any "bot", "cc", > malware-related list. Also the domain used were not blacklisted.... > > Installing Suricata I was expecting to find a lot more infected PCs. I > enabled only the "malware"/"botnet" related rules and I found several > PCs with: > ET MALWARE dialno Dialer User-Agent (dialno) > ET MALWARE Suspicious Mozilla User-Agent - Likely Fake (Mozilla/4.0) > ET MALWARE AskSearch Toolbar Spyware User-Agent (AskTBar) > ET MALWARE Fun Web Products Agent Traffic > ET MALWARE Simbar Spyware User-Agent Detected > ET MALWARE Casalemedia Spyware > > I also found several DROP/C&C log lines (but I don't have them with me) > but several of them were FP and infact updating ET rules some of the IPs > were not listed anymore... > > > Now I'd like to ask about my results: > - is this a classic outcome of a Suricata network log ? I was expecting > way more infected PCs ?! Or are the "free" rules not enough to catch in > the wild malware ? > - are these toolbar/spyware really really malware ? > > About the logs data: > - how do you handle all this massive log data ? > - how do you analyze it ? > > ok, I have tons of other questions but the message is already too long, > so I can summarize with the message title: > Suricata now works... and now what to do with all this data ? > > Thanks > Francesco > > _______________________________________________ > Oisf-users mailing list > Oisf-users at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users From mc8647 at mclink.it Mon Feb 27 18:24:33 2012 From: mc8647 at mclink.it (Travel Factory S.r.l.) Date: Tue, 28 Feb 2012 00:24:33 +0100 Subject: [Oisf-users] Suricata logs perfectly... and now ?! Message-ID: An embedded message was scrubbed... From: "Travel Factory S.r.l." Subject: Re: [Oisf-users] Suricata logs perfectly... and now ?! Date: Tue, 28 Feb 2012 00:23:35 +0100 Size: 2985 Url: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120228/61401c4b/attachment.mht From mcholste at gmail.com Mon Feb 27 18:46:29 2012 From: mcholste at gmail.com (Martin Holste) Date: Mon, 27 Feb 2012 17:46:29 -0600 Subject: [Oisf-users] Suricata logs perfectly... and now ?! In-Reply-To: References: Message-ID: > Nice to know, tomorrow I will check if they are enabled... Jut one thing: our firewall blocks some subnets so there is never a tcp connection established from some of the IPs. > Today I enabled the bot and bot-cc rule files and I found a couple of hits I will investigate tomorrow. But they are based on ip traffic and as far as I understand doesn't need a tcp session so they fire when a client tries to establish a connection without success, while the rules based on content won't fire if firewall blocks traffic.... It is for this reason that I recommend NOT blocking IP's which are known bot controllers. It is a bit counter-intuitive, but blocking those IP's only delays the amount of time in which you discover infected hosts, and if you do not notice the blocks, the C2 domains will eventually resolve to different, previously unknown, IP's which are not blocked by your firewall. Ideally, a sinkhole solution is setup so that you can both prevent data from being posted to the C2 while still taking advantage of the IDS rules. > Yes, really ! http.log is fantastic... and grep is not the right tool to handle 1,5GB of daily http.log. In order to pipe the http.log into syslog (for ELSA), you will need to configure rsyslog (default on Ubuntu) or syslog-ng to send the file. This is simple, and I have an example (using Bro logs) on my blog here: http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html . > So, now I have the logs, I will have a tool to analize them and... what is missing ? Knowledge about the attacks ? data correlation ? > For example: an user today triggered a rule about using a "bad" domain. I saw from http.log he/she was on twitter, clicked on a link t.co (the redirector of twitter) then passed through a couple of short-name domains, then on a wear that redirected to www.google.com... > Isn't it strange ? How would you investigate such things ? I tried to use wget and got the same results... The next step I would recommend is setting up StreamDB.googlecode.com, which is a pcap collector of sorts that I created for this purpose. You will need to compile Vortex, but that should be straightforward. See the documentation here on getting started: http://code.google.com/p/streamdb/wiki/INSTALL . That will plug into ELSA (by configuring the pcap_url config variable), so that you can get instant access to the full transcript of any Snort alert, HTTP URL, or other log with two clicks and zero seconds. This will allow you to fully investigate these kinds of alerts which aren't clear at first. If you run into any problems, contact us on the ELSA list over here: https://groups.google.com/forum/?fromgroups#!forum/enterprise-log-search-and-archive . From tingw.liu at gmail.com Wed Feb 29 05:57:29 2012 From: tingw.liu at gmail.com (tingwei liu) Date: Wed, 29 Feb 2012 18:57:29 +0800 Subject: [Oisf-users] IPS mode performance is very poor, why? Message-ID: I have installed suricata-1.2.1 with enable nfqueue on fedora 15 system. #>iptables -I FORWARD -j NFQUEUE --queue-num 3 #>suricata -c /etc/suricata/suricata.yaml -q 3 -D Only emergency-ftp.rules loaded. It works, but performance is very poor. I test it by transfer files from ftp server. Before running last two commands, the bandwidth is 100Mbps; After nfqueue and suricata running, the bandwidth only 1Mbps. Who can tell me which parameters should be changed ? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120229/cf825594/attachment.html