[Oisf-devel] [Oisf-users] Suricata 1.3 Available!

[Anoop Saldanha]

Strange.  Might be a bug.

Will have a look at the mpm.   Thanks for the info.

On Sun, Jul 15, 2012 at 10:53 PM, Martin Holste <mcholste at gmail.com> wrote:
> Unfortunately, yes, it ended up dropping about 20-30% in the following
> days.  Interestingly, it wasn't necessarily during peak times, but
> seemed more to be with process time.  That is, instead of dropping at
> peak, it would drop the longer the process would run, so there must be
> something accruing that's slowing it down over time.  Since we have
> enough ram, I switched back to ac, but hopefully that info is helpful.
>
> On Sun, Jul 15, 2012 at 8:53 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>> Hey Martin
>>
>> Anything(drops) from the ac-bs run?
>>
>> On Sun, Jul 8, 2012 at 12:42 AM, Martin Holste <mcholste at gmail.com> wrote:
>>> Well, I can't say no drops ever, but on a normal day, I don't see any.
>>>  I'll run with ac-bs for a few days and see how it does to further
>>> confirm my initial findings.  The initial test did cover peak traffic,
>>> so I'm betting that we'll continue to see no drops.
>>>
>>> On Sat, Jul 7, 2012 at 11:45 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>>>> On Sat, Jul 7, 2012 at 9:28 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>> Thanks for the explanation.  We run about 7k rules on about 800 Mb/sec
>>>>> of traffic.  We had run ac full which had performed very well (no
>>>>> drops).  I tried out ac-bs yesterday for the latter part of the day
>>>>> and we got about the same performance as ac full.  The ram usage was
>>>>> indeed significantly less.  It decreased from about 35G with ac full
>>>>> to about 4G with ac-bs.  Since we couldn't get it to drop in
>>>>> yesterday's testing, I guess we don't know where the line is where
>>>>> ac-bs is inferior to ac full, but it's clear that for many, ac-bs will
>>>>> be very important as folks are far more likely to have 8G of ram than
>>>>> the 144G we have, which means that a machine with 8G should now be
>>>>> able to handle 1 Gb line-rate with a large rule set.
>>>>>
>>>>
>>>> Indeed.
>>>>
>>>> Good to hear about no drops.  Planning to run it longer and see how it
>>>> holds out against some peak traffic?
>>>>
>>>> You haven't seen any drops at all so far with ac full?
>>>>
>>>>> On Fri, Jul 6, 2012 at 11:33 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>>>>>> On Fri, Jul 6, 2012 at 9:05 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>>>> Can Anoop describe a bit about ac-bs?  I read the .c header and it
>>>>>>> interested in the design goals and overall idea.
>>>>>>>
>>>>>>
>>>>>> I have removed all the state transitions that are part of 0 state transition.
>>>>>>
>>>>>> For example,
>>>>>>
>>>>>> 0-state transitions :
>>>>>> ascii_code - transition
>>>>>> 0 - 1
>>>>>> 1 - 2
>>>>>> 2 - 3
>>>>>> 3 - 4
>>>>>> .....
>>>>>>
>>>>>> 5-state transitions:
>>>>>> 0 - 25
>>>>>> 1 - 30
>>>>>> 2 - 35
>>>>>> 3 - 1 /* this transition actually comes out of state-0.  We'll remove this */
>>>>>> .....
>>>>>>
>>>>>> This would compress the state table by quite a margin.  Leaves us with
>>>>>> just 1 or 2 transitions per state which is quite a reduction from 256
>>>>>> transitions.  I use binary search to search for the right transition.
>>>>>> Brought down the table sizes to a couple of mbs from a 100 odd.  These
>>>>>> are all my table sizes -
>>>>>>
>>>>>>
>>>>>> * in bytes
>>>>>>
>>>>>> "ac-bs"
>>>>>> 24348
>>>>>> 38486
>>>>>> 118900
>>>>>> 47736
>>>>>> 4716
>>>>>> 4648804
>>>>>> 558
>>>>>> 15874
>>>>>> 266202
>>>>>> 6838
>>>>>> 696
>>>>>> 692
>>>>>> 3982784
>>>>>> 10756976
>>>>>>
>>>>>> In single mode it's not as fast as "ac".  "full" mode performs well.
>>>>>>
>>>>>>> looks like I should use this one since I have a ton of RAM, but I'm
>>>>>>
>>>>>> mem consumption would be low by *quite* a margin compared to "ac".
>>>>>> The small table size should give good cache perf, but not as good as
>>>>>> "ac"
>>>>>>
>>>>>> Probably can give it a shot with both "single" and "full" mode and see
>>>>>> how it performs.
>>>>>>
>>>>>>> On Fri, Jul 6, 2012 at 10:23 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>>>>> The OISF development team is proud to announce Suricata 1.3. This
>>>>>>>> release is a major improvement over the previous releases with regard to
>>>>>>>> performance, scalability and accuracy. Also, a number of great features
>>>>>>>> have been added.
>>>>>>>>
>>>>>>>> Major new features:
>>>>>>>>
>>>>>>>> - TLS/SSL handshake parser and rule keywords for detecting anomolies in
>>>>>>>> TLS/SSL traffic
>>>>>>>> - HTTP user agent keyword for matching directly on User-Agent header
>>>>>>>> - On the fly MD5 calculation and matching for files in HTTP streams
>>>>>>>>
>>>>>>>> New / improved hardware support
>>>>>>>>
>>>>>>>> - Napatech support added
>>>>>>>> - Endace support improved
>>>>>>>> - New runmode for users of pcap wrappers (Myricom, PF_RING, others)
>>>>>>>>
>>>>>>>> Get the new release here:
>>>>>>>> http://www.openinfosecfoundation.org/download/suricata-1.3.tar.gz
>>>>>>>>
>>>>>>>> The configuration file has evolved but backward compatibility is
>>>>>>>> provided. We thus encourage you to update your suricata configuration
>>>>>>>> file. Upgrade guidance is provided here:
>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_12_to_Suricata_13
>>>>>>>>
>>>>>>>> Detailed list of changes:
>>>>>>>>
>>>>>>>> New features
>>>>>>>>
>>>>>>>> - TLS/SSL handshake parser, tls.subjectdn and tls.issuerdn keywords
>>>>>>>> (#296, contributed by Pierre Chifflier)
>>>>>>>> - http_user_agent keyword for matching on the HTTP User-Agent header
>>>>>>>> - experimental live rule reload by sending a USR2 signal (#279)
>>>>>>>> - AF_PACKET BPF support (#449)
>>>>>>>> - AF_PACKET live packet loss counters (#441)
>>>>>>>> - Ringbuffer and zero copy support for AF_PACKET
>>>>>>>> - add pcap workers runmode for use with libpcap wrappers that support
>>>>>>>> load balancing, such as  Napatech's or Myricom's
>>>>>>>> - Napatech capture card support (contributed by Randy Caldejon -- nPulse)
>>>>>>>> - Test mode: -T option to test the config (#271)
>>>>>>>> - Rule analyzer (#349)
>>>>>>>> - On the fly md5 checksum calculation of extracted files
>>>>>>>> - File extraction for HTTP POST request that do not use multipart bodies
>>>>>>>> - Scripts for looking up files / file md5's at Virus Total and others
>>>>>>>> (contributed by Martin Holste)
>>>>>>>> - Experimental support for matching on large lists of known file MD5
>>>>>>>> checksums
>>>>>>>> - negated filemd5 matching, allowing for md5 whitelisting
>>>>>>>> - Line based file log, in json format
>>>>>>>> - New multi pattern engine: ac-bs
>>>>>>>> - Basic support for including other yaml files into the main yaml
>>>>>>>> - Commandline options to list supported app layer protocols and keywords
>>>>>>>> (#344, #414)
>>>>>>>> - Profiling improvements, added lock profiling code
>>>>>>>>
>>>>>>>>
>>>>>>>> Improvements
>>>>>>>>
>>>>>>>> - Major rewrite of flow engine, improving scalability.
>>>>>>>> - New default runmode: "autofp" (#433)
>>>>>>>> - Improved scalability for Tag and Threshold subsystems
>>>>>>>> - Support for PF_RING 5.4 added. Many thanks to Chris Wakelin (#459).
>>>>>>>> - Improved Endace DAG support (#431, Jason Ish -- Endace)
>>>>>>>> - Split "file" output into "file-store" and "file-log" outputs
>>>>>>>> - Much improved file extraction
>>>>>>>> - Improvements to HTTP handling: multipart parsing, gzip decompression.
>>>>>>>> - Improved performance for file_data, http_server_body and
>>>>>>>> http_client_body keywords.
>>>>>>>> - Improved HTTP CONNECT support in libhtp (#427, Brian Rectanus -- Qualys)
>>>>>>>> - http_cookie keyword now also inspects "Set-Cookie" header (#479)
>>>>>>>> - http_raw_header keyword inspects original header line terminators (#475)
>>>>>>>> - deal with double encoded URI (#464)
>>>>>>>> - Improved http_stat_msg and http_stat_code keywords (#394)
>>>>>>>> - Unified yaml naming convention, including fallback support (by Nikolay
>>>>>>>> Denev)
>>>>>>>> - Made the rule keyword parser much stricter in detecting syntax errors
>>>>>>>> - Improved error reporting when using too long address strings (#451).
>>>>>>>> - Rule parser is made more strict.
>>>>>>>> - Byte_extract can support negative offsets now (#445).
>>>>>>>> - HOME_NET and EXTERNAL_NET and the other vars are now checked for
>>>>>>>> common errors (#454).
>>>>>>>> - Unified2 output overhaul, logging individual segments in more cases.
>>>>>>>> - signatures with depth and/or offset are now checked against packets in
>>>>>>>> addition to the stream (#404)
>>>>>>>>
>>>>>>>>
>>>>>>>> Changes since 1.3rc1
>>>>>>>>
>>>>>>>> - make live rule reloads optional and disabled by default
>>>>>>>> - fix a shutdown bug
>>>>>>>> - fix several memory leaks (#492)
>>>>>>>> - warn user if global and rule thresholding conflict (#455)
>>>>>>>> - set thread names on FreeBSD (Nikolay Denev)
>>>>>>>> - Fix PF_RING building on Ubuntu 12.04
>>>>>>>> - rule analyzer updates
>>>>>>>> - file inspection improvements when dealing with limits (#493)
>>>>>>>>
>>>>>>>>
>>>>>>>> Credits
>>>>>>>>
>>>>>>>>   Brian Rectanus -- Qualys
>>>>>>>>   Randy Caldejon -- nPulse
>>>>>>>>   Pierre Chifflier
>>>>>>>>   Coverity
>>>>>>>>   Nikolay Denev
>>>>>>>>   Jason Ish -- Endace
>>>>>>>>   Martin Holste
>>>>>>>>   Napatech
>>>>>>>>   Rmkml
>>>>>>>>   Michel Sarborde
>>>>>>>>   Chris Wakelin
>>>>>>>>   Joshua White
>>>>>>>>
>>>>>>>> Known issues & missing features
>>>>>>>>
>>>>>>>> If you encounter issues, please let us know! As always, we are doing our
>>>>>>>> best to make you aware of continuing development and items within the
>>>>>>>> engine that are not yet complete or optimal.  With this in mind, please
>>>>>>>> notice the list we have included of known items we are working on.
>>>>>>>>
>>>>>>>> See http://redmine.openinfosecfoundation.org/projects/suricata/issues
>>>>>>>> for an up to date list and to report new issues. See
>>>>>>>> http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues
>>>>>>>> for a discussion and time line for the major issues.
>>>>>>>>
>>>>>>>> --
>>>>>>>> ---------------------------------------------
>>>>>>>> Victor Julien
>>>>>>>> http://www.inliniac.net/
>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>> ---------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-users mailing list
>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>> _______________________________________________
>>>>>>> Oisf-devel mailing list
>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anoop Saldanha
>>>>
>>>>
>>>>
>>>> --
>>>> Anoop Saldanha
>>
>>
>>
>> --
>> Anoop Saldanha



-- 
Anoop Saldanha