[Oisf-devel] [Oisf-users] Suricata 1.3 Available!

Martin Holste mcholste at gmail.com
Sun Jul 15 17:23:40 UTC 2012 

Unfortunately, yes, it ended up dropping about 20-30% in the following
days.  Interestingly, it wasn't necessarily during peak times, but
seemed more to be with process time.  That is, instead of dropping at
peak, it would drop the longer the process would run, so there must be
something accruing that's slowing it down over time.  Since we have
enough ram, I switched back to ac, but hopefully that info is helpful.

On Sun, Jul 15, 2012 at 8:53 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
> Hey Martin
>
> Anything(drops) from the ac-bs run?
>
> On Sun, Jul 8, 2012 at 12:42 AM, Martin Holste <mcholste at gmail.com> wrote:
>> Well, I can't say no drops ever, but on a normal day, I don't see any.
>>  I'll run with ac-bs for a few days and see how it does to further
>> confirm my initial findings.  The initial test did cover peak traffic,
>> so I'm betting that we'll continue to see no drops.
>>
>> On Sat, Jul 7, 2012 at 11:45 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>>> On Sat, Jul 7, 2012 at 9:28 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>> Thanks for the explanation.  We run about 7k rules on about 800 Mb/sec
>>>> of traffic.  We had run ac full which had performed very well (no
>>>> drops).  I tried out ac-bs yesterday for the latter part of the day
>>>> and we got about the same performance as ac full.  The ram usage was
>>>> indeed significantly less.  It decreased from about 35G with ac full
>>>> to about 4G with ac-bs.  Since we couldn't get it to drop in
>>>> yesterday's testing, I guess we don't know where the line is where
>>>> ac-bs is inferior to ac full, but it's clear that for many, ac-bs will
>>>> be very important as folks are far more likely to have 8G of ram than
>>>> the 144G we have, which means that a machine with 8G should now be
>>>> able to handle 1 Gb line-rate with a large rule set.
>>>>
>>>
>>> Indeed.
>>>
>>> Good to hear about no drops.  Planning to run it longer and see how it
>>> holds out against some peak traffic?
>>>
>>> You haven't seen any drops at all so far with ac full?
>>>
>>>> On Fri, Jul 6, 2012 at 11:33 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>>>>> On Fri, Jul 6, 2012 at 9:05 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>>> Can Anoop describe a bit about ac-bs?  I read the .c header and it
>>>>>> interested in the design goals and overall idea.
>>>>>>
>>>>>
>>>>> I have removed all the state transitions that are part of 0 state transition.
>>>>>
>>>>> For example,
>>>>>
>>>>> 0-state transitions :
>>>>> ascii_code - transition
>>>>> 0 - 1
>>>>> 1 - 2
>>>>> 2 - 3
>>>>> 3 - 4
>>>>> .....
>>>>>
>>>>> 5-state transitions:
>>>>> 0 - 25
>>>>> 1 - 30
>>>>> 2 - 35
>>>>> 3 - 1 /* this transition actually comes out of state-0.  We'll remove this */
>>>>> .....
>>>>>
>>>>> This would compress the state table by quite a margin.  Leaves us with
>>>>> just 1 or 2 transitions per state which is quite a reduction from 256
>>>>> transitions.  I use binary search to search for the right transition.
>>>>> Brought down the table sizes to a couple of mbs from a 100 odd.  These
>>>>> are all my table sizes -
>>>>>
>>>>>
>>>>> * in bytes
>>>>>
>>>>> "ac-bs"
>>>>> 24348
>>>>> 38486
>>>>> 118900
>>>>> 47736
>>>>> 4716
>>>>> 4648804
>>>>> 558
>>>>> 15874
>>>>> 266202
>>>>> 6838
>>>>> 696
>>>>> 692
>>>>> 3982784
>>>>> 10756976
>>>>>
>>>>> In single mode it's not as fast as "ac".  "full" mode performs well.
>>>>>
>>>>>> looks like I should use this one since I have a ton of RAM, but I'm
>>>>>
>>>>> mem consumption would be low by *quite* a margin compared to "ac".
>>>>> The small table size should give good cache perf, but not as good as
>>>>> "ac"
>>>>>
>>>>> Probably can give it a shot with both "single" and "full" mode and see
>>>>> how it performs.
>>>>>
>>>>>> On Fri, Jul 6, 2012 at 10:23 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>>>> The OISF development team is proud to announce Suricata 1.3. This
>>>>>>> release is a major improvement over the previous releases with regard to
>>>>>>> performance, scalability and accuracy. Also, a number of great features
>>>>>>> have been added.
>>>>>>>
>>>>>>> Major new features:
>>>>>>>
>>>>>>> - TLS/SSL handshake parser and rule keywords for detecting anomolies in
>>>>>>> TLS/SSL traffic
>>>>>>> - HTTP user agent keyword for matching directly on User-Agent header
>>>>>>> - On the fly MD5 calculation and matching for files in HTTP streams
>>>>>>>
>>>>>>> New / improved hardware support
>>>>>>>
>>>>>>> - Napatech support added
>>>>>>> - Endace support improved
>>>>>>> - New runmode for users of pcap wrappers (Myricom, PF_RING, others)
>>>>>>>
>>>>>>> Get the new release here:
>>>>>>> http://www.openinfosecfoundation.org/download/suricata-1.3.tar.gz
>>>>>>>
>>>>>>> The configuration file has evolved but backward compatibility is
>>>>>>> provided. We thus encourage you to update your suricata configuration
>>>>>>> file. Upgrade guidance is provided here:
>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_12_to_Suricata_13
>>>>>>>
>>>>>>> Detailed list of changes:
>>>>>>>
>>>>>>> New features
>>>>>>>
>>>>>>> - TLS/SSL handshake parser, tls.subjectdn and tls.issuerdn keywords
>>>>>>> (#296, contributed by Pierre Chifflier)
>>>>>>> - http_user_agent keyword for matching on the HTTP User-Agent header
>>>>>>> - experimental live rule reload by sending a USR2 signal (#279)
>>>>>>> - AF_PACKET BPF support (#449)
>>>>>>> - AF_PACKET live packet loss counters (#441)
>>>>>>> - Ringbuffer and zero copy support for AF_PACKET
>>>>>>> - add pcap workers runmode for use with libpcap wrappers that support
>>>>>>> load balancing, such as  Napatech's or Myricom's
>>>>>>> - Napatech capture card support (contributed by Randy Caldejon -- nPulse)
>>>>>>> - Test mode: -T option to test the config (#271)
>>>>>>> - Rule analyzer (#349)
>>>>>>> - On the fly md5 checksum calculation of extracted files
>>>>>>> - File extraction for HTTP POST request that do not use multipart bodies
>>>>>>> - Scripts for looking up files / file md5's at Virus Total and others
>>>>>>> (contributed by Martin Holste)
>>>>>>> - Experimental support for matching on large lists of known file MD5
>>>>>>> checksums
>>>>>>> - negated filemd5 matching, allowing for md5 whitelisting
>>>>>>> - Line based file log, in json format
>>>>>>> - New multi pattern engine: ac-bs
>>>>>>> - Basic support for including other yaml files into the main yaml
>>>>>>> - Commandline options to list supported app layer protocols and keywords
>>>>>>> (#344, #414)
>>>>>>> - Profiling improvements, added lock profiling code
>>>>>>>
>>>>>>>
>>>>>>> Improvements
>>>>>>>
>>>>>>> - Major rewrite of flow engine, improving scalability.
>>>>>>> - New default runmode: "autofp" (#433)
>>>>>>> - Improved scalability for Tag and Threshold subsystems
>>>>>>> - Support for PF_RING 5.4 added. Many thanks to Chris Wakelin (#459).
>>>>>>> - Improved Endace DAG support (#431, Jason Ish -- Endace)
>>>>>>> - Split "file" output into "file-store" and "file-log" outputs
>>>>>>> - Much improved file extraction
>>>>>>> - Improvements to HTTP handling: multipart parsing, gzip decompression.
>>>>>>> - Improved performance for file_data, http_server_body and
>>>>>>> http_client_body keywords.
>>>>>>> - Improved HTTP CONNECT support in libhtp (#427, Brian Rectanus -- Qualys)
>>>>>>> - http_cookie keyword now also inspects "Set-Cookie" header (#479)
>>>>>>> - http_raw_header keyword inspects original header line terminators (#475)
>>>>>>> - deal with double encoded URI (#464)
>>>>>>> - Improved http_stat_msg and http_stat_code keywords (#394)
>>>>>>> - Unified yaml naming convention, including fallback support (by Nikolay
>>>>>>> Denev)
>>>>>>> - Made the rule keyword parser much stricter in detecting syntax errors
>>>>>>> - Improved error reporting when using too long address strings (#451).
>>>>>>> - Rule parser is made more strict.
>>>>>>> - Byte_extract can support negative offsets now (#445).
>>>>>>> - HOME_NET and EXTERNAL_NET and the other vars are now checked for
>>>>>>> common errors (#454).
>>>>>>> - Unified2 output overhaul, logging individual segments in more cases.
>>>>>>> - signatures with depth and/or offset are now checked against packets in
>>>>>>> addition to the stream (#404)
>>>>>>>
>>>>>>>
>>>>>>> Changes since 1.3rc1
>>>>>>>
>>>>>>> - make live rule reloads optional and disabled by default
>>>>>>> - fix a shutdown bug
>>>>>>> - fix several memory leaks (#492)
>>>>>>> - warn user if global and rule thresholding conflict (#455)
>>>>>>> - set thread names on FreeBSD (Nikolay Denev)
>>>>>>> - Fix PF_RING building on Ubuntu 12.04
>>>>>>> - rule analyzer updates
>>>>>>> - file inspection improvements when dealing with limits (#493)
>>>>>>>
>>>>>>>
>>>>>>> Credits
>>>>>>>
>>>>>>>   Brian Rectanus -- Qualys
>>>>>>>   Randy Caldejon -- nPulse
>>>>>>>   Pierre Chifflier
>>>>>>>   Coverity
>>>>>>>   Nikolay Denev
>>>>>>>   Jason Ish -- Endace
>>>>>>>   Martin Holste
>>>>>>>   Napatech
>>>>>>>   Rmkml
>>>>>>>   Michel Sarborde
>>>>>>>   Chris Wakelin
>>>>>>>   Joshua White
>>>>>>>
>>>>>>> Known issues & missing features
>>>>>>>
>>>>>>> If you encounter issues, please let us know! As always, we are doing our
>>>>>>> best to make you aware of continuing development and items within the
>>>>>>> engine that are not yet complete or optimal.  With this in mind, please
>>>>>>> notice the list we have included of known items we are working on.
>>>>>>>
>>>>>>> See http://redmine.openinfosecfoundation.org/projects/suricata/issues
>>>>>>> for an up to date list and to report new issues. See
>>>>>>> http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues
>>>>>>> for a discussion and time line for the major issues.
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------------------
>>>>>>> Victor Julien
>>>>>>> http://www.inliniac.net/
>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-users mailing list
>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>> _______________________________________________
>>>>>> Oisf-devel mailing list
>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anoop Saldanha
>>>
>>>
>>>
>>> --
>>> Anoop Saldanha
>
>
>
> --
> Anoop Saldanha

More information about the Oisf-devel mailing list