[Oisf-devel] [Oisf-users] Suricata 1.3 Available!

Sun Jul 15 13:53:46 UTC 2012

Hey Martin

Anything(drops) from the ac-bs run?

On Sun, Jul 8, 2012 at 12:42 AM, Martin Holste <mcholste at gmail.com> wrote:
> Well, I can't say no drops ever, but on a normal day, I don't see any.
>  I'll run with ac-bs for a few days and see how it does to further
> confirm my initial findings.  The initial test did cover peak traffic,
> so I'm betting that we'll continue to see no drops.
>
> On Sat, Jul 7, 2012 at 11:45 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>> On Sat, Jul 7, 2012 at 9:28 PM, Martin Holste <mcholste at gmail.com> wrote:
>>> Thanks for the explanation.  We run about 7k rules on about 800 Mb/sec
>>> of traffic.  We had run ac full which had performed very well (no
>>> drops).  I tried out ac-bs yesterday for the latter part of the day
>>> and we got about the same performance as ac full.  The ram usage was
>>> indeed significantly less.  It decreased from about 35G with ac full
>>> to about 4G with ac-bs.  Since we couldn't get it to drop in
>>> yesterday's testing, I guess we don't know where the line is where
>>> ac-bs is inferior to ac full, but it's clear that for many, ac-bs will
>>> be very important as folks are far more likely to have 8G of ram than
>>> the 144G we have, which means that a machine with 8G should now be
>>> able to handle 1 Gb line-rate with a large rule set.
>>>
>>
>> Indeed.
>>
>> Good to hear about no drops.  Planning to run it longer and see how it
>> holds out against some peak traffic?
>>
>> You haven't seen any drops at all so far with ac full?
>>
>>> On Fri, Jul 6, 2012 at 11:33 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>>>> On Fri, Jul 6, 2012 at 9:05 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>> Can Anoop describe a bit about ac-bs?  I read the .c header and it
>>>>> interested in the design goals and overall idea.
>>>>>
>>>>
>>>> I have removed all the state transitions that are part of 0 state transition.
>>>>
>>>> For example,
>>>>
>>>> 0-state transitions :
>>>> ascii_code - transition
>>>> 0 - 1
>>>> 1 - 2
>>>> 2 - 3
>>>> 3 - 4
>>>> .....
>>>>
>>>> 5-state transitions:
>>>> 0 - 25
>>>> 1 - 30
>>>> 2 - 35
>>>> 3 - 1 /* this transition actually comes out of state-0.  We'll remove this */
>>>> .....
>>>>
>>>> This would compress the state table by quite a margin.  Leaves us with
>>>> just 1 or 2 transitions per state which is quite a reduction from 256
>>>> transitions.  I use binary search to search for the right transition.
>>>> Brought down the table sizes to a couple of mbs from a 100 odd.  These
>>>> are all my table sizes -
>>>>
>>>>
>>>> * in bytes
>>>>
>>>> "ac-bs"
>>>> 24348
>>>> 38486
>>>> 118900
>>>> 47736
>>>> 4716
>>>> 4648804
>>>> 558
>>>> 15874
>>>> 266202
>>>> 6838
>>>> 696
>>>> 692
>>>> 3982784
>>>> 10756976
>>>>
>>>> In single mode it's not as fast as "ac".  "full" mode performs well.
>>>>
>>>>> looks like I should use this one since I have a ton of RAM, but I'm
>>>>
>>>> mem consumption would be low by *quite* a margin compared to "ac".
>>>> The small table size should give good cache perf, but not as good as
>>>> "ac"
>>>>
>>>> Probably can give it a shot with both "single" and "full" mode and see
>>>> how it performs.
>>>>
>>>>> On Fri, Jul 6, 2012 at 10:23 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>>> The OISF development team is proud to announce Suricata 1.3. This
>>>>>> release is a major improvement over the previous releases with regard to
>>>>>> performance, scalability and accuracy. Also, a number of great features
>>>>>> have been added.
>>>>>>
>>>>>> Major new features:
>>>>>>
>>>>>> - TLS/SSL handshake parser and rule keywords for detecting anomolies in
>>>>>> TLS/SSL traffic
>>>>>> - HTTP user agent keyword for matching directly on User-Agent header
>>>>>> - On the fly MD5 calculation and matching for files in HTTP streams
>>>>>>
>>>>>> New / improved hardware support
>>>>>>
>>>>>> - Napatech support added
>>>>>> - Endace support improved
>>>>>> - New runmode for users of pcap wrappers (Myricom, PF_RING, others)
>>>>>>
>>>>>> Get the new release here:
>>>>>> http://www.openinfosecfoundation.org/download/suricata-1.3.tar.gz
>>>>>>
>>>>>> The configuration file has evolved but backward compatibility is
>>>>>> provided. We thus encourage you to update your suricata configuration
>>>>>> file. Upgrade guidance is provided here:
>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_12_to_Suricata_13
>>>>>>
>>>>>> Detailed list of changes:
>>>>>>
>>>>>> New features
>>>>>>
>>>>>> - TLS/SSL handshake parser, tls.subjectdn and tls.issuerdn keywords
>>>>>> (#296, contributed by Pierre Chifflier)
>>>>>> - http_user_agent keyword for matching on the HTTP User-Agent header
>>>>>> - experimental live rule reload by sending a USR2 signal (#279)
>>>>>> - AF_PACKET BPF support (#449)
>>>>>> - AF_PACKET live packet loss counters (#441)
>>>>>> - Ringbuffer and zero copy support for AF_PACKET
>>>>>> - add pcap workers runmode for use with libpcap wrappers that support
>>>>>> load balancing, such as  Napatech's or Myricom's
>>>>>> - Napatech capture card support (contributed by Randy Caldejon -- nPulse)
>>>>>> - Test mode: -T option to test the config (#271)
>>>>>> - Rule analyzer (#349)
>>>>>> - On the fly md5 checksum calculation of extracted files
>>>>>> - File extraction for HTTP POST request that do not use multipart bodies
>>>>>> - Scripts for looking up files / file md5's at Virus Total and others
>>>>>> (contributed by Martin Holste)
>>>>>> - Experimental support for matching on large lists of known file MD5
>>>>>> checksums
>>>>>> - negated filemd5 matching, allowing for md5 whitelisting
>>>>>> - Line based file log, in json format
>>>>>> - New multi pattern engine: ac-bs
>>>>>> - Basic support for including other yaml files into the main yaml
>>>>>> - Commandline options to list supported app layer protocols and keywords
>>>>>> (#344, #414)
>>>>>> - Profiling improvements, added lock profiling code
>>>>>>
>>>>>>
>>>>>> Improvements
>>>>>>
>>>>>> - Major rewrite of flow engine, improving scalability.
>>>>>> - New default runmode: "autofp" (#433)
>>>>>> - Improved scalability for Tag and Threshold subsystems
>>>>>> - Support for PF_RING 5.4 added. Many thanks to Chris Wakelin (#459).
>>>>>> - Improved Endace DAG support (#431, Jason Ish -- Endace)
>>>>>> - Split "file" output into "file-store" and "file-log" outputs
>>>>>> - Much improved file extraction
>>>>>> - Improvements to HTTP handling: multipart parsing, gzip decompression.
>>>>>> - Improved performance for file_data, http_server_body and
>>>>>> http_client_body keywords.
>>>>>> - Improved HTTP CONNECT support in libhtp (#427, Brian Rectanus -- Qualys)
>>>>>> - http_cookie keyword now also inspects "Set-Cookie" header (#479)
>>>>>> - http_raw_header keyword inspects original header line terminators (#475)
>>>>>> - deal with double encoded URI (#464)
>>>>>> - Improved http_stat_msg and http_stat_code keywords (#394)
>>>>>> - Unified yaml naming convention, including fallback support (by Nikolay
>>>>>> Denev)
>>>>>> - Made the rule keyword parser much stricter in detecting syntax errors
>>>>>> - Improved error reporting when using too long address strings (#451).
>>>>>> - Rule parser is made more strict.
>>>>>> - Byte_extract can support negative offsets now (#445).
>>>>>> - HOME_NET and EXTERNAL_NET and the other vars are now checked for
>>>>>> common errors (#454).
>>>>>> - Unified2 output overhaul, logging individual segments in more cases.
>>>>>> - signatures with depth and/or offset are now checked against packets in
>>>>>> addition to the stream (#404)
>>>>>>
>>>>>>
>>>>>> Changes since 1.3rc1
>>>>>>
>>>>>> - make live rule reloads optional and disabled by default
>>>>>> - fix a shutdown bug
>>>>>> - fix several memory leaks (#492)
>>>>>> - warn user if global and rule thresholding conflict (#455)
>>>>>> - set thread names on FreeBSD (Nikolay Denev)
>>>>>> - Fix PF_RING building on Ubuntu 12.04
>>>>>> - rule analyzer updates
>>>>>> - file inspection improvements when dealing with limits (#493)
>>>>>>
>>>>>>
>>>>>> Credits
>>>>>>
>>>>>>   Brian Rectanus -- Qualys
>>>>>>   Randy Caldejon -- nPulse
>>>>>>   Pierre Chifflier
>>>>>>   Coverity
>>>>>>   Nikolay Denev
>>>>>>   Jason Ish -- Endace
>>>>>>   Martin Holste
>>>>>>   Napatech
>>>>>>   Rmkml
>>>>>>   Michel Sarborde
>>>>>>   Chris Wakelin
>>>>>>   Joshua White
>>>>>>
>>>>>> Known issues & missing features
>>>>>>
>>>>>> If you encounter issues, please let us know! As always, we are doing our
>>>>>> best to make you aware of continuing development and items within the
>>>>>> engine that are not yet complete or optimal.  With this in mind, please
>>>>>> notice the list we have included of known items we are working on.
>>>>>>
>>>>>> See http://redmine.openinfosecfoundation.org/projects/suricata/issues
>>>>>> for an up to date list and to report new issues. See
>>>>>> http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues
>>>>>> for a discussion and time line for the major issues.
>>>>>>
>>>>>> --
>>>>>> ---------------------------------------------
>>>>>> Victor Julien
>>>>>> http://www.inliniac.net/
>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>> ---------------------------------------------
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Oisf-users mailing list
>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>
>>>>
>>>>
>>>> --
>>>> Anoop Saldanha
>>
>>
>>
>> --
>> Anoop Saldanha

-- 
Anoop Saldanha