[Oisf-devel] [Oisf-users] Suricata 1.3 Available!

Fri Jul 6 16:33:01 UTC 2012

On Fri, Jul 6, 2012 at 9:05 PM, Martin Holste <mcholste at gmail.com> wrote:
> Can Anoop describe a bit about ac-bs?  I read the .c header and it
> interested in the design goals and overall idea.
>

I have removed all the state transitions that are part of 0 state transition.

For example,

0-state transitions :
ascii_code - transition
0 - 1
1 - 2
2 - 3
3 - 4
.....

5-state transitions:
0 - 25
1 - 30
2 - 35
3 - 1 /* this transition actually comes out of state-0.  We'll remove this */
.....

This would compress the state table by quite a margin.  Leaves us with
just 1 or 2 transitions per state which is quite a reduction from 256
transitions.  I use binary search to search for the right transition.
Brought down the table sizes to a couple of mbs from a 100 odd.  These
are all my table sizes -

* in bytes

"ac-bs"
24348
38486
118900
47736
4716
4648804
558
15874
266202
6838
696
692
3982784
10756976

In single mode it's not as fast as "ac".  "full" mode performs well.

> looks like I should use this one since I have a ton of RAM, but I'm

mem consumption would be low by *quite* a margin compared to "ac".
The small table size should give good cache perf, but not as good as
"ac"

Probably can give it a shot with both "single" and "full" mode and see
how it performs.

> On Fri, Jul 6, 2012 at 10:23 AM, Victor Julien <victor at inliniac.net> wrote:
>> The OISF development team is proud to announce Suricata 1.3. This
>> release is a major improvement over the previous releases with regard to
>> performance, scalability and accuracy. Also, a number of great features
>> have been added.
>>
>> Major new features:
>>
>> - TLS/SSL handshake parser and rule keywords for detecting anomolies in
>> TLS/SSL traffic
>> - HTTP user agent keyword for matching directly on User-Agent header
>> - On the fly MD5 calculation and matching for files in HTTP streams
>>
>> New / improved hardware support
>>
>> - Napatech support added
>> - Endace support improved
>> - New runmode for users of pcap wrappers (Myricom, PF_RING, others)
>>
>> Get the new release here:
>> http://www.openinfosecfoundation.org/download/suricata-1.3.tar.gz
>>
>> The configuration file has evolved but backward compatibility is
>> provided. We thus encourage you to update your suricata configuration
>> file. Upgrade guidance is provided here:
>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_12_to_Suricata_13
>>
>> Detailed list of changes:
>>
>> New features
>>
>> - TLS/SSL handshake parser, tls.subjectdn and tls.issuerdn keywords
>> (#296, contributed by Pierre Chifflier)
>> - http_user_agent keyword for matching on the HTTP User-Agent header
>> - experimental live rule reload by sending a USR2 signal (#279)
>> - AF_PACKET BPF support (#449)
>> - AF_PACKET live packet loss counters (#441)
>> - Ringbuffer and zero copy support for AF_PACKET
>> - add pcap workers runmode for use with libpcap wrappers that support
>> load balancing, such as  Napatech's or Myricom's
>> - Napatech capture card support (contributed by Randy Caldejon -- nPulse)
>> - Test mode: -T option to test the config (#271)
>> - Rule analyzer (#349)
>> - On the fly md5 checksum calculation of extracted files
>> - File extraction for HTTP POST request that do not use multipart bodies
>> - Scripts for looking up files / file md5's at Virus Total and others
>> (contributed by Martin Holste)
>> - Experimental support for matching on large lists of known file MD5
>> checksums
>> - negated filemd5 matching, allowing for md5 whitelisting
>> - Line based file log, in json format
>> - New multi pattern engine: ac-bs
>> - Basic support for including other yaml files into the main yaml
>> - Commandline options to list supported app layer protocols and keywords
>> (#344, #414)
>> - Profiling improvements, added lock profiling code
>>
>>
>> Improvements
>>
>> - Major rewrite of flow engine, improving scalability.
>> - New default runmode: "autofp" (#433)
>> - Improved scalability for Tag and Threshold subsystems
>> - Support for PF_RING 5.4 added. Many thanks to Chris Wakelin (#459).
>> - Improved Endace DAG support (#431, Jason Ish -- Endace)
>> - Split "file" output into "file-store" and "file-log" outputs
>> - Much improved file extraction
>> - Improvements to HTTP handling: multipart parsing, gzip decompression.
>> - Improved performance for file_data, http_server_body and
>> http_client_body keywords.
>> - Improved HTTP CONNECT support in libhtp (#427, Brian Rectanus -- Qualys)
>> - http_cookie keyword now also inspects "Set-Cookie" header (#479)
>> - http_raw_header keyword inspects original header line terminators (#475)
>> - deal with double encoded URI (#464)
>> - Improved http_stat_msg and http_stat_code keywords (#394)
>> - Unified yaml naming convention, including fallback support (by Nikolay
>> Denev)
>> - Made the rule keyword parser much stricter in detecting syntax errors
>> - Improved error reporting when using too long address strings (#451).
>> - Rule parser is made more strict.
>> - Byte_extract can support negative offsets now (#445).
>> - HOME_NET and EXTERNAL_NET and the other vars are now checked for
>> common errors (#454).
>> - Unified2 output overhaul, logging individual segments in more cases.
>> - signatures with depth and/or offset are now checked against packets in
>> addition to the stream (#404)
>>
>>
>> Changes since 1.3rc1
>>
>> - make live rule reloads optional and disabled by default
>> - fix a shutdown bug
>> - fix several memory leaks (#492)
>> - warn user if global and rule thresholding conflict (#455)
>> - set thread names on FreeBSD (Nikolay Denev)
>> - Fix PF_RING building on Ubuntu 12.04
>> - rule analyzer updates
>> - file inspection improvements when dealing with limits (#493)
>>
>>
>> Credits
>>
>>   Brian Rectanus -- Qualys
>>   Randy Caldejon -- nPulse
>>   Pierre Chifflier
>>   Coverity
>>   Nikolay Denev
>>   Jason Ish -- Endace
>>   Martin Holste
>>   Napatech
>>   Rmkml
>>   Michel Sarborde
>>   Chris Wakelin
>>   Joshua White
>>
>> Known issues & missing features
>>
>> If you encounter issues, please let us know! As always, we are doing our
>> best to make you aware of continuing development and items within the
>> engine that are not yet complete or optimal.  With this in mind, please
>> notice the list we have included of known items we are working on.
>>
>> See http://redmine.openinfosecfoundation.org/projects/suricata/issues
>> for an up to date list and to report new issues. See
>> http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues
>> for a discussion and time line for the major issues.
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel

-- 
Anoop Saldanha