[Discussion] Suricata 2.0 Available!

Tue Mar 25 10:41:13 UTC 2014

The OISF development team is proud to announce Suricata 2.0. This
release is a major improvement over the previous releases with regard to
performance, scalability and accuracy. Also, a number of great features
have been added.

The biggest new features of this release are the addition of "Eve", our
all JSON output for events: alerts, HTTP, DNS, SSH, TLS and (extracted)
files; much improved VLAN handling; a detectionless 'NSM' runmode; much
improved CUDA performance.

The Eve log allows for easy 3rd party integration. It has been created
with Logstash in mind specifically and we have a quick setup guide here
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

*Download*

Get the new release here:
http://www.openinfosecfoundation.org/download/suricata-2.0.tar.gz

*Notable new features, improvements and changes*

- Eve log, all JSON event output for alerts, HTTP, DNS, SSH, TLS and
files. Written by Tom Decanio of nPulse Technologies
- NSM runmode, where detection engine is disabled. Development supported
by nPulse Technologies
- Various scalability improvements, clean ups and fixes by Ken Steel of
Tilera
- Add --set commandline option to override any YAML option, by Jason Ish
of Emulex
- Several fixes and improvements of AF_PACKET and PF_RING
- ICMPv6 handling improvements by Jason Ish of Emulex
- Alerting over PCIe bus (Tilera only), by Ken Steel of Tilera
- Feature #792: DNS parser, logger and keyword support, funded by
Emerging Threats
- Feature #234: add option disable/enable individual app layer protocol
inspection modules
- Feature #417: ip fragmentation time out feature in yaml
- Feature #1009: Yaml file inclusion support
- Feature #478: XFF (X-Forwarded-For) support in Unified2
- Feature #602: availability for http.log output - identical to apache
log format
- Feature #813: VLAN flow support
- Feature #901: VLAN defrag support
- Features #814, #953, #1102: QinQ VLAN handling
- Feature #751: Add invalid packet counter
- Feature #944: detect nic offloading
- Feature #956: Implement IPv6 reject
- Feature #775: libhtp 0.5.x support
- Feature #470: Deflate support for HTTP response bodies
- Feature #593: Lua flow vars and flow ints support
- Feature #983: Provide rule support for specifying icmpv4 and icmpv6
- Feature #1008: Optionally have http_uri buffer start with uri path for
use in proxied environments
- Feature #1032: profiling: per keyword stats
- Feature #878: add storage api

*Upgrading*

The configuration file has evolved but backward compatibility is
provided. We thus encourage you to update your suricata configuration
file. Upgrade guidance is provided here:
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_14_to_Suricata_20

*Special thanks*

We'd like to thank the following people and corporations for their
contributions and feedback:

- Ken Steele, Tilera
- Tom DeCanio, nPulse
- Jason Ish, Endace / Emulex
- Duarte Silva
- Giuseppe Longo
- Ignacio Sanchez
- Florian Westphal
- Nelson Escobar, Myricom
- Christian Kreibich, Lastline
- Phil Schroeder, Emerging Threats
- Luca Deri & Alfredo Cardigliano, ntop
- Will Metcalf, Emerging Threats
- Ivan Ristic, Qualys
- Chris Wakelin
- Francis Trudeau, Emerging Threats
- Rmkml
- Laszlo Madarassy
- Alessandro Guido
- Amin Latifi
- Darrell Enns
- Paolo Dangeli
- Victor Serbu
- Jack Flemming
- Mark Ashley
- Marc-Andre Heroux
- Alessandro Guido
- Petr Chmelar
- Coverity

*Known issues & missing features*

If you encounter issues, please let us know! As always, we are doing our
best to make you aware of continuing development and items within the
engine that are not yet complete or optimal. With this in mind, please
notice the list we have included of known items we are working on. See
http://redmine.openinfosecfoundation.org/projects/suricata/issues for an
up to date list and to report new issues. See
http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues
for a discussion and time line for the major issues.

*About Suricata*

Suricata is a high performance Network IDS, IPS and Network Security
Monitoring engine. Open Source and owned by a community run non-profit
foundation, the Open Information Security Foundation (OISF). Suricata is
developed by the OISF, its supporting vendors and the community.
-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

-------------- next part --------------
2.0beta1
Bug #709: tls - append: yes missing from suricata.yaml
Bug #710: tls - certs directory
Bug #711: make install-full does not create certs directory
Bug #753: TX handling improvement
Bug #769: Be sure to always apply verdict to NFQ packet
Bug #771: curious ip proto break fast.log
Bug #774: negative depth and offset:0 fire ? Dup of #770 agianst dev(2.0) branch
Bug #777: Generate error if bpf is used in IPS mode
Bug #779: sig id parsing
Bug #791: pcre relative match with content FN (master)
Bug #794: stream: SACK allocs should adhere to memcap
Bug #802: deadlock in flowvar capture code (master)
Bug #803: improve error message if directory is passed instead of yaml
Bug #817: file_data relative positive and negative match at same offset problem (master)
Bug #819: af-packet ips mode rule processing bug
Bug #827: bytetest, bytejump and byteextract negative offset failure
Bug #830: OS X unix socket build failure (master)
Bug #847: FP on IP frag and sig use udp port 0 (master)
Bug #874: Segfault with git master
Bug #881: OpenBSD buildbot broken
Bug #883: Flowbit check with content doesn't match consistently (master)
Feature #470: gzip extension support incomplete
Feature #593: luajit: per flow vars and ints
Feature #729: Cuda redesigned
Feature #775: libhtp 0.5 support
Feature #782: Unix Socket being enabled during configure with --enable-unix-socket
Feature #792: DNS log feature to be introduced
Feature #796: stream: deal with multiple different SYN/ACK's better
Feature #797: Dynamic test on flag usage
Feature #804: Randomize stream chunk size
Feature #875: libhtp 0.5.4
Optimization #519: Introduce per stream thread ssn pool
Optimization #564: pattern id: rewrite pattern id assignment
Optimization #565: pattern id unused if uricontent and content+http_uri are mixed
Optimization #718: "pass" IP-only rules should bypass detection engine after matching
Optimization #832: clean up packet action macros
2.0beta2
Bug #463: Suricata not fire on http reply detect if request are not http
Bug #640: app-layer-event:http.host_header_ambiguous set when it shouldn't
Bug #714: some logs not created in daemon mode
Bug #810: Alerts on http traffic storing the wrong packet as the IDS event payload
Bug #815: address parsing with negation
Bug #820: several issues found by clang 3.2
Bug #837: Af-packet statistics inconsistent under very high traffic
Bug #882: MpmACCudaRegister shouldn't call PatternMatchDefaultMatcher
Bug #887: http.log printing unknown hostname most of the time
Bug #890: af-packet segv
Bug #892: detect-engine.profile - custom - does not err out in incorrect toclient/srv values - suricata.yaml
Bug #895: response: rst packet bug
Bug #896: pfring dna mode issue
Bug #897: make install-full fails if wget is missing
Bug #903: libhtp valgrind warning
Bug #907: icmp_seq and icmp_id keyword with icmpv6 traffic (master)
Bug #910: make check fails w/o sudo/root privs
Bug #911: HUP signal
Bug #912: 1.4.3: Unit test in util-debug.c: line too long.
Bug #914: Having a high number of pickup queues (216+) makes suricata crash
Bug #915: 1.4.3: log-pcap.c: crash on printing a null filename
Bug #917: 1.4.5: decode-ipv6.c: void function cannot return value
Bug #920: Suricata failed to parse address
Bug #922: trackers value in suricata.yaml
Bug #925: prealloc-sessions value bigger than allowed in suricata.yaml
Bug #926: prealloc host value in suricata.yaml
Bug #927: detect-thread-ratio given a non numeric value in suricata.yaml
Bug #928: Max number of threads
Bug #932: wrong IP version - on stacked layers
Bug #939: thread name buffers are sized inconsistently
Bug #943: pfring: see if we can report that the module is not loaded
Bug #948: apple ppc64 build broken: thread-local storage not supported for this target
Bug #958: SSL parsing issue (master)
Bug #963: XFF compile failure on OSX
Bug #964: Modify negated content handling
Bug #967: threshold rule clobbers suppress rules
Bug #968: unified2 not logging tagged packets
Bug #970: AC memory read error
Bug #973: Use different ids for content patterns which are the same, but one of them has a fast_pattern chop set on it.
Bug #976: ip_rep supplying different no of alerts for 2 different but semantically similar rules
Bug #979: clean up app layer protocol detection memory
Bug #982: http events missing
Bug #987: default config generates error(s)
Bug #988: suricata don't exit in live mode
Bug #989: Segfault in HTPStateGetTxCnt after a few minutes
Bug #991: threshold mem leak
Bug #994: valgrind warnings in unittests
Bug #995: tag keyword: tagging sessions per time is broken
Bug #998: rule reload triggers app-layer-event FP's
Bug #999: delayed detect inits thresholds before de_ctx
Bug #1003: Segmentation fault
Bug #1023: block rule reloads during delayed detect init
Bug #1026: pfring: update configure to link with -lrt
Bug #1031: Fix IPv6 stream pseudo packets
Bug #1035: http uri/query normalization normalizes 'plus' sign to space
Bug #1042: Can't match "emailAddress" field in tls.subject and tls.issuerdn
Bug #1061: Multiple flowbit set in one rule
Feature #234: add option disable/enable individual app layer protocol inspection modules
Feature #417: ip fragmentation time out feature in yaml
Feature #478: XFF (X-Forwarded-For)
Feature #602: availability for http.log output - identical to apache log format
Feature #622: Specify number of pf_ring/af_packet receive threads on the command line
Feature #727: Explore the support for negated alprotos in sigs.
Feature #746: Decoding API modification
Feature #751: Add invalid packet counter
Feature #752: Improve checksum detection algorithm
Feature #789: Clean-up start and stop code
Feature #813: VLAN flow support
Feature #878: add storage api
Feature #901: VLAN defrag support
Feature #904: store tx id when generating an alert
Feature #940: randomize http body chunks sizes
Feature #944: detect nic offloading
Feature #956: Implement IPv6 reject
Feature #957: reject: iface setup
Feature #959: Move post config initialisation code to PostConfLoadedSetup
Feature #981: Update all switch case fall throughs with comments on false throughs
Feature #983: Provide rule support for specifying icmpv4 and icmpv6.
Feature #986: set htp request and response size limits
Feature #1008: Optionally have http_uri buffer start with uri path for use in proxied environments
Feature #1009: Yaml file inclusion support
Feature #1028: How to identify which rule called the lua script
Feature #1032: profiling: per keyword stats
Optimization #583: improve Packet_ structure layout
Optimization #1018: clean up counters api
Optimization #1041: remove mkinstalldirs from git
2.0rc1
Bug #839: http events alert multiple times
Bug #954: VLAN decoder stats with AF Packet get written to the first thread only - stats.log
Bug #980: memory leak in http buffers at shutdown
Bug #1066: logger API's for packet based logging and tx based logging
Bug #1068: format string issues with size_t + qa not catching them
Bug #1072: Segmentation fault in 2.0beta2: Custom HTTP log segmentation fault
Bug #1073: radix tree lookups are not thread safe
Bug #1075: CUDA 5.5 doesn't compile with 2.0 beta 2
Bug #1079: Err loading rules with variables that contain negated content.
Bug #1080: segfault - 2.0dev (rev 6e389a1)
Bug #1081: 100% CPU utilization with suricata 2.0 beta2+
Bug #1082: af-packet vlan handling is broken
Bug #1097: tls: negated match too much
Bug #1103: stats.log not incrementing decoder.ipv4/6 stats when reading in QinQ packets
Bug #1104: vlan tagged fragmentation
Bug #1106: Git compile fails on Ubuntu Lucid
Bug #1107: flow timeout causes decoders to run on pseudo packets
Feature #424: App layer registration cleanup - Support specifying same alproto names in rules for different ip protocols
Feature #542: TLS JSON output
Feature #597: case insensitive fileext match
Feature #772: JSON output for alerts
Feature #814: QinQ tag flow support
Feature #894: clean up output
Feature #921: Override conf parameters
Feature #1007: united output
Feature #1040: Suricata should compile with -Werror
Feature #1067: memcap for http inside suricata
Feature #1086: dns memcap
Feature #1093: stream: configurable segment pools
Feature #1102: Add a decoder.QinQ stats in stats.log
Feature #1105: Detect icmpv6 on ipv4
2.0rc2
Bug #611: fp: rule with ports matching on portless proto
Bug #985: default config generates rule warnings and errors
Bug #1021: 1.4.6: conf_filename not checked before use
Bug #1089: SMTP: move depends on uninitialised value
Bug #1090: FTP: Memory Leak
Bug #1091: TLS-Handshake: Uninitialized value
Bug #1092: HTTP: Memory Leak
Bug #1108: suricata.yaml config parameter - segfault
Bug #1109: PF_RING vlan handling
Bug #1110: Can have the same Pattern ID (pid) for the same pattern but different case flags
Bug #1111: capture stats at exit incorrect
Bug #1112: tls-events.rules file missing
Bug #1115: nfq: exit stats not working
Bug #1120: segv with pfring/afpacket and eve-log enabled
Bug #1121: crash in eve-log
Bug #1124: ipfw build broken
Feature #952: Add VLAN tag ID to all outputs
Feature #953: Add QinQ tag ID to all outputs
Feature #1012: Introduce SSH log
Feature #1118: app-layer protocols http memcap - info in verbose mode (-v)
Feature #1119: restore SSH protocol detection and parser
2.0rc3
Bug #1127: logstash & suricata parsing issue
Bug #1128: Segmentation fault - live rule reload
Bug #1129: pfring cluster & ring initialization
Bug #1130: af-packet flow balancing problems
Bug #1131: eve-log: missing user agent reported inconsistently
Bug #1133: eve-log: http depends on regular http log
Bug #1135: 2.0rc2 release doesn't set optimization flag on GCC
Bug #1138: alert fastlog drop info missing
2.0
Bug #1151: tls.store not working when a TLS filter keyword is used