[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-3.0.1-303-gf947539
OISF Git
noreply at openinfosecfoundation.org
Fri May 20 12:40:00 UTC 2016
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via f947539d7971d83f9b4cef0aeeb473ff3946a21d (commit)
via 49612128f3b24b4224d58842cff1540fc9a42b08 (commit)
via 88f5d7d16605d43afb44d134facf26239af48b45 (commit)
via ff05fb760b76d2ad3ea7b407af394ad6d13aa0dd (commit)
via a40f08a21335e25d8b7aa1fe5182ec2687954b7d (commit)
via 8035d834678cfe82de597b9b0bc72a7f87b23df3 (commit)
via 7fea0ec6f9c704a27db0c78c039bfd191f79d906 (commit)
via 876b356bbe335a50aec1c9e1222fa9584fba51b4 (commit)
via c2d0d93806c6a684ced8492e86157a4f28e36bf4 (commit)
via f5c20191672de24d55852eb48dcd6a33524de4b7 (commit)
via 234aefdff9c87623ac2f1a7f60436c79ac03561b (commit)
via fa902abedf92e82768519c72ce3e180ace0784da (commit)
via c7bde9dff66bd9722b7d4e8c80abd4b29fc5571b (commit)
via 7fa963718fa630320cb426702304c4bf8cdab5a8 (commit)
via 5f84b55d98ca36b199d1c6b84dc89a5461687df8 (commit)
via b797fd926c0aa080c6ace55f25a7f0dff12b4bb7 (commit)
via 9500d12c9f8be4edef896decab896b69be88a3ea (commit)
via bae1b03cf5ccab79c0d8af896c7c2200688edc00 (commit)
via d094039600ac832325bacb014583cca6a6eaa2f4 (commit)
via 27adbfa86828d21d06ff9fc3a999270da7bed00e (commit)
via 5f400785c850887b9e2856f2afbf89dfba7b53e2 (commit)
via f77bc5195cb7c81214a57f3c0e06993923f82b3a (commit)
via e43ce0a9ecc32fa1e574fc3c9e1bfc246a45bc01 (commit)
via e836a750c8d333c5d14b70e9621c8b69b39ad32c (commit)
via feafc838db4cac7002580d8f72e9cf4f742db03c (commit)
via 24a2f51569e59b5e9506fe746cb263a5e0e1a460 (commit)
via 6fb808fc1aebc1ce4b2d7f601f529395f6d98fe6 (commit)
via 46e55f1e346101ed1998ad45681e288a857a7766 (commit)
via 81b2984c4e4b244afd87502bec94209a3d0d8094 (commit)
via 78ecfe8780ca3d06bba9318c42775f51e132e45f (commit)
via 61ce05e7ed8a8ef7b02ff57f2e7c6281003ef62c (commit)
via 52d500c670a343a1503dc959c2b087979eb8346f (commit)
via 408948815f4cc4a6b7a9fbcbc7bc9bbe073a30f3 (commit)
via 6f560144c1b97906ad41f621cadc08f4c99dff89 (commit)
via 2f0e0f17dbb4f289f045ab38cf13dc2ef209a148 (commit)
via ae7aae81dc25271f30d4c26f0588f65ad8f44c09 (commit)
via a81766c046ef463c3a1c527770702552765f9843 (commit)
from a13df67864d05ba56a9751e7310001296f7c6d59 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit f947539d7971d83f9b4cef0aeeb473ff3946a21d
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 20 10:41:45 2016 +0200
af-packet: CentOS6 build fixes
commit 49612128f3b24b4224d58842cff1540fc9a42b08
Author: Eric Leblond <eric at regit.org>
Date: Tue Apr 26 19:55:51 2016 +0200
af-packet: use time() instead of GetTime()
As we only use the second we don't need GetTime() which is slower
and get us milliseconds.
commit 88f5d7d16605d43afb44d134facf26239af48b45
Author: Eric Leblond <eric at regit.org>
Date: Tue Apr 26 19:46:19 2016 +0200
af-packet: print errno on mmap error
commit ff05fb760b76d2ad3ea7b407af394ad6d13aa0dd
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 22 19:50:10 2016 +0200
af-packet: fix some typos in yaml
commit a40f08a21335e25d8b7aa1fe5182ec2687954b7d
Author: Eric Leblond <eric at regit.org>
Date: Sat Apr 23 00:30:39 2016 +0200
af-packet: ask for hardware timestamp
commit 8035d834678cfe82de597b9b0bc72a7f87b23df3
Author: Eric Leblond <eric at regit.org>
Date: Mon Apr 18 14:24:08 2016 +0200
af-packet: make mmap options parsing conditional
Only parse them if mmap is activated.
commit 7fea0ec6f9c704a27db0c78c039bfd191f79d906
Author: Eric Leblond <eric at regit.org>
Date: Tue Apr 12 11:46:43 2016 +0200
af-packet: reset stats at start of capture
We can loose packets during setup because we are reading nothing.
So it is logical to discard the counter at start of capture to
start from a clean state. This means we don't need to account the
drop at start. But the stats call that will reset the drop counts
will also return and reset the packets count. So we need to know
how many packets we really have. This is in fact the number of
packets coming from the stats call minus the number of discarded
packets and the drop count. All the other packets will have to be
read.
commit 876b356bbe335a50aec1c9e1222fa9584fba51b4
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 14:01:09 2016 +0200
af-packet: use mmap capture by default
Update the code to use mmap capture by default even in unset in
configuration file. mmap capture is now be turned off by using
explicitely 'use-mmap: no' in configuration.
commit c2d0d93806c6a684ced8492e86157a4f28e36bf4
Author: Eric Leblond <eric at regit.org>
Date: Sat Apr 9 16:07:23 2016 +0200
af-packet: detect availability of tpacket_v3
If TPACKET_V3 is not defined then it is not available and we should
not build anything related to tpacket_v3. This will allow us to
activate it dy default and fallback to v2 if not available.
commit f5c20191672de24d55852eb48dcd6a33524de4b7
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 8 17:05:55 2016 +0200
af-packet: add option to use memory locked mmap
commit 234aefdff9c87623ac2f1a7f60436c79ac03561b
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 8 15:29:18 2016 +0200
af-packet: configurable tpacket_v3 block timeout
Block timeout defines the maximum filling duration of a block.
commit fa902abedf92e82768519c72ce3e180ace0784da
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 8 10:03:52 2016 +0200
af-packet: configurable tpacket_v3 block size
It is used to set the block size in tpacket_v3. It will allow user
to tune the capture depending on his bandwidth.
Default block size value has been updated to a bigger value to
allow more efficient wlak on block.
commit c7bde9dff66bd9722b7d4e8c80abd4b29fc5571b
Author: Eric Leblond <eric at regit.org>
Date: Thu Apr 7 21:58:48 2016 +0200
af-packet: put ring setup in a separate function
commit 7fa963718fa630320cb426702304c4bf8cdab5a8
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 22:52:37 2016 +0200
af-packet: pack AFPPeer structure
commit 5f84b55d98ca36b199d1c6b84dc89a5461687df8
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 22 22:15:53 2016 +0200
af-packet: AFPWalkBlock error handling
Error handling was not done. The implementation is making the
choice to consider we must detroy the socket in case of parsing
error. The same was done for tpacket_v2.
commit b797fd926c0aa080c6ace55f25a7f0dff12b4bb7
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 22:33:22 2016 +0200
af-packet: continuing cleaning and hole hunting
Suppress useless fields in AFPThreadVars. This patch also get rid
of bytes counter as it was only used to display a message at exit.
Information on livedev and on packet counters are enough.
commit 9500d12c9f8be4edef896decab896b69be88a3ea
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 19:46:41 2016 +0200
af-packet: cleaning and hole hunting
Reorder fields in AFPThreadVars and suppress some that were not
used elsewhere than in the initialization.
commit bae1b03cf5ccab79c0d8af896c7c2200688edc00
Author: Eric Leblond <eric at regit.org>
Date: Mon Oct 28 17:19:31 2013 +0100
af-packet: tpacket_v3 implementation
This patch adds a basic implementation of AF_PACKET tpacket v3. It
is basic in the way it is only working for 'workers' runnning mode.
If not in 'workers' mode there is a fallback to tpacket_v2. Feature
is activated via tpacket-v3 option in the af-packet section of
Suricata YAML.
commit d094039600ac832325bacb014583cca6a6eaa2f4
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 13:52:24 2016 +0200
af-packet: remove useless code
No need for cooked header in the case of mmap capture.
commit 27adbfa86828d21d06ff9fc3a999270da7bed00e
Author: Eric Leblond <eric at regit.org>
Date: Fri Apr 1 08:48:31 2016 +0200
af-packet: micro optimization
commit 5f400785c850887b9e2856f2afbf89dfba7b53e2
Author: Eric Leblond <eric at regit.org>
Date: Thu Jan 3 20:29:54 2013 +0100
af-packet: avoid test for each packet
commit f77bc5195cb7c81214a57f3c0e06993923f82b3a
Author: Justin Viiret <justin.viiret at intel.com>
Date: Fri May 20 13:31:05 2016 +1000
spm: handle null ptrs in destroy funcs gracefully
This will handle minimal DetectEngineCtx structures (used in delayed
detect mode) safely, since they don't get SPM global contexts allocated.
Also added BUG_ON checks for valid spm_table entries.
commit e43ce0a9ecc32fa1e574fc3c9e1bfc246a45bc01
Author: Victor Julien <victor at inliniac.net>
Date: Mon Nov 30 21:21:50 2015 +0100
file: switch to streaming buffer API
Make the file storage use the streaming buffer API.
As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
commit e836a750c8d333c5d14b70e9621c8b69b39ad32c
Author: Victor Julien <victor at inliniac.net>
Date: Mon Nov 16 10:05:56 2015 +0100
http: improve body inspection
Enforce inspect window also in IDS mode. Try always to get at least
'inspect win' worth of data. In case there is more new data, take
some of the old data as well to make sure there is always some overlap.
This unifies IDS and IPS modes, the only difference left is the start
of inspection. IDS waits until min_size is available, IPS starts right
away.
commit feafc838db4cac7002580d8f72e9cf4f742db03c
Author: Victor Julien <victor at inliniac.net>
Date: Sun Nov 15 13:21:59 2015 +0100
http: make htpstate cfg ptr const
commit 24a2f51569e59b5e9506fe746cb263a5e0e1a460
Author: Victor Julien <victor at inliniac.net>
Date: Sun Nov 15 13:20:14 2015 +0100
http: move body settings into per dir struct
commit 6fb808fc1aebc1ce4b2d7f601f529395f6d98fe6
Author: Victor Julien <victor at inliniac.net>
Date: Sat Nov 14 00:14:02 2015 +0100
http: add per direction config for body parsing
The HTPCfgDir structure is meant to contain config for per direction
body parsing parameters.
This patch stores the streaming API config.
commit 46e55f1e346101ed1998ad45681e288a857a7766
Author: Victor Julien <victor at inliniac.net>
Date: Thu Nov 12 00:19:52 2015 +0100
http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.
In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.
Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
commit 81b2984c4e4b244afd87502bec94209a3d0d8094
Author: Victor Julien <victor at inliniac.net>
Date: Sun Nov 8 18:30:05 2015 +0100
streaming: buffer API
Add a new API to store data from streaming sources, like HTTP body
processing or TCP data.
Currently most of the code uses a pattern of list of data chunks
(e.g. TcpSegment) that is reassembled into a large buffer on-demand.
The Streaming Buffer API changes the logic to store the data in
reassembled form from the start, with the segments/chunks pointing
to the reassembled data.
The main buffer storing the data slides forward, automatically or
manually. The *NoTrack calls allows for a segmentless mode of
operation.
This approach has two main advantages:
1. accessing the reassembled data is virtually cost-free
2. reduction of allocations and memory management
commit 78ecfe8780ca3d06bba9318c42775f51e132e45f
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 17 18:34:55 2016 +0200
autofp: update queue handlers
Now that the flow lookup is done in the worker threads the flow
queue handlers running after the capture thread(s) no longer have
access to the flow. This limits the options of how flow balancing
can be done.
This patch removes all code that is now useless. The only 2 methods
that still make sense are 'hash' and 'ippair'.
commit 61ce05e7ed8a8ef7b02ff57f2e7c6281003ef62c
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 17 18:05:26 2016 +0200
flow: remove dead code
commit 52d500c670a343a1503dc959c2b087979eb8346f
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 19 18:06:32 2016 +0200
flowworker: initial support
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
commit 408948815f4cc4a6b7a9fbcbc7bc9bbe073a30f3
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 16 23:06:33 2016 +0200
detect: simplify flow locking
To simplify locking, move all locking out of the individual detect
code. Instead at the start of detection lock the flow, and at the
end of detection unlock it.
The lua code can be called without a lock still (from the output
code paths), so still pass around a lock hint to take care of this.
commit 6f560144c1b97906ad41f621cadc08f4c99dff89
Author: Victor Julien <victor at inliniac.net>
Date: Mon Apr 18 15:09:13 2016 +0200
time: improve offline time handling
When we run on live traffic, time handling is simple. Packets have a
timestamp set by the capture method. Management threads can simply
use 'gettimeofday' to know the current time. There should never be
any serious gap between the two or major differnces between the
threads.
In offline mode, things are dramatically different. Here we try to keep
the time from the pcap, which means that if the packets are recorded in
2011 the log output should also reflect this. Multiple issues:
1. merged pcaps might have huge time jumps or time going backward
2. slowly recorded pcaps may be processed much faster than their
'realtime'
3. management threads need a concept of what the 'current' time is for
enforcing timeouts
4. due to (1) individual threads may have very different views on what
the current time is. E.g. T1 processed packet 1 with TS X, while T2
at the very same time processes packet 2 with TS X+100000s.
The changes in flow handling make the problems worse. The capture thread
no longer handles the flow lookup, while it did set the global 'time'.
This meant that a thread may be working on Packet 1 with TS 1, while the
capture thread already saw packet 2 with TS 10000. Management threads
would take TS 10000 as the 'current time', considering a flow created by
the first thread as timed out immediately.
This was less of a problem before the flow changes as the capture thread
would also create a flow reference for a packet, meaning the flow
couldn't time out as easily. Packets in the queues between capture
thread and workers would all hold such references.
The patch updates the time handling to be as follows.
In offline mode we keep the timestamp per thread. If a management thread
needs current time, it will get the minimum of the threads' values. This
is to avoid the problem that T2s time value might already trigger a flow
timeout as the flow lastts + 100000s is almost certainly meaning the
flow would be considered timed out.
commit 2f0e0f17dbb4f289f045ab38cf13dc2ef209a148
Author: Victor Julien <victor at inliniac.net>
Date: Fri Apr 15 17:08:50 2016 +0200
flow: move flow handling into worker threads
Instead of handling the packet update during flow lookup, handle
it in the stream/detect threads. This lowers the load of the
capture thread(s) in autofp mode.
The decoders now set a flag in the packet if the packet needs a
flow lookup. Then the workers will take care of this. The decoders
also already calculate the raw flow hash value. This is so that
this value can be used in flow balancing in autofp.
Because the flow lookup/creation is now done in the worker threads,
the flow balancing can no longer use the flow. It's not yet
available. Autofp load balancing uses raw hash values instead.
In the same line, move UDP AppLayer out of the DecodeUDP module,
and also into the stream/detect threads.
Handle TCP session reuse inside the flow engine itself. If a looked up
flow matches the packet, but is a TCP stream starter, check if the
ssn needs to be reused. If that is the case handle it within the
lookup function. Simplies the locking and removes potential race
conditions.
commit ae7aae81dc25271f30d4c26f0588f65ad8f44c09
Author: Victor Julien <victor at inliniac.net>
Date: Tue Mar 17 12:48:14 2015 +0100
flow: get flow reference during lookup
Update Flow lookup functions to get a flow reference during lookup.
This reference is set under the FlowBucket lock.
This paves the way to not getting a flow lock during lookups.
commit a81766c046ef463c3a1c527770702552765f9843
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 16 21:30:32 2016 +0200
detect: split detect entry into flow/noflow
This is a preparation for flow locking updates.
-----------------------------------------------------------------------
Summary of changes:
configure.ac | 9 +
src/Makefile.am | 2 +
src/app-layer-htp-body.c | 92 ++---
src/app-layer-htp-body.h | 2 +-
src/app-layer-htp-file.c | 92 ++---
src/app-layer-htp.c | 167 +++-----
src/app-layer-htp.h | 27 +-
src/app-layer-smtp.c | 119 +++---
src/app-layer-smtp.h | 3 +
src/app-layer.c | 5 +-
src/decode-icmpv4.c | 2 +-
src/decode-icmpv6.c | 3 +-
src/decode-ipv4.c | 8 +-
src/decode-sctp.c | 3 +-
src/decode-tcp.c | 3 +-
src/decode-udp.c | 10 +-
src/decode.h | 8 +
src/detect-engine-alert.c | 10 +-
src/detect-engine-content-inspection.c | 6 +-
src/detect-engine-file.c | 6 +-
src/detect-engine-filedata-smtp.c | 79 +---
src/detect-engine-hcbd.c | 105 +++--
src/detect-engine-hsbd.c | 210 +++-------
src/detect-engine-state.c | 18 -
src/detect-engine-tag.c | 4 -
src/detect-engine.c | 20 +-
src/detect-filemagic.c | 87 ++--
src/detect-filesize.c | 14 +-
src/detect-filestore.c | 6 -
src/detect-flowint.c | 6 -
src/detect-flowvar.c | 4 -
src/detect-http-client-body.c | 8 +-
src/detect-lua.c | 20 +-
src/detect-lua.h | 4 +-
src/detect-pcre.c | 8 +-
src/detect.c | 118 +++---
src/detect.h | 4 +-
src/flow-bit.c | 4 -
src/flow-hash.c | 174 +++-----
src/flow-hash.h | 2 +-
src/flow-util.h | 6 -
src/flow-worker.c | 233 +++++++++++
src/{util-spm-hs.h => flow-worker.h} | 17 +-
src/flow.c | 56 +--
src/flow.h | 51 +--
src/log-file.c | 2 +-
src/log-filestore.c | 11 +-
src/output-filedata.c | 94 ++---
src/output-filedata.h | 2 +-
src/output-json-file.c | 2 +-
src/output-streaming.c | 12 +-
src/runmode-af-packet.c | 72 +++-
src/runmode-erf-file.c | 28 +-
src/runmode-pcap-file.c | 28 +-
src/runmode-tile.c | 13 +-
src/runmode-unittests.c | 4 +
src/source-af-packet.c | 700 +++++++++++++++++++++++--------
src/source-af-packet.h | 21 +-
src/source-pcap-file.c | 4 -
src/stream-tcp.c | 193 +--------
src/stream-tcp.h | 4 -
src/suricata.c | 3 +
src/tm-modules.c | 1 +
src/tm-threads-common.h | 1 +
src/tm-threads.c | 69 +++-
src/tm-threads.h | 3 +
src/tmqh-flow.c | 173 +-------
src/tmqh-flow.h | 4 -
src/util-file.c | 238 +++--------
src/util-file.h | 33 +-
src/util-lua-common.c | 2 +-
src/util-lua-http.c | 11 +-
src/util-runmodes.c | 72 +---
src/util-spm.c | 21 +-
src/util-streaming-buffer.c | 725 +++++++++++++++++++++++++++++++++
src/util-streaming-buffer.h | 140 +++++++
src/util-time.c | 59 ++-
src/util-time.h | 5 +-
src/util-unittest-helper.c | 6 +-
src/util-validate.h | 2 -
suricata.yaml.in | 21 +-
81 files changed, 2633 insertions(+), 1981 deletions(-)
create mode 100644 src/flow-worker.c
copy src/{util-spm-hs.h => flow-worker.h} (74%)
create mode 100644 src/util-streaming-buffer.c
create mode 100644 src/util-streaming-buffer.h
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list