[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-2.0.2-85-g0ac94ef
OISF Git
noreply at openinfosecfoundation.org
Mon Jul 28 16:15:26 UTC 2014
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via 0ac94ef777d222bb530e6f314b1092b2b4b24744 (commit)
via e0841218f0b609cbd91b8e758c1c196f5d3a2bc5 (commit)
via 46cee88ef8fbd44a383066405ae3397fde939f86 (commit)
via f1185d051c210ca0daacdddbe865a51af24f4ea3 (commit)
via 9f55ca00576b9eb04feb9267f08814af7acf3772 (commit)
via fc6ad56944d801772e4384e853c3c8077def545c (commit)
via e6ed6731b1447be1e187128abb5908e8668b4595 (commit)
via 8c231702d9885b4821e9e1eb2c0cc7567ca7d4fb (commit)
via 6f9a2fcd58f4b10c43cc265a49d145dca285370f (commit)
via f4dfaacff36bcd1553291abf6deb2fa4fb44da7c (commit)
via eaf01449e341ac53f633e26677cc1ad03d6bd4a5 (commit)
via db15339f479796a77655b9174a482e2063f80f34 (commit)
via 07b7f66f3ccc3ab7627769c872fdbc3bfb179f57 (commit)
via 3bb0ccba9824f54c983a2058c1af69c15c43ad60 (commit)
via d19a15701c996dce47ecf72d1aa90db701dc86da (commit)
via 98c88d5170ab08facc8de12ce78c6eb6be681296 (commit)
via de034f186745ee6ef1b53cf34e28a9a8c0403ac6 (commit)
via bd490736c2fc673e3c88ffe256aa43a89d941b48 (commit)
via e6ee5feaba02639b3f93bc2931d6878503899d14 (commit)
via 52b0ec027ef39fb2ed8d3353d23cc7f491640d9b (commit)
via 4aff4c650ffac6f3700cfa62349ea3d672c9dbd4 (commit)
via 7acea2c66d4933847021b188e994915a762c4556 (commit)
via c66a29b67d07420d28e08dd2ad6e352793736fde (commit)
via f828793f8f6f3dfeee5c320e8c000ef27ddc72d2 (commit)
via 672f6523a71b199778986adffddf5e7b1005aacb (commit)
via fddeca8aae7242ed6066959b3b1afcd829fa1690 (commit)
via ec7d446f16fefdb609b0f79478eeed4099a7c813 (commit)
via 3c7af020673da35b9baf1a7fa4020710ca7539bb (commit)
via c7ebfd1b68f839ab6ec79a832587433f747ebe0d (commit)
via e30c083cffee57b34465fabd0c91762997694e5b (commit)
via 115ad1e81f8a765c05117b35052d6cce884469b1 (commit)
via a52a4ae9d480a504dd1373e96f0fa1cc41ad3fb2 (commit)
via f476732139917ca6400370974a56dba7226a3a79 (commit)
via f26f82e9a601c956556576f7921d1874abf9eb72 (commit)
via 94cb52897b27b8a5d4de9ae878cd72f8f52dbcd1 (commit)
via e892d99827be4422dbd3c42847551eeafac80240 (commit)
from fdd407751e3ebfdd17750ff414dd6fa2f7ce9a9f (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 0ac94ef777d222bb530e6f314b1092b2b4b24744
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jul 21 15:13:42 2014 +0200
flow-recycler: support multiple instances
Use new management API to run the flow recycler.
Make number of threads configurable:
flow:
memcap: 64mb
hash-size: 65536
prealloc: 10000
emergency-recovery: 30
managers: 2
recyclers: 2
This sets up 2 flow recyclers.
commit e0841218f0b609cbd91b8e758c1c196f5d3a2bc5
Author: Victor Julien <victor at inliniac.net>
Date: Sat Jul 12 13:47:33 2014 +0200
flow-manager: support multiple instances
Use new management API to run the flow manager.
Support multiple flow managers, where each of them works with it's
own part of the flow hash.
Make number of threads configurable:
flow:
memcap: 64mb
hash-size: 65536
prealloc: 10000
emergency-recovery: 30
managers: 2
This sets up 2 flow managers.
Handle misc tasks only in instance 1: Handle defrag hash timeout
handing, host hash timeout handling and flow spare queue updating
only from the first instance.
commit 46cee88ef8fbd44a383066405ae3397fde939f86
Author: Victor Julien <victor at inliniac.net>
Date: Wed Jul 16 09:59:48 2014 +0200
threads: add management API
Currently management threads do their own thread setup and handling. This
patch introduces a new way of handling management threads.
Functionality that needs to run as a management thread can now register
itself as a regular 'thread module' (TmModule), where the 'Management'
callback is registered.
commit f1185d051c210ca0daacdddbe865a51af24f4ea3
Author: Victor Julien <victor at inliniac.net>
Date: Wed Jun 4 13:37:02 2014 +0200
flow id: quick and dirty first stab at a flow id
Add a 'flow_id' that is the same for all records produced for packets
belonging to the same flow.
This patch simply takes the flow's memory address.
commit 9f55ca00576b9eb04feb9267f08814af7acf3772
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 23 14:54:05 2014 +0200
flow: add flow_end_flags field, add logging
The flow end flags field is filled by the flow manager or the flow
hash (in case of forced timeout of a flow) to record the timeout
conditions in the flow:
- emergency mode
- state
- reason (timed out or forced)
Add logging to the flow logger.
commit fc6ad56944d801772e4384e853c3c8077def545c
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 23 14:51:44 2014 +0200
flow: move FlowGetFlowState
Move FlowGetFlowState to flow-private.h so that all parts of the flow
engine can use it.
commit e6ed6731b1447be1e187128abb5908e8668b4595
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 23 11:49:37 2014 +0200
flow log: log TCP state
Log the TCP state at timeout.
commit 8c231702d9885b4821e9e1eb2c0cc7567ca7d4fb
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 22 12:53:51 2014 +0200
flow-recycler: speed up flow-recycler shutdown
Thread was killed by the generic TmThreadKillThreads instead of
the FlowKillFlowRecyclerThread. The latter wakes the thread up, so
that shutdown is quite a bit faster.
commit 6f9a2fcd58f4b10c43cc265a49d145dca285370f
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 22 12:39:12 2014 +0200
flow: log individual tcp flags
Log the tcp flags.
commit f4dfaacff36bcd1553291abf6deb2fa4fb44da7c
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 22 12:37:27 2014 +0200
netflow: log individual tcp flags
Log the tcp flags.
commit eaf01449e341ac53f633e26677cc1ad03d6bd4a5
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 22 12:36:45 2014 +0200
json: add tcp flags to json utility function
Turns a flags bitfield into a set of json bools.
commit db15339f479796a77655b9174a482e2063f80f34
Author: Victor Julien <victor at inliniac.net>
Date: Wed May 21 16:54:18 2014 +0200
netflow-json: initial version
Initial version of netflow module, a flow logger that logs each
direction in a completely separate record (line).
commit 07b7f66f3ccc3ab7627769c872fdbc3bfb179f57
Author: Victor Julien <victor at inliniac.net>
Date: Wed May 21 15:37:10 2014 +0200
flow-log: log TCP flags per direction
In addition to flags for the entire session, also log out TCP flags
for both directions separately.
commit 3bb0ccba9824f54c983a2058c1af69c15c43ad60
Author: Victor Julien <victor at inliniac.net>
Date: Wed May 21 14:29:15 2014 +0200
stream: track TCP flags per stream direction
For netflow logging track TCP flags per stream direction. As the struct
had no more space left without expanding it, the flags and wscale
fields are now compressed.
commit d19a15701c996dce47ecf72d1aa90db701dc86da
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 9 14:56:55 2014 +0200
flow: init logger thread data for decoders
Initialize the output flow api thread data for the decoder threads.
commit 98c88d5170ab08facc8de12ce78c6eb6be681296
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 9 15:10:54 2014 +0200
decode: pass ThreadVars to DecodeThreadVarsFree
Flow output thread data deinit function which will be called from
DecodeThreadVarsFree will need it.
commit de034f186745ee6ef1b53cf34e28a9a8c0403ac6
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 9 14:37:07 2014 +0200
flow: prepare flow forced reuse logging
Most flows are marked for clean up by the flow manager, which then
passes them to the recycler. The recycler logs and cleans up. However,
under resource stress conditions, the packet threads can recycle
existing flow directly. So here the recycler has no role to play, as
the flow is immediately used.
For this reason, the packet threads need to be able to invoke the
flow logger directly.
The flow logging thread ctx will stored in the DecodeThreadVars
stucture. Therefore, this patch makes the DecodeThreadVars an argument
to FlowHandlePacket.
commit bd490736c2fc673e3c88ffe256aa43a89d941b48
Author: Victor Julien <victor at inliniac.net>
Date: Wed May 7 09:16:46 2014 +0200
flow: take flow pkt & byte count out of debug
Until now the flow packet and byte counters were only available in
DEBUG mode. For logging purposes they are now available always.
commit e6ee5feaba02639b3f93bc2931d6878503899d14
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 20:48:47 2014 +0200
flow: don't BUG_ON if no loggers are enabled
API is always called, even if no loggers are enabled. Don't abort()
in this case.
commit 52b0ec027ef39fb2ed8d3353d23cc7f491640d9b
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 20:37:41 2014 +0200
flow: clean up recycle queue at shutdown
Mostly for tests that don't start the recycler thread, make sure
all flows are cleaned up.
commit 4aff4c650ffac6f3700cfa62349ea3d672c9dbd4
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 13:47:18 2014 +0200
flow unittest: update flow manager unit test
Test now tests a different queue.
commit 7acea2c66d4933847021b188e994915a762c4556
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 12:15:39 2014 +0200
flow: track lastts in struct timeval
Track full timestamp for lastts in flows to be able to log it.
commit c66a29b67d07420d28e08dd2ad6e352793736fde
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 11:54:28 2014 +0200
flow: track bytes per direction
Track bytes in both flow directions for logging purposes.
commit f828793f8f6f3dfeee5c320e8c000ef27ddc72d2
Author: Victor Julien <victor at inliniac.net>
Date: Tue May 6 10:11:49 2014 +0200
flow log: log start/end times
Log time of first packet (flow creation) and of the last packet.
commit 672f6523a71b199778986adffddf5e7b1005aacb
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 2 09:54:16 2014 +0200
flow-log: log TCP flags seen
Log TCP flags seen during the life time of a flow/session.
commit fddeca8aae7242ed6066959b3b1afcd829fa1690
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 2 09:45:01 2014 +0200
tcp: track TCP packet flags per session
For logging out in flow logging.
commit ec7d446f16fefdb609b0f79478eeed4099a7c813
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 2 08:25:22 2014 +0200
flow-log: log pkts, bytes
Only in DEBUG currently.
commit 3c7af020673da35b9baf1a7fa4020710ca7539bb
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 1 17:31:31 2014 +0200
flow-json-log: stub
Stub for JSON flow logger.
commit c7ebfd1b68f839ab6ec79a832587433f747ebe0d
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 1 17:30:32 2014 +0200
flow: flow log threading setup
Set up threading for the flow logger.
commit e30c083cffee57b34465fabd0c91762997694e5b
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 1 14:18:59 2014 +0200
flow log: call logger from recycler
Call the flow logger API from the recycler thread, so that timed
out flows are logged.
commit 115ad1e81f8a765c05117b35052d6cce884469b1
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 1 13:31:10 2014 +0200
flow: output api stub
Basic output API for flow logging.
commit a52a4ae9d480a504dd1373e96f0fa1cc41ad3fb2
Author: Victor Julien <victor at inliniac.net>
Date: Thu May 1 12:50:02 2014 +0200
flow recycler: unix socket support
Support starting and shutting down the flow recycler thread in the
unix socket runmode.
commit f476732139917ca6400370974a56dba7226a3a79
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 30 18:49:39 2014 +0200
flow recycler: shutdown
Only shut down when all flows in the recycle queue have been processed.
commit f26f82e9a601c956556576f7921d1874abf9eb72
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 30 13:57:17 2014 +0200
flow: move flow cleanup to new 'recycler'
Move Flow clean up from the flow manager to the new flow recycler.
commit 94cb52897b27b8a5d4de9ae878cd72f8f52dbcd1
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 30 13:37:30 2014 +0200
flow: introduce FlowRecycler stub
FlowRecycler thread stub. Start/stop code.
commit e892d99827be4422dbd3c42847551eeafac80240
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 30 12:30:30 2014 +0200
flow: new flow queue: flow_recycle_q
This queue will be used by the FlowManager to pass timed out flows
to another thread that will do the actual cleanup.
-----------------------------------------------------------------------
Summary of changes:
src/Makefile.am | 3 +
src/alert-debuglog.c | 4 +-
src/app-layer-htp.c | 4 +-
src/decode-icmpv4.c | 2 +-
src/decode-icmpv6.c | 2 +-
src/decode-sctp.c | 2 +-
src/decode-tcp.c | 2 +-
src/decode-udp.c | 4 +-
src/decode.c | 15 +-
src/decode.h | 5 +-
src/flow-hash.c | 48 +-
src/flow-hash.h | 2 +-
src/flow-manager.c | 533 +++++++++++++++++-----
src/flow-manager.h | 12 +
src/flow-private.h | 21 +
src/flow-util.h | 15 +-
src/flow.c | 23 +-
src/flow.h | 23 +-
src/{output-packet.c => output-flow.c} | 88 ++--
src/{output-tx.h => output-flow.h} | 23 +-
src/output-json-flow.c | 482 +++++++++++++++++++
src/{detect-msg.h => output-json-flow.h} | 12 +-
src/output-json-netflow.c | 466 +++++++++++++++++++
src/{output-json-ssh.h => output-json-netflow.h} | 10 +-
src/output-json.c | 36 ++
src/output-json.h | 2 +
src/output.c | 70 +++
src/output.h | 8 +
src/runmode-unix-socket.c | 16 +-
src/runmodes.c | 6 +
src/source-af-packet.c | 2 +-
src/source-erf-dag.c | 2 +-
src/source-erf-file.c | 2 +-
src/source-ipfw.c | 2 +-
src/source-mpipe.c | 2 +-
src/source-napatech.c | 2 +-
src/source-nflog.c | 2 +-
src/source-nfq.c | 2 +-
src/source-pcap-file.c | 2 +-
src/source-pcap.c | 2 +-
src/source-pfring.c | 2 +-
src/stream-tcp-private.h | 10 +-
src/stream-tcp.c | 18 +
src/suricata.c | 15 +
src/tm-modules.c | 4 +
src/tm-modules.h | 3 +
src/tm-threads-common.h | 6 +
src/tm-threads.c | 111 +++++
src/tm-threads.h | 6 +
src/util-unittest-helper.c | 2 +-
suricata.yaml.in | 2 +
51 files changed, 1900 insertions(+), 238 deletions(-)
copy src/{output-packet.c => output-flow.c} (69%)
copy src/{output-tx.h => output-flow.h} (60%)
create mode 100644 src/output-json-flow.c
copy src/{detect-msg.h => output-json-flow.h} (80%)
create mode 100644 src/output-json-netflow.c
copy src/{output-json-ssh.h => output-json-netflow.h} (79%)
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list