[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-3.2.1-313-gd31cb08
OISF Git
noreply at openinfosecfoundation.org
Thu Apr 27 14:12:55 UTC 2017
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via d31cb083e910d779279e47ffbb32d5da096220fe (commit)
via eb5857b68aecbe27beee4703b7a39c8aece734b3 (commit)
via 15dcac92f267a013dbaecb82bab6fa98b7320b37 (commit)
via cf7f819888afd1cefa9a4965b81c1a1f2837123c (commit)
via 1bbf5553186c7d38b678f93db24773bd14ff84cf (commit)
via d1b7a8390581f0ed7a4f221e6a2af86a6df954a5 (commit)
via a0fad6bb7fd57080f35ac500cf623cc21bcb2f92 (commit)
via d304be5bc3fff43ef27eddec56569ab6b512689a (commit)
via bc1698cfbe347ad0f5c714c96756b3dd1f9e8922 (commit)
via 53b21e5ee16a9478b8952df4bfee4f042f5c20b4 (commit)
via 8d2f3b46e6e6888ba493292fe3e594cf997c0569 (commit)
via 8d18be1fdb38f6988603b03217ffd26f2eaa6e7f (commit)
via 1c46af477e948d185f32cd7da5fd46eb3b7e9860 (commit)
via a744d00f459508dbeecae8eb43c8cce0bc8b50c8 (commit)
via e3bd5f371dc77a8d8ab9a2d29a8e2ad996be1aaf (commit)
via 6d562f3b5ee71dc782dd28af1ebe25aaf05fffa3 (commit)
via 358e41b935a7921c829abaf230e0e4c08eec799c (commit)
via 9f4884a1323580f97f145fd69817e39714c0d9d4 (commit)
via 5c31f22e09f241d1ca16722a07318295620d985d (commit)
via 5a210984d50c2630ea8e64e9efb8eb763e01782d (commit)
via bea2b2c00cde2130f104edfbbede0a03187b4cd9 (commit)
via 34f7cb2b553bd06bb04f383c44f3ba055c917124 (commit)
via aba9cd7d0285f63874b6e66f314fc99ab26555d5 (commit)
via 2b433fab53298e2e4d652ecc991863cefd1a9aef (commit)
via 606f515fe923c6dfd2ff8365898f782f5b2a111f (commit)
via 314516ffe23fedf2cb05f9e8ed53038fd7a687af (commit)
via 89af036336a0443f3793e2dbd7a824dbde4b8a15 (commit)
via 2f77302eeb39503881b1588052e08f02cd3b0b21 (commit)
via d6d7f65050ae081aa1207fda2d9d62a83f82181e (commit)
via 7bddd0e168ef6efef7caec5874c1b179295ece61 (commit)
via 6fefe7019629f62e97d2fa71a02318c4633ad234 (commit)
via 422095668e94cee289825ec9cfbf9cb05bc5008a (commit)
via 79389558ac2c7b3386d514b5f2601d5159f90d82 (commit)
via a995734b3a394b3b46c3810b595c22b64d9cb7ed (commit)
via ee00a6f2ec9d91d343a15f7de487dd58a03edbd4 (commit)
via e1aba7d6c2525b720fd8bdc9c189ef328aaf2113 (commit)
via 61c35d3c3971f1f2f236fc8e942944706b1d2815 (commit)
via f49150ddb9abc70f210bd183027034855050d740 (commit)
via 6f76cbb870495507f02d91bde4603b45c691367b (commit)
via 04b24cf24e670df92f3d24501aa90e79fc258e30 (commit)
via 55e19bfb89d9a5d55367a4535c470fd25a7a407a (commit)
via bf3f3ce6b27ee4210e09e93072c119d4caf71725 (commit)
via de4f4e23a08498004f8b1474b5d2e80e7d3df7d7 (commit)
via 7c56c9ada0a9db67c6ca0fc8736a1cc46cbf52d5 (commit)
via 89d0267df2ea50ed23432795ad6eee43ee08aad7 (commit)
via 0c1ec17c923d852c90cf495d4ed55135aad26efa (commit)
via 69519bda48f24b84b7d621f759d0d3045660225b (commit)
via b099008b943c1c95cbc1a0db60ea35037609c72a (commit)
via 0ef46a8fd2a87d31c3f3439451df8a0b4173c3fa (commit)
via 149e3240602e070d88c833088a5bf045d3b349a3 (commit)
via 2d223b69cd74f5afbed24c309cb355a4de65ba97 (commit)
via 971ab18b95fad189f863ef7f201c458e43540139 (commit)
via 564c0bd2c1100c1bded16f62c44a1f4059a342e9 (commit)
via 0bff0de516dd029e0baf304b7600a8aeae5b1c25 (commit)
via 807312320f46fb1c5d7c86e927063d2cc223598d (commit)
via bd821f57f2c147e078126d83bd4998c971d3fbee (commit)
via c2a5b9c393168272a6759b94ef9fc111dd0af919 (commit)
via dd2b8bb2986b78892282a4c36ba4025966f7ca16 (commit)
via 8924653cd4cc1f1f8aa51d7ecf1d0f702d83c1b8 (commit)
via 91f57200c7c4d5f32961aa497ed436163a8f8e41 (commit)
via f02dc377efd7f5b1aa8a961f18cbe0ef49bc395e (commit)
via 8c36e52d9315e120f41e10439aad03cc5dd41f40 (commit)
via 5ee36a0c8bb205ae5d8f169af875fbb5ca0db80e (commit)
via fa2a832022829f64424418b9305c8123b5468c8e (commit)
via 187e2381c89f887a7a16f24d0c28204dcb55f6a0 (commit)
via 9bd11bcda5079fbb34727605c2f23ce605ebbbce (commit)
via b3e9d397719a972d16b4a4eb2c6fa4a456a4d281 (commit)
via bbb0df14d2697960c531ea400737665438bee081 (commit)
via 8c9f521707715110c8ffe9fe0c82102a49314ec0 (commit)
via 3fa2e8689cb86d7d6aae94aead2da7c3ffbd3194 (commit)
via d789dc7e6de057d756b992c562b7f1583cc70c05 (commit)
via 9e1470d81c79677b689a0fc4b570d20232953dad (commit)
via 3411697106296da912601005ff8cb963f6cdea66 (commit)
via 245a89b7e74cfa4d60ab4f93d9708dd1af7d803f (commit)
via 595f6d1f266650f9f7a16e740b90a08c6d7a79cf (commit)
via d96cbddbe45a01cc1a0c534551c46735cf2b9d05 (commit)
via 46febef7cd1ceda05b9ab1bf0dbc94e50bc4ce17 (commit)
via 9d5bbc3af6009697848268b6a6221ffadf037a8f (commit)
via 1d1176b0d549e7f13e3b873eb6b396e9f845358b (commit)
via 5714129e32f4dd26a309c08826037a53dd7a84cc (commit)
via 88cfb99910f3b70ea565fcf9635f86afdaf46f1e (commit)
from 753997310919cc041ac2b4058a7a46037bc2d613 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit d31cb083e910d779279e47ffbb32d5da096220fe
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 19 15:57:31 2017 +0200
detect: update tests that mix state/stream inspect
commit eb5857b68aecbe27beee4703b7a39c8aece734b3
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 19 15:57:08 2017 +0200
unittests: add/improve helpers for stream/flow
commit 15dcac92f267a013dbaecb82bab6fa98b7320b37
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 11:48:06 2017 +0200
http_header: enable trailer prefilter engines
Now that the mpm engines run only for the proper 'progress'
value, the trailing headers need their own engine registration.
commit cf7f819888afd1cefa9a4965b81c1a1f2837123c
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 18 09:38:22 2017 +0200
state: check progress before calling engine
Make sure progress of an inspect engine is available.
commit 1bbf5553186c7d38b678f93db24773bd14ff84cf
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 11 15:24:49 2017 +0200
detect: improve stateful detection
Now that MPM runs when the TX progress is right, stateful detection
operates differently.
Changes:
1. raw stream inspection is now also an inspect engine
Since this engine doesn't take the transactions into account, it
could potentially run multiple times on the same data. To avoid
this, basic result caching is in place.
2. the engines are sorted by progress, but the 'MPM' engine is first
even if the progress is higher
If MPM flags a rule to be inspected, the inspect engine for that
buffer runs first. If this step fails, the rule is no longer
evaluated. No state is stored.
commit d1b7a8390581f0ed7a4f221e6a2af86a6df954a5
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 11 18:15:16 2017 +0200
detect: change mask logic
Previously the MPM/Prefilter engines would suggest the same rule
candidates multiple times.
For example, while processing the request body, the http headers
would be inspected by MPM multiple times.
The mask check was one way to quickly decide which rules could be
skipped.
Now that the MPM engines normally return a rule just once, this
mask check no longer makes sense. If the rule meets the ip/port/
direction based conditions, it needs to be evaluated if the MPM
said so. Even if not all conditions are yet true.
WIP disable mask as it no longer makes sense
WIP redo mask match
commit a0fad6bb7fd57080f35ac500cf623cc21bcb2f92
Author: Victor Julien <victor at inliniac.net>
Date: Mon Apr 10 20:42:25 2017 +0200
mpm: run engines as few times as possible
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.
For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.
This was obviously inefficent, so this patch changes the logic.
The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.
Implemented for HTTP, TLS and SSH.
commit d304be5bc3fff43ef27eddec56569ab6b512689a
Author: Victor Julien <victor at inliniac.net>
Date: Mon Apr 17 10:59:32 2017 +0200
detect: register progress in inspect engines
Register required progress so we can stop inspecting as soon
as the progress isn't far enough yet.
commit bc1698cfbe347ad0f5c714c96756b3dd1f9e8922
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 18 09:38:48 2017 +0200
detect-state: don't use casts to uint
commit 53b21e5ee16a9478b8952df4bfee4f042f5c20b4
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 16:15:51 2017 +0200
http_uri: unittest cleanup
commit 8d2f3b46e6e6888ba493292fe3e594cf997c0569
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 13:03:42 2017 +0200
http_header: add another trailer test
commit 8d18be1fdb38f6988603b03217ffd26f2eaa6e7f
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 11:30:01 2017 +0200
http_header (trailer) test cleanup
commit 1c46af477e948d185f32cd7da5fd46eb3b7e9860
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 16:07:01 2017 +0200
ssh: fix test
commit a744d00f459508dbeecae8eb43c8cce0bc8b50c8
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 20 16:04:34 2017 +0200
ssh: fix banner state setting
commit e3bd5f371dc77a8d8ab9a2d29a8e2ad996be1aaf
Author: Victor Julien <victor at inliniac.net>
Date: Mon Apr 17 09:53:50 2017 +0200
detect: more detailed state profiling
commit 6d562f3b5ee71dc782dd28af1ebe25aaf05fffa3
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 12 14:31:25 2017 +0200
app-layer: set stream-depth after stream init
commit 358e41b935a7921c829abaf230e0e4c08eec799c
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 11 09:58:40 2017 +0200
detect: clean up stateful detect
commit 9f4884a1323580f97f145fd69817e39714c0d9d4
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 19 15:47:09 2017 +0200
stream: reduce scope of new ssn func
commit 5c31f22e09f241d1ca16722a07318295620d985d
Author: Victor Julien <victor at inliniac.net>
Date: Tue Apr 4 16:16:17 2017 +0200
autotools: add src/tests to extra dist
commit 5a210984d50c2630ea8e64e9efb8eb763e01782d
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 23:28:51 2017 +0100
stream: move inline tests
commit bea2b2c00cde2130f104edfbbede0a03187b4cd9
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 23:20:44 2017 +0100
stream: list management cleanups
commit 34f7cb2b553bd06bb04f383c44f3ba055c917124
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 12:52:56 2017 +0100
stream: debug improvements
commit aba9cd7d0285f63874b6e66f314fc99ab26555d5
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 12:52:34 2017 +0100
stream inspection: add debug counters
commit 2b433fab53298e2e4d652ecc991863cefd1a9aef
Author: Victor Julien <victor at inliniac.net>
Date: Tue Mar 7 22:41:23 2017 +0100
stream: pack config struct
commit 606f515fe923c6dfd2ff8365898f782f5b2a111f
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 09:54:56 2017 +0100
stream: enforce gap earlier in app reassembly
commit 314516ffe23fedf2cb05f9e8ed53038fd7a687af
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 09:54:16 2017 +0100
stream: don't call app reassembly if disable flag set
commit 89af036336a0443f3793e2dbd7a824dbde4b8a15
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 09:53:54 2017 +0100
stream: app-layer micro optimizations
commit 2f77302eeb39503881b1588052e08f02cd3b0b21
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 00:12:38 2017 +0100
stream: raw reassembly explicit disable raw handling
commit d6d7f65050ae081aa1207fda2d9d62a83f82181e
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 00:12:12 2017 +0100
stream: mpm inspect micro optimizations
commit 7bddd0e168ef6efef7caec5874c1b179295ece61
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 1 16:51:22 2017 +0100
stream: improve --disable-detection GAP handling
commit 6fefe7019629f62e97d2fa71a02318c4633ad234
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 1 15:34:14 2017 +0100
stream: remove unused StreamTcpGetStreamSize function
commit 422095668e94cee289825ec9cfbf9cb05bc5008a
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 1 14:50:20 2017 +0100
stream: optimize session pruning
commit 79389558ac2c7b3386d514b5f2601d5159f90d82
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 1 14:04:29 2017 +0100
doc: update for stream changes
commit a995734b3a394b3b46c3810b595c22b64d9cb7ed
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 1 11:24:13 2017 +0100
yaml: sync with new stream engine
commit ee00a6f2ec9d91d343a15f7de487dd58a03edbd4
Author: Victor Julien <victor at inliniac.net>
Date: Tue Feb 28 23:13:00 2017 +0100
stream: validate code
commit e1aba7d6c2525b720fd8bdc9c189ef328aaf2113
Author: Victor Julien <victor at inliniac.net>
Date: Mon Feb 27 23:28:46 2017 +0100
detect: only do flow dependent cleanup if a flow is present
commit 61c35d3c3971f1f2f236fc8e942944706b1d2815
Author: Victor Julien <victor at inliniac.net>
Date: Mon Feb 27 23:26:37 2017 +0100
detect: make SigMatchSignatures void
None of the callers cared for it's retval, so get rid of it.
commit f49150ddb9abc70f210bd183027034855050d740
Author: Victor Julien <victor at inliniac.net>
Date: Mon Feb 27 23:14:39 2017 +0100
detect: turn single detect flag into bool
commit 6f76cbb870495507f02d91bde4603b45c691367b
Author: Victor Julien <victor at inliniac.net>
Date: Mon Feb 27 23:12:09 2017 +0100
detect: remove unused detect flag
commit 04b24cf24e670df92f3d24501aa90e79fc258e30
Author: Victor Julien <victor at inliniac.net>
Date: Tue Feb 28 12:44:02 2017 +0100
stream: improve needs reassembly code
commit 55e19bfb89d9a5d55367a4535c470fd25a7a407a
Author: Victor Julien <victor at inliniac.net>
Date: Tue Feb 28 11:23:27 2017 +0100
stream: more aggressive StreamReassembleRawHasDataReady
commit bf3f3ce6b27ee4210e09e93072c119d4caf71725
Author: Victor Julien <victor at inliniac.net>
Date: Sun Feb 26 14:19:43 2017 +0100
app-layer: change logic of setting 'no reassembly'
Instead of killing all reassembly instantly do things slightly more
gracefully:
1. disable app-layer reassembly immediately
2. flag raw reassembly not to accept new data
This will allow the current data to be inspected still.
After detect as run the raw reassembly will be fully disabled and
thus all reassembly will be as well.
commit de4f4e23a08498004f8b1474b5d2e80e7d3df7d7
Author: Victor Julien <victor at inliniac.net>
Date: Sun Feb 26 14:06:05 2017 +0100
stream: new depth / disable raw logic
Depth reach sets NOREASSEMBLY after detect.
No new raw sets NORAW after detect.
commit 7c56c9ada0a9db67c6ca0fc8736a1cc46cbf52d5
Author: Victor Julien <victor at inliniac.net>
Date: Sat Feb 25 13:33:40 2017 +0100
stream: allow raw reassembly catch up
If raw reassembly falls behind, for example because no raw mpm is
active, then we need to sync up to the app progress if that is
available, or to the generic tcp tracking otherwise.
commit 89d0267df2ea50ed23432795ad6eee43ee08aad7
Author: Victor Julien <victor at inliniac.net>
Date: Sat Feb 25 13:33:15 2017 +0100
stream: detect stream GAP also during reassembly
commit 0c1ec17c923d852c90cf495d4ed55135aad26efa
Author: Victor Julien <victor at inliniac.net>
Date: Fri Feb 24 21:05:43 2017 +0100
debug-validation: add stream checks
commit 69519bda48f24b84b7d621f759d0d3045660225b
Author: Victor Julien <victor at inliniac.net>
Date: Thu Feb 23 13:09:46 2017 +0100
stream: StreamTcpReassembleRawCheckLimit cleanup
commit b099008b943c1c95cbc1a0db60ea35037609c72a
Author: Victor Julien <victor at inliniac.net>
Date: Wed Feb 22 11:14:02 2017 +0100
stream: handle no stream scanning case
Now that detect moves the raw progress forward, it's important
to deal with the case where detect don't consider raw inspection.
If no 'stream' rules are active, disable raw. For this the disable
raw flag is now per stream.
commit 0ef46a8fd2a87d31c3f3439451df8a0b4173c3fa
Author: Victor Julien <victor at inliniac.net>
Date: Mon Feb 20 11:04:29 2017 +0100
stream: raw content inspection inline mode
Implement the inline mode for raw content inspection. Packets
are leading, and when a packet's payload has been added to the
stream, the packet is inspected in the context of the stream.
Reassembly will return a buffer with the packet data with older
data in front of it and after it, if available.
commit 149e3240602e070d88c833088a5bf045d3b349a3
Author: Victor Julien <victor at inliniac.net>
Date: Sun Feb 19 12:32:23 2017 +0100
flow/stream: reduce/disable pseudo packet injections
At flow timeout, we no longer need to first run reassembly in
one dir, then inspection in the other. We can do both in single
packet now.
Disable pseudo packets when receiving stream end packets. Instead
call the app-layer parser in the packet direction for stream end
packets and flow end packets.
These changes in handling of those stream end packets make the
pseudo packets unnecessary.
commit 2d223b69cd74f5afbed24c309cb355a4de65ba97
Author: Victor Julien <victor at inliniac.net>
Date: Sun Feb 19 00:54:45 2017 +0100
stream: set 'trigger raw' per direction
commit 971ab18b95fad189f863ef7f201c458e43540139
Author: Victor Julien <victor at inliniac.net>
Date: Fri Feb 17 17:59:43 2017 +0100
detect / stream: new 'raw' stream inspection
Remove the 'StreamMsg' approach from the engine. In this approach the
stream engine would create a list of chunks for inspection by the
detection engine. There were several issues:
1. the messages had a fixed size, so blocks of data bigger than ~4k
would be cut into multiple messages
2. it lead to lots of data copying and unnecessary memory use
3. the StreamMsgs used a central pool
The Stream engine switched over to the streaming buffer API, which
means that the reassembled data is always available. This made the
StreamMsg approach even clunkier.
The new approach exposes the streaming buffer data to the detection
engine. It has to pay attention to an important issue though: packet
loss. The data may have gaps. The streaming buffer API tracks the
blocks of continuous data.
To access the data for inspection a callback approach is used. The
'StreamReassembleRaw' function is called with a callback and data.
This way it runs the MPM and individual rule inspection code. At
the end of each detection run the stream engine is notified that it
can move forward it's 'progress'.
commit 564c0bd2c1100c1bded16f62c44a1f4059a342e9
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 13:26:54 2017 +0100
stream: constify StreamTcpReassembleRawCheckLimit
commit 0bff0de516dd029e0baf304b7600a8aeae5b1c25
Author: Victor Julien <victor at inliniac.net>
Date: Sat May 7 17:24:32 2016 +0200
unittests: fail if TCP memory still in use
abort() so test can be analyzed.
commit 807312320f46fb1c5d7c86e927063d2cc223598d
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 6 19:45:30 2016 +0200
stream-tcp: implement thread pool for segments
Config option:
stream:
reassembly:
segment-prealloc: 2048
commit bd821f57f2c147e078126d83bd4998c971d3fbee
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 6 17:12:42 2016 +0200
stream: implement memory handling functions
commit c2a5b9c393168272a6759b94ef9fc111dd0af919
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 17:21:28 2016 +0200
stream: use static instead of dynamic streaming buffer structure
commit dd2b8bb2986b78892282a4c36ba4025966f7ca16
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 17:20:11 2016 +0200
stream: test cleanups and fixes
commit 8924653cd4cc1f1f8aa51d7ecf1d0f702d83c1b8
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 12:50:32 2017 +0100
stream: add insert failure counters
commit 91f57200c7c4d5f32961aa497ed436163a8f8e41
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 11:48:30 2016 +0200
stream: add stream.reassembly.check-overlap-different-data option
commit f02dc377efd7f5b1aa8a961f18cbe0ef49bc395e
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 10:53:49 2016 +0200
stream: add tcp.overlap and tcp.overlap_diff_data counters
commit 8c36e52d9315e120f41e10439aad03cc5dd41f40
Author: Victor Julien <victor at inliniac.net>
Date: Sat Feb 25 10:20:51 2017 +0100
stream: improve no app and no raw case
commit 5ee36a0c8bb205ae5d8f169af875fbb5ca0db80e
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 09:53:24 2016 +0200
stream: make raw_progress relative to STREAM_BASE_OFFSET
commit fa2a832022829f64424418b9305c8123b5468c8e
Author: Victor Julien <victor at inliniac.net>
Date: Thu Apr 28 08:44:10 2016 +0200
stream: make app_progress relative to STREAM_BASE_OFFSET
commit 187e2381c89f887a7a16f24d0c28204dcb55f6a0
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 27 22:13:27 2016 +0200
stream: reduce space used for progress tracking
Instead of the explicit base_seq_offset, use a macro instead. The
macro points to the stream buffer offset. The two were always
in sync.
commit 9bd11bcda5079fbb34727605c2f23ce605ebbbce
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 27 21:47:10 2016 +0200
stream: small cleanups
commit b3e9d397719a972d16b4a4eb2c6fa4a456a4d281
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 27 21:37:28 2016 +0200
stream: remove unused zero copy setting
commit bbb0df14d2697960c531ea400737665438bee081
Author: Victor Julien <victor at inliniac.net>
Date: Wed Mar 8 23:24:59 2017 +0100
stream: safety check in overlap handling
commit 8c9f521707715110c8ffe9fe0c82102a49314ec0
Author: Victor Julien <victor at inliniac.net>
Date: Tue Dec 22 10:26:04 2015 +0100
tcp: streaming implementation
Make stream engine use the streaming buffer API for it's data storage.
This means that the data is stored in a single reassembled sliding
buffer. The subleties of the reassembly, e.g. overlap handling, are
taken care of at segment insertion.
The TcpSegments now have a StreamingBufferSegment that contains an
offset and a length. Using this the segment data can be retrieved
per segment.
Redo segment insertion. The insertion code is moved to it's own file
and is simplified a lot.
A major difference with the previous implementation is that the segment
list now contains overlapping segments if the traffic is that way.
Previously there could be more and smaller segments in the memory list
than what was seen on the wire.
Due to the matching of in memory segments and on the wire segments,
the overlap with different data detection (potential mots attacks)
is much more accurate.
Raw and App reassembly progress is no longer tracked per segment using
flags, but there is now a progress tracker in the TcpStream for each.
When pruning we make sure we don't slide beyond in-use segments. When
both app-layer and raw inspection are beyond the start of the segment
list, the segments might not be freed even though the data in the
streaming buffer is already gone. This is caused by the 'in-use' status
that the segments can implicitly have. This patch accounts for that
when calculating the 'left_edge' of the streaming window.
Raw reassembly still sets up 'StreamMsg' objects for content
inspection. They are set up based on either the full StreamingBuffer,
or based on the StreamingBufferBlocks if there are gaps in the data.
Reworked 'stream needs work' logic. When a flow times out the flow
engine checks whether a TCP flow still needs work. The
StreamNeedsReassembly function is used to test if a stream still has
unreassembled segments or uninspected stream chunks.
This patch updates the function to consider the app and/or raw
progress. It also cleans the function up and adds more meaningful
debug messages. Finally it makes it non-inline.
Unittests have been overhauled, and partly moved into their own files.
Remove lots of dead code.
commit 3fa2e8689cb86d7d6aae94aead2da7c3ffbd3194
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 6 19:45:11 2016 +0200
streaming: remove BUG_ON and other improvements
Can be triggered by memory limits.
commit d789dc7e6de057d756b992c562b7f1583cc70c05
Author: Victor Julien <victor at inliniac.net>
Date: Tue Jan 19 22:31:22 2016 -0500
streaming: add blocklist
Add list of 'blocks'. This list contains offsets and lengths to
continuous data blocks. This is useful for TCP tracking where we
can have data gaps.
The blocks don't contain any data themselves, instead they contain
lenght and offsets. This way no extra copying is needed.
On inserting new data, existing blocks are expanded instead of
having multiple neighbouring blocks.
commit 9e1470d81c79677b689a0fc4b570d20232953dad
Author: Victor Julien <victor at inliniac.net>
Date: Thu Mar 17 10:08:00 2016 +0100
stream-tcp: StreamTcpUTAddPayload unittest helper
commit 3411697106296da912601005ff8cb963f6cdea66
Author: Victor Julien <victor at inliniac.net>
Date: Fri Mar 3 09:08:23 2017 +0100
profile: account flow-worker tcp-prune step
commit 245a89b7e74cfa4d60ab4f93d9708dd1af7d803f
Author: Victor Julien <victor at inliniac.net>
Date: Wed Apr 5 10:33:23 2017 +0200
doc: http keywords update
commit 595f6d1f266650f9f7a16e740b90a08c6d7a79cf
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 23:58:51 2017 +0200
detect: implement http_content_len sticky buffer
This implements inspection of the Content-Length buffer as a content
sticky buffer.
commit d96cbddbe45a01cc1a0c534551c46735cf2b9d05
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 23:28:12 2017 +0200
detect: implement http_content_type sticky buffer
commit 46febef7cd1ceda05b9ab1bf0dbc94e50bc4ce17
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 12:38:46 2017 +0200
detect: implement http_accept_enc sticky buffer
Inspects Accept-Encoding header.
commit 9d5bbc3af6009697848268b6a6221ffadf037a8f
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 12:33:49 2017 +0200
detect: implement http_accept_lang sticky buffer
Inspects Accept-Language header
commit 1d1176b0d549e7f13e3b873eb6b396e9f845358b
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 12:23:05 2017 +0200
detect: implement http_connection sticky buffer
commit 5714129e32f4dd26a309c08826037a53dd7a84cc
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 12:13:17 2017 +0200
detect: implement http referer sticky buffer
commit 88cfb99910f3b70ea565fcf9635f86afdaf46f1e
Author: Victor Julien <victor at inliniac.net>
Date: Sat Apr 1 11:49:20 2017 +0200
detect: http_accept sticky buffer + common code
Implement common code to easily add more per HTTP header detection
keywords.
Implement http_accept sticky buffer. It operates on the HTTP Accept
header.
-----------------------------------------------------------------------
Summary of changes:
doc/userguide/configuration/suricata-yaml.rst | 64 +-
doc/userguide/rules/http-keywords.rst | 168 +-
src/Makefile.am | 11 +-
src/app-layer-detect-proto.c | 105 -
src/app-layer-htp.c | 28 +-
src/app-layer-htp.h | 4 +-
src/app-layer-parser.c | 51 +-
src/app-layer-parser.h | 10 +-
src/app-layer-smtp.c | 4 +-
src/app-layer-ssh.c | 19 +-
src/app-layer-ssh.h | 3 +
src/app-layer-ssl.c | 21 +-
src/app-layer-ssl.h | 3 +
src/app-layer.c | 56 +-
src/decode.h | 2 +
src/detect-app-layer-event.c | 4 +-
src/detect-cipservice.c | 8 +-
src/detect-dce-iface.c | 8 +-
src/detect-dce-stub-data.c | 8 +-
src/detect-dnp3.c | 8 +-
src/detect-dns-query.c | 6 +-
src/detect-engine-alert.c | 1 -
src/detect-engine-mpm.c | 23 +
src/detect-engine-payload.c | 201 +-
src/detect-engine-payload.h | 6 +-
src/detect-engine-prefilter.c | 22 +-
src/detect-engine-prefilter.h | 2 +-
src/detect-engine-state.c | 296 +-
src/detect-engine-uri.c | 100 +-
src/detect-engine.c | 124 +-
src/detect-engine.h | 4 +-
src/detect-file-data.c | 4 +-
src/detect-filename.c | 6 +-
src/detect-ftpbounce.c | 3 +-
src/{util-random.h => detect-http-accept-enc.c} | 26 +-
src/{util-random.h => detect-http-accept-enc.h} | 15 +-
src/{util-random.h => detect-http-accept-lang.c} | 26 +-
src/{util-random.h => detect-http-accept-lang.h} | 15 +-
src/{util-random.h => detect-http-accept.c} | 26 +-
src/{util-random.h => detect-http-accept.h} | 15 +-
src/detect-http-client-body.c | 2 +-
src/{util-random.h => detect-http-connection.c} | 26 +-
src/{util-random.h => detect-http-connection.h} | 15 +-
src/{util-random.h => detect-http-content-len.c} | 27 +-
src/{util-random.h => detect-http-content-len.h} | 15 +-
src/{util-random.h => detect-http-content-type.c} | 27 +-
src/{util-random.h => detect-http-content-type.h} | 15 +-
src/detect-http-cookie.c | 4 +-
src/detect-http-header-names.c | 4 +-
src/detect-http-header.c | 206 +-
...p-header-names.c => detect-http-headers-stub.h} | 300 +-
src/{util-random.h => detect-http-headers.c} | 28 +-
src/{util-random.h => detect-http-headers.h} | 15 +-
src/detect-http-hh.c | 2 +-
src/detect-http-hrh.c | 2 +-
src/detect-http-method.c | 2 +-
src/detect-http-protocol.c | 4 +-
src/detect-http-raw-header.c | 4 +-
src/detect-http-raw-uri.c | 2 +-
src/{util-random.h => detect-http-referer.c} | 26 +-
src/{util-random.h => detect-http-referer.h} | 15 +-
src/detect-http-request-line.c | 2 +-
src/detect-http-response-line.c | 2 +-
src/detect-http-start.c | 4 +-
src/detect-http-stat-code.c | 2 +-
src/detect-http-stat-msg.c | 2 +-
src/detect-http-ua.c | 2 +-
src/detect-http-uri.c | 3 +-
src/detect-lua.c | 4 +-
src/detect-modbus.c | 4 +-
src/detect-pcre.c | 40 +-
src/detect-ssh-proto-version.c | 64 +-
src/detect-ssh-proto.c | 4 +-
src/detect-ssh-software-version.c | 6 +-
src/detect-ssh-software.c | 4 +-
src/detect-ssl-state.c | 4 +-
src/detect-ssl-version.c | 100 -
src/detect-template-buffer.c | 4 +-
src/detect-tls-cert-issuer.c | 2 +-
src/detect-tls-cert-serial.c | 3 +-
src/detect-tls-cert-subject.c | 2 +-
src/detect-tls-cert-validity.c | 2 +-
src/detect-tls-sni.c | 2 +-
src/detect-tls-version.c | 101 -
src/detect-tls.c | 2 +-
src/detect-uricontent.c | 321 +-
src/detect.c | 300 +-
src/detect.h | 33 +-
src/flow-manager.c | 8 +-
src/flow-timeout.c | 105 +-
src/flow-worker.c | 6 +-
src/flow-worker.h | 1 +
src/output-streaming.c | 8 +-
src/stream-tcp-inline.c | 595 +-
src/stream-tcp-inline.h | 6 +-
src/stream-tcp-list.c | 952 ++
src/{detect-filesha1.h => stream-tcp-list.h} | 16 +-
src/stream-tcp-private.h | 55 +-
src/stream-tcp-reassemble.c | 9841 +++++---------------
src/stream-tcp-reassemble.h | 53 +-
src/stream-tcp-util.c | 46 +-
src/stream-tcp-util.h | 3 +-
src/stream-tcp.c | 765 +-
src/stream-tcp.h | 59 +-
src/stream.c | 239 +-
src/stream.h | 39 +-
src/suricata-common.h | 4 +-
src/suricata.c | 1 +
src/suricata.h | 1 -
src/tests/stream-tcp-inline.c | 159 +
src/tests/stream-tcp-list.c | 733 ++
src/tests/stream-tcp-reassemble.c | 208 +
src/util-mpm-ac.c | 1 -
src/util-profiling.c | 4 +-
src/util-streaming-buffer.c | 757 +-
src/util-streaming-buffer.h | 25 +-
src/util-unittest-helper.c | 53 +-
src/util-unittest-helper.h | 3 +
src/util-unittest.c | 18 +
suricata.yaml.in | 45 +-
120 files changed, 7416 insertions(+), 10659 deletions(-)
copy src/{util-random.h => detect-http-accept-enc.c} (61%)
copy src/{util-random.h => detect-http-accept-enc.h} (80%)
copy src/{util-random.h => detect-http-accept-lang.c} (60%)
copy src/{util-random.h => detect-http-accept-lang.h} (80%)
copy src/{util-random.h => detect-http-accept.c} (63%)
copy src/{util-random.h => detect-http-accept.h} (80%)
copy src/{util-random.h => detect-http-connection.c} (61%)
copy src/{util-random.h => detect-http-connection.h} (80%)
copy src/{util-random.h => detect-http-content-len.c} (59%)
copy src/{util-random.h => detect-http-content-len.h} (80%)
copy src/{util-random.h => detect-http-content-type.c} (59%)
copy src/{util-random.h => detect-http-content-type.h} (80%)
copy src/{detect-http-header-names.c => detect-http-headers-stub.h} (54%)
copy src/{util-random.h => detect-http-headers.c} (56%)
copy src/{util-random.h => detect-http-headers.h} (80%)
copy src/{util-random.h => detect-http-referer.c} (62%)
copy src/{util-random.h => detect-http-referer.h} (80%)
create mode 100644 src/stream-tcp-list.c
copy src/{detect-filesha1.h => stream-tcp-list.h} (77%)
create mode 100644 src/tests/stream-tcp-inline.c
create mode 100644 src/tests/stream-tcp-list.c
create mode 100644 src/tests/stream-tcp-reassemble.c
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list