[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-4.0.1-427-g7da805f
OISF Git
noreply at openinfosecfoundation.org
Tue Feb 6 20:15:42 UTC 2018
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via 7da805ffd9a9202c67d53ef6a06c3215436495e9 (commit)
via 6f69c65c48a1f1466472f8e88398c817a3c0cc72 (commit)
via 194751654fba60424df9c2af1bdceae29c672fa1 (commit)
via 8030e3f66b663ad76a845487ea1b7efea26413f2 (commit)
via 15260460726ba24c9d5c25e61f258b3f3f63e708 (commit)
via 796ec08dd7a6311b108b5352ab66653b10f2aff9 (commit)
via 460a0a6977614a521ebb17cc8e531c6528ff8c69 (commit)
via 0e1a4173ff53f6eb156630f216e195e1f38caa70 (commit)
via 276b93fb53c8b7f52ca0e1382a05aa2788e3680d (commit)
via 0356293d848a9d7ed82c60e7c2106eac737bfd0f (commit)
via 93f5b5f1e82ddc8c08e86a0a46993d8cd2fb53db (commit)
via b93548d2ab4df83e6ec289ba09c4be70d28995fd (commit)
via 027c903f50bcba8da10a4577891c6142cac09429 (commit)
via 8c7b5cb0889a493c1bd13feb2facf45d14f96b44 (commit)
via f04391031f113dbab7bae506906628e606a6adaf (commit)
via 7bec54158fc09a772457c52448cfc4fc6cc743f6 (commit)
via bc34703c093ff2d4e7d7e11d894eca393969df8a (commit)
via 0807dd674063062d22e0677837f2b8e7bea4a965 (commit)
via 2b56b02569d8d79da3de98d5f11884afa502aec0 (commit)
via 94a622cb552ece8a403830855c735499c696a6ca (commit)
via 3379311e661d9eb2ab9ba520f230734d45d95156 (commit)
via e1515b48ad4fe42bbdd474b461bf0a9a6dcca0d5 (commit)
via ce8b74b524da51104e48826caebd9cc386bde97b (commit)
via 4f57008a2353dc15878c0bcd0c53c5af6430bb28 (commit)
via 33072a49fe38420e15bd167057bf9b16b94b2f8b (commit)
via ac5957d4615c45e4de67f99cac571904dd2f6dd3 (commit)
via 2598078ee6da1169326c3f59e4ac8766896b8297 (commit)
via cde438f670e61dcf87a6fe2dc8c94eef57b39f4a (commit)
via 3f3a206722e1bde8ac8ecdcb2c0eb856c9e69529 (commit)
via 60265e023a4c146503d49f6b56c00e9cdb92a027 (commit)
via 5ee44c877c9baf1a840a38729f79d672a0bfa810 (commit)
via 1e729f059f438fa2955a10e15b63f31731c540b7 (commit)
via 126488f74dbaf1121fcf614b3f75a77e97562604 (commit)
via 4474889667d664a66c1c123f4f7d2756e8a7fbb9 (commit)
via 0998f37b78fd86d8473d35074b245b01d1505e9e (commit)
via 3ae714d3544e80c40c873112b0b8aeefb0805d42 (commit)
via e2d31e1c571bc63da76ebf37d4aeb86e4269c965 (commit)
via 7a0d53448d87ad3d141050e09422e4e0597d8e0a (commit)
via 9cb591aa6cd8c005e3514f843ef796b24756cd82 (commit)
via eff10fce1054257a46349904bc043edd38bcea72 (commit)
via d65f45856cd596dee61d30435ba17fbd7bb4c100 (commit)
via 17a32bdaa0d836e7e33c848492d2becf9f850df7 (commit)
via b937e1afefe065ce7489b1455fccf99c5ad3f0d0 (commit)
via 60752d231c3d2faaecacd44c62754d3facc59850 (commit)
via 08eec0833e4990ade0a196a0c8b56398ec7eb4f0 (commit)
via 43ecf0d78d5862cbda341ec9521758b6e7e9df81 (commit)
via a2296357927bca505a1ee033f034ae9b186e287d (commit)
via 0654c313975c184c013650e5565b81d333660c67 (commit)
via 8c880879486705fc2c121c4791128bc7f20ef6c8 (commit)
via 31c947b4d8ecd97829ce5293ddbafb89c3aaaebc (commit)
via 8640cc5dcf6677e6c74c46fdb7b8007c5e119f37 (commit)
via 06173267c6d6c5702db6c4d3a9710b4f8dd2b56e (commit)
via 91e1256b0134ebe89b89e18bf785d20679c25225 (commit)
from d2121945c93ea7db0454a2865c8696b940df477a (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 7da805ffd9a9202c67d53ef6a06c3215436495e9
Author: Eric Leblond <eric at regit.org>
Date: Fri Feb 2 21:39:14 2018 +0100
doc: improve eBPF and XDP doc
Remove reference to `buggy` clang as a workaround has been found in
libbpf.
Proof read and add information on the structure of eBPF code.
commit 6f69c65c48a1f1466472f8e88398c817a3c0cc72
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 31 21:20:52 2018 +0100
util-ebpf: rename local variable
commit 194751654fba60424df9c2af1bdceae29c672fa1
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 31 21:12:59 2018 +0100
af-packet: count only CPUs once
commit 8030e3f66b663ad76a845487ea1b7efea26413f2
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 26 10:14:44 2018 +0100
doc: update documentation
This patch adds info on kernel requirement for XDP and rework a few
things.
commit 15260460726ba24c9d5c25e61f258b3f3f63e708
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 26 10:11:56 2018 +0100
ebpf: allow modified build of xdp_filter
Using BUILD_CPUMAP constant will allow user to use XDP bypass on
kernel prior to 4.15.
commit 796ec08dd7a6311b108b5352ab66653b10f2aff9
Author: Jesper Dangaard Brouer <netoptimizer at brouer.com>
Date: Thu Jan 25 20:24:38 2018 +0100
ebpf: improve xdp-cpu-redirect distribution in xdp_filter.c
The XDP CPU destination array/set, configured via xdp-cpu-redirect,
will always be fairly small. My different benchmarking showed that
the current modulo hashing into the CPU array can easily result in bad
distribution, expecially if the number of CPU is an even number.
This patch uses a proper hashing function on the input key. The key
used for hashing is inspired by the ippair hashing code in
src/tmqh-flow.c, and is based on the IP src + dst.
An important property is that the hashing is flow symmetric, meaning
that if the source and destintation gets swapped then the selected CPU
will remain the same. This is important for Suricate.
That hashing INITVAL (15485863 the 10^6th prime number) was fairly
arbitrary choosen, but experiments with kernel tree pktgen scripts
(pktgen_sample04_many_flows.sh +pktgen_sample05_flow_per_thread.sh)
showed this improved the distribution.
Signed-off-by: Jesper Dangaard Brouer <netoptimizer at brouer.com>
commit 460a0a6977614a521ebb17cc8e531c6528ff8c69
Author: Jesper Dangaard Brouer <brouer at redhat.com>
Date: Thu Jan 25 20:24:33 2018 +0100
ebpf: add Paul Hsieh's (LGPL 2.1) hash function SuperFastHash
Adjusted function call API to take an initval. This allow the API
user to set the initial value, as a seed. This could also be used for
inputting the previous hash.
Signed-off-by: Jesper Dangaard Brouer <brouer at redhat.com>
commit 0e1a4173ff53f6eb156630f216e195e1f38caa70
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 17 16:59:23 2018 +0100
doc: how to get live info about ebpf behavior
commit 276b93fb53c8b7f52ca0e1382a05aa2788e3680d
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 17 03:45:01 2018 +0100
unix-socket: add ebpf-bypassed-stats command
This command output the count of element in IPv4 and IPv6 flow
table of interfaces using eBPF/XDP bypass.
commit 0356293d848a9d7ed82c60e7c2106eac737bfd0f
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 17 03:28:20 2018 +0100
unix-socket: add bypassed counter to iface-stat
commit 93f5b5f1e82ddc8c08e86a0a46993d8cd2fb53db
Author: Eric Leblond <eric at regit.org>
Date: Wed Jan 17 03:24:58 2018 +0100
util-ebpf: add bypassed counters
Use LiveDevice bypassed counter and also add hash size counters
for the v4 and v6 flow table.
commit b93548d2ab4df83e6ec289ba09c4be70d28995fd
Author: Jesper Dangaard Brouer <netoptimizer at brouer.com>
Date: Mon Jan 15 19:31:27 2018 +0100
ebpf: maintain a copy of kernel UAPI header file linux/bpf.h
commit 027c903f50bcba8da10a4577891c6142cac09429
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 12 16:18:34 2018 +0100
ebpf: fix detection of llc
commit 8c7b5cb0889a493c1bd13feb2facf45d14f96b44
Author: Eric Leblond <eric at regit.org>
Date: Thu Jan 11 16:56:18 2018 +0100
doc: add info about xdp IPS bypass
commit f04391031f113dbab7bae506906628e606a6adaf
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 9 00:25:02 2018 +0100
af-packet: XDP bypass in IPS mode
Implement XDP bypass in IPS mode by using XDP redirect to send
packets from bypassed flow directly to the transmission interface.
commit 7bec54158fc09a772457c52448cfc4fc6cc743f6
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 9 23:11:23 2018 +0100
flow-bypass: introduce update function
Main objective of the function is to be able to bypass a flow on
other interfaces. This is necessary in AF_PACKET case as the flow
table are per interface.
commit bc34703c093ff2d4e7d7e11d894eca393969df8a
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 22:05:40 2018 +0100
ebpf: add some comments to eBPF filter
commit 0807dd674063062d22e0677837f2b8e7bea4a965
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 21:58:09 2018 +0100
ebpf: slight bypass_filter optimization
commit 2b56b02569d8d79da3de98d5f11884afa502aec0
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 21:47:43 2018 +0100
util-ebpf: simplify code cleaning
Avoid to use an unnecessary callback strategy as the purpose of the
function using the callback is hardcoded.
commit 94a622cb552ece8a403830855c735499c696a6ca
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 13:16:27 2018 +0100
af-packet: add comments to eBPF/XDP code
commit 3379311e661d9eb2ab9ba520f230734d45d95156
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 13:15:37 2018 +0100
af-packet: fix error handling in bypass case
If the key is already in the hash table then the bypass is
succesful.
commit e1515b48ad4fe42bbdd474b461bf0a9a6dcca0d5
Author: Eric Leblond <eric at regit.org>
Date: Sun Jan 7 12:59:16 2018 +0100
util-ebpf: fix libbpf error handling
commit ce8b74b524da51104e48826caebd9cc386bde97b
Author: Eric Leblond <eric at regit.org>
Date: Thu Jan 11 16:52:21 2018 +0100
doc: document XDP CPU redirect
commit 4f57008a2353dc15878c0bcd0c53c5af6430bb28
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 2 22:08:21 2018 +0100
af-packet: add support for XDP cpu redirect map
This patch adds a boolean option "xdp-cpu-redirect" to af-packet
interface configuration. If set, then the XDP filter will load
balance the skb creation on specified CPUs instead of doing the
creation on the CPU handling the packet. In the case of a card
with asymetric hashing this will allow to avoid saturating the
single CPU handling the trafic.
The XDP filter must contains a set of map allowing load balancing.
This is the case of xdp_filter.bpf.
Fixed-by: Jesper Dangaard Brouer <netoptimizer at brouer.com>
commit 33072a49fe38420e15bd167057bf9b16b94b2f8b
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 2 19:30:00 2018 +0100
ebpf: import more recent version of helpers
commit ac5957d4615c45e4de67f99cac571904dd2f6dd3
Author: Eric Leblond <eric at regit.org>
Date: Sat Jan 6 09:44:00 2018 +0100
util-affinity: export CPU set parsing function
commit 2598078ee6da1169326c3f59e4ac8766896b8297
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 5 23:32:54 2018 +0100
af-packet: code cleaning and comments
commit cde438f670e61dcf87a6fe2dc8c94eef57b39f4a
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 5 13:27:59 2018 +0100
af-packet: add missing copyright header
And also fixes the copyright date in some files.
commit 3f3a206722e1bde8ac8ecdcb2c0eb856c9e69529
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 2 12:50:26 2018 +0100
util-ebpf: add error handling in hash value fetch
commit 60265e023a4c146503d49f6b56c00e9cdb92a027
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 5 19:31:43 2018 +0100
doc: update xdp documentation
Also remove configuration info from yaml as they are now in the
documentation.
commit 5ee44c877c9baf1a840a38729f79d672a0bfa810
Author: Peter Manev <pmanev at stamus-networks.com>
Date: Sat Dec 30 22:11:35 2017 +0100
doc: add XDP setup documentation
commit 1e729f059f438fa2955a10e15b63f31731c540b7
Author: Eric Leblond <eric at regit.org>
Date: Sat Dec 30 22:04:04 2017 +0100
af-packet: improve xdp error handling
Don't try to bypass the flow if the flow table is unknown.
Also continue after error message if ever XDP was not correctly
setup.
commit 126488f74dbaf1121fcf614b3f75a77e97562604
Author: Eric Leblond <eric at regit.org>
Date: Tue Dec 26 22:55:55 2017 +0100
af-packet: add support for multi iface bypass
commit 4474889667d664a66c1c123f4f7d2756e8a7fbb9
Author: Eric Leblond <eric at regit.org>
Date: Sat Dec 30 19:50:12 2017 +0100
util-device: change logic of registration
Device storage requires the devices to be created after storage
is finalized so we need to first get the list of devices then
create them when the storage is finalized.
This patch introduces the LiveDeviceName structure that is a list
of device name used during registration.
Code uses LiveRegisterDeviceName for pre registration and keep
using the LiveRegisterDevice function for part of the code that
create the interface during the runmode creation.
commit 0998f37b78fd86d8473d35074b245b01d1505e9e
Author: Eric Leblond <eric at regit.org>
Date: Tue Dec 26 22:47:44 2017 +0100
util-device: add an iteration function
commit 3ae714d3544e80c40c873112b0b8aeefb0805d42
Author: Eric Leblond <eric at regit.org>
Date: Tue Jan 2 00:33:23 2018 +0100
device-storage: introduce feature
The capture method may have to store data depending related to the
offloading so having a per interface storage via LiveDevice seems
interesting.
commit e2d31e1c571bc63da76ebf37d4aeb86e4269c965
Author: Eric Leblond <eric at regit.org>
Date: Thu Dec 28 19:17:56 2017 +0100
flow-bypass: fix sleep strategy
commit 7a0d53448d87ad3d141050e09422e4e0597d8e0a
Author: Eric Leblond <eric at regit.org>
Date: Tue Dec 26 20:14:24 2017 +0100
tm-threads: fix build warning in afl mode
commit 9cb591aa6cd8c005e3514f843ef796b24756cd82
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 24 16:19:03 2017 +0100
af-packet: remove done fixme in XDP
commit eff10fce1054257a46349904bc043edd38bcea72
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 24 16:17:19 2017 +0100
af-packet: end of map factoring
commit d65f45856cd596dee61d30435ba17fbd7bb4c100
Author: Eric Leblond <eric at regit.org>
Date: Thu Dec 21 16:52:37 2017 +0100
af-packet: cache map fd search
commit 17a32bdaa0d836e7e33c848492d2becf9f850df7
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 17 22:09:23 2017 +0100
af-packet: fix bypassing of IPv6
Also misc fixes.
commit b937e1afefe065ce7489b1455fccf99c5ad3f0d0
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 17 11:58:59 2017 +0100
util-ebpf: fix ipv6 cleaning and add comments
commit 60752d231c3d2faaecacd44c62754d3facc59850
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 17 11:47:11 2017 +0100
util-ebpf: fix XDP delete key
The key was deleted twice so let's remove the local deletion.
commit 08eec0833e4990ade0a196a0c8b56398ec7eb4f0
Author: Eric Leblond <eric at regit.org>
Date: Sun Dec 17 00:01:42 2017 +0100
flow-bypass: add abstraction layer
The flow bypass thread can now be used by any capture method that
register it timeout check function.
commit 43ecf0d78d5862cbda341ec9521758b6e7e9df81
Author: Eric Leblond <eric at regit.org>
Date: Wed Dec 13 23:09:42 2017 +0100
util-ebpf: add call to remove memlock limit
Without that, user has to use ulimit to be able to load the eBPF
file.
commit a2296357927bca505a1ee033f034ae9b186e287d
Author: Eric Leblond <eric at regit.org>
Date: Fri Dec 1 20:07:27 2017 +0100
ebpf: implement vlan filter
Basic filter allowing only a list of VLANs.
commit 0654c313975c184c013650e5565b81d333660c67
Author: Eric Leblond <eric at regit.org>
Date: Sun Nov 12 15:53:39 2017 +0100
util-ebpf: suppress call on loop init
commit 8c880879486705fc2c121c4791128bc7f20ef6c8
Author: Eric Leblond <eric at regit.org>
Date: Tue Nov 28 00:21:54 2017 +0100
af-packet: implementation of XDP bypass
This patch adds support for XDP bypass. It provides an XDP
filter that can be loaded to realize the bypass of flows.
commit 31c947b4d8ecd97829ce5293ddbafb89c3aaaebc
Author: Eric Leblond <eric at regit.org>
Date: Fri Jan 5 22:33:48 2018 +0100
af-packet: use per CPU hash in bypass
eBPF has a data type which is a per CPU array. By adding one element
to the array it is in fact added to all per CPU arrays in the kernel.
This allows to have a lockless structure in the kernel even when doing
counter update.
In userspace, we need to update the flow bypass code to fetch all
elements of the per CPU arrays.
commit 8640cc5dcf6677e6c74c46fdb7b8007c5e119f37
Author: Eric Leblond <eric at regit.org>
Date: Fri Sep 29 22:24:08 2017 +0200
flow-bypass: only start thread on demand
commit 06173267c6d6c5702db6c4d3a9710b4f8dd2b56e
Author: Eric Leblond <eric at regit.org>
Date: Sun Nov 26 20:29:52 2017 +0100
af-packet: kernel bypass implementation
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
commit 91e1256b0134ebe89b89e18bf785d20679c25225
Author: Eric Leblond <eric at regit.org>
Date: Sun Nov 26 20:27:13 2017 +0100
af-packet: add support for eBPF cluster and filter
This patch introduces the ebpf cluster mode. This mode is using
an extended BPF function that is loaded into the kernel and
provide the load balancing.
An example of cluster function is provided in the ebpf
subdirectory and provide ippair load balancing function.
This is a function which uses the same method as
the one used in autofp ippair to provide a symetrical
load balancing based on IP addresses.
A simple filter example allowing to drop IPv6 is added to the
source.
This patch also prepares the infrastructure to be able to load
and use map inside eBPF files. This will be used later for flow
bypass.
-----------------------------------------------------------------------
Summary of changes:
Makefile.am | 2 +-
configure.ac | 77 +-
doc/userguide/capture-hardware/ebpf-xdp.rst | 365 +++++++++
doc/userguide/capture-hardware/index.rst | 1 +
ebpf/Makefile.am | 13 +
ebpf/bpf_helpers.h | 262 +++++++
ebpf/bypass_filter.c | 226 ++++++
ebpf/filter.c | 59 ++
ebpf/hash_func01.h | 55 ++
ebpf/include/linux/bpf.h | 1027 ++++++++++++++++++++++++++
ebpf/include/linux/bpf_common.h | 57 ++
ebpf/lb.c | 110 +++
src/output-filestore.h => ebpf/vlan_filter.c | 27 +-
ebpf/xdp_filter.c | 379 ++++++++++
src/Makefile.am | 3 +
src/device-storage.c | 111 +++
src/{util-conf.h => device-storage.h} | 27 +-
src/flow-bypass.c | 181 +++++
src/flow-bypass.h | 49 ++
src/flow.c | 7 +
src/runmode-af-packet.c | 158 +++-
src/runmodes.c | 18 +
src/runmodes.h | 3 +
src/source-af-packet.c | 403 +++++++++-
src/source-af-packet.h | 13 +
src/source-ipfw.c | 2 +-
src/source-nfq.c | 2 +-
src/suricata-common.h | 4 +-
src/suricata.c | 26 +-
src/tm-modules.c | 1 +
src/tm-threads-common.h | 1 +
src/tm-threads.c | 4 +
src/unix-manager.c | 4 +
src/util-affinity.c | 16 +-
src/util-affinity.h | 5 +
src/util-buffer.h | 8 +-
src/util-device.c | 93 ++-
src/util-device.h | 10 +
src/util-ebpf.c | 732 ++++++++++++++++++
src/util-ebpf.h | 87 +++
src/util-storage.c | 2 +
src/util-storage.h | 1 +
suricata.yaml.in | 4 +
43 files changed, 4589 insertions(+), 46 deletions(-)
create mode 100644 doc/userguide/capture-hardware/ebpf-xdp.rst
create mode 100644 ebpf/Makefile.am
create mode 100644 ebpf/bpf_helpers.h
create mode 100644 ebpf/bypass_filter.c
create mode 100644 ebpf/filter.c
create mode 100644 ebpf/hash_func01.h
create mode 100644 ebpf/include/linux/bpf.h
create mode 100644 ebpf/include/linux/bpf_common.h
create mode 100644 ebpf/lb.c
copy src/output-filestore.h => ebpf/vlan_filter.c (58%)
create mode 100644 ebpf/xdp_filter.c
create mode 100644 src/device-storage.c
copy src/{util-conf.h => device-storage.h} (50%)
create mode 100644 src/flow-bypass.c
create mode 100644 src/flow-bypass.h
create mode 100644 src/util-ebpf.c
create mode 100644 src/util-ebpf.h
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list