<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 20, 2014 at 8:00 PM, Darren Spruell <span dir="ltr"><<a href="mailto:phatbuckett@gmail.com" target="_blank">phatbuckett@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Hoping for some performance tuning guidance on the following system:<br>
<br>
Suricata 2.0 RELEASE<br>
CentOS 6.5 amd64<br>
Linux 2.6.32-431.17.1.el6.x86_64<br>
libpcre 8.3.5<br>
luajit 2.0.3<br>
libpcap-1.4.0-1.20130826git2dbcaa1.el6.x86_64<br>
12 core (2x6) Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz (24 cpu w/HT)<br>
64GB RAM<br>
<br>
For the time being we're held on this 2.6.32 kernel and wonder if we<br>
can achieve suitable performance/minimal packet drop with AF_PACKET.<br>
PF_RING may be an option but not likely DNA/zero_copy due to<br>
licensing, only vanilla. Can we achieve close to 0% drop with this<br>
hardware/kernel/ruleset as described below? Is an upgraded kernel a<br>
major factor in achieving better performance in this<br>
configuration/traffic profile?<br>
<br>
I find pretty good information about 10Gbps Suricata tuning (Intel<br>
82599 adapters, typically) but I'm not certain what pieces of network<br>
adapter setup would apply to a 1Gbps adapter:<br>
<br>
43:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network<br>
Connection (rev 01)<br>
43:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network<br>
Connection (rev 01)<br>
44:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network<br>
Connection (rev 01)<br>
44:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network<br>
Connection (rev 01)<br>
<br>
Updated igb driver to latest:<br>
<br>
driver: igb<br>
version: 5.2.5<br>
firmware-version: 1.2.1<br>
bus-info: 0000:43:00.1<br>
supports-statistics: yes<br>
supports-test: yes<br>
supports-eeprom-access: yes<br>
supports-register-dump: yes<br>
supports-priv-flags: no<br>
<br>
Is it proper to disable all NIC offloading features?<br>
<br>
$ sudo ethtool -k eth6<br>
Features for eth6:<br>
rx-checksumming: off<br>
tx-checksumming: off<br>
scatter-gather: off<br>
tcp-segmentation-offload: off<br>
udp-fragmentation-offload: off<br>
generic-segmentation-offload: off<br>
generic-receive-offload: off<br>
large-receive-offload: off<br>
ntuple-filters: off<br>
receive-hashing: off<br>
<br>
$ sudo ethtool -g eth6<br>
Ring parameters for eth6:<br>
Pre-set maximums:<br>
RX: 4096<br>
RX Mini: 0<br>
RX Jumbo: 0<br>
TX: 4096<br>
Current hardware settings:<br>
RX: 256<br>
RX Mini: 0<br>
RX Jumbo: 0<br>
TX: 256<br>
<br>
$ sudo ethtool -n eth6<br>
1 RX rings available<br>
rxclass: Cannot get RX class rule count: Operation not supported<br>
RX classification rule retrieval failed<br>
<br>
Traffic throughput is around 500 Mbps -- 800 Mbps , composed mostly of<br>
TCP streams (forward proxy requests, clients <-> proxies).<br>
<br>
Packet size brackets for interface bond1<br>
<br>
Packet Size (bytes) Count Packet Size (bytes) Count<br>
1 to 75: 31628146 751 to 825: 458232<br>
76 to 150: 2284335 826 to 900: 361877<br>
151 to 225: 651222 901 to 975: 672172<br>
226 to 300: 524288 976 to 1050: 501744<br>
301 to 375: 613914 1051 to 1125: 323661<br>
376 to 450: 1391229 1126 to 1200: 362991<br>
451 to 525: 1032754 1201 to 1275: 1312685<br>
526 to 600: 818482 1276 to 1350: 341888<br>
601 to 675: 7107770 1351 to 1425: 636282<br>
676 to 750: 718379 1426 to 1500+: 34102516<br>
<br>
Proto/Port Pkts Bytes PktsTo BytesTo<br>
PktsFrom BytesFrom<br>
TCP/3128 3788780 2948M 1397043 219783k<br>
2391737 2728M<br>
TCP/443 40160 11690875 21846 2569412<br>
18314 9121463<br>
TCP/80 13948 10720193 5295 342935<br>
8653 10377258<br>
UDP/53 5749 756317 2925 204953<br>
2824 551364<br>
UDP/161 3260 527866 1710 219259<br>
1550 308607<br>
TCP/43 4969 373030 2493 154131<br>
2476 218899<br>
UDP/514 281 67116 267 66004<br>
14 1112<br>
TCP/22 129 27147 64 7581<br>
65 19566<br>
UDP/123 8 608 4 304<br>
4 304<br>
<br>
Protocol data rates:555192.46 kbps total 36979.02 kbps in 518213.43 kbps out<br>
<br>
Some flows are sizable (top 10 in monitored period of several minutes):<br>
<br>
123715221 bytes<br>
43233291 bytes<br>
25762925 bytes<br>
23353052 bytes<br>
18263680 bytes<br>
16888624 bytes<br>
15858329 bytes<br>
14250494 bytes<br>
14081114 bytes<br>
13980641 bytes<br>
<br>
...many are quite small (smallest -> 46 bytes)<br>
<br>
I intend to run a lightly tuned Emerging Threats ruleset with<br>
something around 12K-13K rules enabled (current untuned rule<br>
breakout):<br>
<br>
20/5/2014 -- 09:39:42 - <Info> - 14341 signatures processed. 750 are<br>
IP-only rules, 4046 are inspecting packet payload, 10997 inspect<br>
application layer, 85 are decoder event only<br>
<br>
The current configuration attempt has about a 50% drop rate. stats.log<br>
at <a href="http://dpaste.com/246MJP9.txt" target="_blank">http://dpaste.com/246MJP9.txt</a>. Changes in config:<br>
<br>
- max-pending-packets: 3000<br>
- eve-log and http-log disabled<br>
- af-packet.ring-size: 524288<br>
- af-packet.buffer-size: 65536<br>
- detection-engine.profile: high<br>
- defrag.memcap: 8gb<br>
- flow.memcap: 16gb<br>
- flow.prealloc: 1000000<br>
- stream.memcap: 32gb<br>
- stream.prealloc-sessions: 1024000<br>
- reassembly.memcap: 16gb<br>
- stream.depth: 6mb<br>
<br>
Rules that fire are probably a result of the packet drop affecting<br>
session reassembly (my guess): <a href="http://dpaste.com/1B0RZC2.txt" target="_blank">http://dpaste.com/1B0RZC2.txt</a><br>
<br>
linux-vdso.so.1 => (0x00007fff64dff000)<br>
libhtp-0.5.10.so.1 => /usr/local/lib/libhtp-0.5.10.so.1<br>
(0x00007f55ad857000)<br>
libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2<br>
(0x00007f55ad5e7000)<br>
libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00000034f8600000)<br>
libcap-ng.so.0 => /usr/local/lib/libcap-ng.so.0 (0x00007f55ad3e2000)<br>
libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00000034f4e00000)<br>
libnet.so.1 => /lib64/libnet.so.1 (0x00007f55ad1c8000)<br>
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000034f3600000)<br>
libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007f55acfa8000)<br>
libpcre.so.1 => /opt/pcre-8.35/lib/libpcre.so.1 (0x00007f55acd40000)<br>
libc.so.6 => /lib64/libc.so.6 (0x00000034f2e00000)<br>
libz.so.1 => /lib64/libz.so.1 (0x00000034f3a00000)<br>
libm.so.6 => /lib64/libm.so.6 (0x00000034f3e00000)<br>
libdl.so.2 => /lib64/libdl.so.2 (0x00000034f3200000)<br>
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000034f4a00000)<br>
/lib64/ld-linux-x86-64.so.2 (0x00000034f2a00000)<br>
<br>
Other questions:<br>
<br>
- This sensor combines two half-duplex traffic feeds using a bonding<br>
interface as the capture interface (bond1). If offload features are<br>
disabled on each physical slave interface tied to the bond master,<br>
does one have to disable offload features on the bond interface?<br>
Attempting to disable some features on the bond pseudo-interface<br>
fails; I'm guessing that it's the physical interfaces that really<br>
matter.<br>
<br>
- If a monitored network segment carries traffic comprised of jumbo<br>
frames (say 9K frames), do the monitor interfaces on the sensor<br>
require their MTU to be set accordingly in order to receive/capture<br>
the full 9K of payload? Or is the MTU only relevant when transmitting,<br>
or irrelevant for a NIDS sensor (i.e. not an endpoint station)?<br>
<br>
- What is the correct practice with regard to the irqbalance daemon<br>
under RHEL-type distributions? Some guidance specifies to disable it<br>
entirely. Is that only a function of a system being tuned with CPU<br>
affinity settings? Is it relevant to 1Gbps installations or 10Gbps<br>
installations only?<br>
<br>
- Some guides suggest setting interface ring parameter limits higher,<br>
and setting network stack limits higher (sysctl). How important are<br>
these settings? For example:<br>
<br>
ethtool -G eth6 rx 4096<br>
sysctl -w net.core.netdev_max_backlog=250000<br>
sysctl -w net.core.rmem_max = 16777216<br>
sysctl -w net.core.rmem_max=16777216<br>
sysctl -w net.core.rmem_default=16777216<br>
sysctl -w net.core.optmem_max=16777216<br>
<br>
Thanks for any assistance. There's a lot of tuning guidance published<br>
already but I'm having difficulty determining just what is useful in a<br>
given situation.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Darren Spruell<br>
<a href="mailto:phatbuckett@gmail.com">phatbuckett@gmail.com</a><br>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
</font></span></blockquote></div><br><br clear="all"></div><div class="gmail_extra">Kernel level 2.6.32 is the bare minimum for pf_ring and not usable for af_packet with more than one thread.<br></div><div class="gmail_extra">
For af_packet (with more than one thread usage, aka 12 threads for example) it is recommended to use kernel level >= 3.2<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks<br></div><div class="gmail_extra">
<br><br>-- <br><div>Regards,</div>
<div>Peter Manev</div>
</div></div>