<div dir="ltr">Hello all.<div><br></div><div>Need some help getting Suricata to a stable point. I've tried many different approaches but the end result is always a poorly performing or crashed Suricata. </div><div><br>
</div><div>The system has 48 cores and I'm using (2) Intel X520-DA2 cards in the system to benefit from the 16 hardware queues per port. I'm only using (3) of those ports so there's a 1:1 relation between hardware queues and CPU cores.<br>
</div><div><br></div><div>Up until recently I was using PF_RING; I briefly tried DNA mode but it didn't seem to be working correctly so I gave up and returned to trying AF-Packet. I've tried to replicate the success with AF-Packet described at the links below. Last night the sensor seemed to be performing OK with 13500 rules, 2Gbps of traffic, and roughly 1% packet loss but this morning I found it had crashed. I know that is a lot of rules so I tried disabling all of them and the system is still dropping lots of packets. (See snapshot below).</div>
<div><br></div><div>I've attached a few files that show my attempts at testing different instances of Suricata with different values for max-pending-packets. This value seems to have different meanings depending on what point in time you read about it and what version of Suricata you're using. The documentation suggests something above 1000 but I've seen recommendations to use 512. Last night I seemed to have the best success with 8192 but that obviously failed as traffic volume increased.</div>
<div><br></div><div>Where I'm confused most is why Suricata is dropping so many packets with no rules enabled. The "AF-Packet" link below used this approach to find a stable point before adding rules. I've tuned the Intel cards as recommended with setpci, ethtool, and ixgbe parameters. The system has also been tuned with various sysctl tweaks to match other's recommendations (_rmem, _wmem, backlog, etc...) as well as set_irq_affinity to balance the interrupts among all CPU cores. (see attached files)</div>
<div><br></div><div>Any help is much appreciated... thanks !</div><div><br></div><div>--TC</div><div><br></div><div><br></div><div><br></div><div>[Suricata 10g]</div><div><a href="https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/">https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/</a><br>
</div><div><br></div><div>[AF-Packet tuning]</div><div><a href="http://comments.gmane.org/gmane.comp.security.ids.oisf.user/2464">http://comments.gmane.org/gmane.comp.security.ids.oisf.user/2464</a><br></div><div><br></div>
<div><br></div><div><br></div><div><div>Funny numbers showing drops with no rules enabled:</div></div><div><br></div><div><div>#############################################################################################################################</div>
<div>Date: 8/14/2013 -- 10:20:50 (uptime: 0d, 00h 31m 00s) </div><div>#############################################################################################################################</div><div>Total MBPS: 650.756644266667 </div>
<div><br></div><div>Thread: AFPacketeth41 Bytes: 2995441515 Mbps: 10.076 Pkt/Pps: 28255344 /10088.067 Drops: 24055897 + 123621 (65.931 mbps) Drop_Ratio: (85.138) </div><div>Thread: AFPacketeth410 Bytes: 3346886293 Mbps: 20.402 Pkt/Pps: 23360115 /19907.867 Drops: 16997822 + 252448 (134.639 mbps) Drop_Ratio: (72.764) </div>
<div>Thread: AFPacketeth411 Bytes: 5242111362 Mbps: 21.581 Pkt/Pps: 22054566 /13044.600 Drops: 12222382 + 120772 (64.412 mbps) Drop_Ratio: (55.419) </div><div>Thread: AFPacketeth412 Bytes: 156680895 Mbps: 0.466 Pkt/Pps: 22221963 /8594.333 Drops: 21875141 + 126863 (67.660 mbps) Drop_Ratio: (98.439) </div>
<div>Thread: AFPacketeth413 Bytes: 2544210363 Mbps: 11.401 Pkt/Pps: 25915536 /13694.333 Drops: 21666833 + 172582 (92.044 mbps) Drop_Ratio: (83.606) </div><div>Thread: AFPacketeth414 Bytes: 3561917298 Mbps: 15.858 Pkt/Pps: 19429689 /10577.867 Drops: 13099237 + 108390 (57.808 mbps) Drop_Ratio: (67.419) </div>
<div>Thread: AFPacketeth415 Bytes: 5647206023 Mbps: 19.740 Pkt/Pps: 47567141 /27139.400 Drops: 38992832 + 344503 (183.735 mbps) Drop_Ratio: (81.974) </div><div>Thread: AFPacketeth416 Bytes: 33653821 Mbps: 0.000 Pkt/Pps: 13081538 /0.000 Drops: 13041501 + 0 (0.000 mbps) Drop_Ratio: (99.694) </div>
<div>Thread: AFPacketeth42 Bytes: 4337730439 Mbps: 19.319 Pkt/Pps: 26591472 /15480.133 Drops: 20194338 + 185007 (98.670 mbps) Drop_Ratio: (75.943) </div><div>Thread: AFPacketeth43 Bytes: 5281383317 Mbps: 22.964 Pkt/Pps: 23774704 /9801.867 Drops: 14058469 + 74196 (39.571 mbps) Drop_Ratio: (59.132) </div>
<div>Thread: AFPacketeth44 Bytes: 76669253 Mbps: 0.000 Pkt/Pps: 22056217 /0.000 Drops: 21957696 + 0 (0.000 mbps) Drop_Ratio: (99.553) </div><div>Thread: AFPacketeth45 Bytes: 3331446249 Mbps: 11.284 Pkt/Pps: 23554504 /9575.800 Drops: 18728170 + 107727 (57.454 mbps) Drop_Ratio: (79.510) </div>
<div>Thread: AFPacketeth46 Bytes: 2948711961 Mbps: 11.851 Pkt/Pps: 36204709 /15019.600 Drops: 29826037 + 176047 (93.892 mbps) Drop_Ratio: (82.382) </div><div>Thread: AFPacketeth47 Bytes: 5604370277 Mbps: 22.842 Pkt/Pps: 40383590 /27318.800 Drops: 30910508 + 341044 (181.890 mbps) Drop_Ratio: (76.542) </div>
<div>Thread: AFPacketeth48 Bytes: 36113443 Mbps: 0.000 Pkt/Pps: 18750810 /0.000 Drops: 18700502 + 0 (0.000 mbps) Drop_Ratio: (99.732) </div><div>Thread: AFPacketeth49 Bytes: 3238122447 Mbps: 10.053 Pkt/Pps: 24588447 /10384.667 Drops: 19794915 + 118835 (63.379 mbps) Drop_Ratio: (80.505) </div>
<div>Thread: AFPacketeth51 Bytes: 2617449334 Mbps: 9.870 Pkt/Pps: 24227684 /12967.667 Drops: 19668050 + 160656 (85.683 mbps) Drop_Ratio: (81.180) </div><div>Thread: AFPacketeth510 Bytes: 1539172968 Mbps: 7.025 Pkt/Pps: 20411967 /10067.067 Drops: 17452693 + 127408 (67.951 mbps) Drop_Ratio: (85.502) </div>
<div>Thread: AFPacketeth511 Bytes: 4670360715 Mbps: 18.840 Pkt/Pps: 24294083 /11410.000 Drops: 16172209 + 109600 (58.453 mbps) Drop_Ratio: (66.569) </div><div>Thread: AFPacketeth512 Bytes: 8859339489 Mbps: 35.883 Pkt/Pps: 33312269 /15514.267 Drops: 22377119 + 148589 (79.247 mbps) Drop_Ratio: (67.174) </div>
<div>Thread: AFPacketeth513 Bytes: 2149074998 Mbps: 8.425 Pkt/Pps: 26949484 /12965.333 Drops: 22584848 + 160618 (85.663 mbps) Drop_Ratio: (83.804) </div><div>Thread: AFPacketeth514 Bytes: 2002945700 Mbps: 8.809 Pkt/Pps: 19955024 /12547.333 Drops: 16574258 + 162560 (86.699 mbps) Drop_Ratio: (83.058) </div>
<div>Thread: AFPacketeth515 Bytes: 5765375800 Mbps: 26.217 Pkt/Pps: 19965863 /11071.333 Drops: 10812913 + 98371 (52.465 mbps) Drop_Ratio: (54.157) </div><div>Thread: AFPacketeth516 Bytes: 8102226635 Mbps: 34.460 Pkt/Pps: 30629858 /14215.133 Drops: 17675503 + 113721 (60.651 mbps) Drop_Ratio: (57.707) </div>
<div>Thread: AFPacketeth52 Bytes: 4058518248 Mbps: 15.818 Pkt/Pps: 24112713 /12745.867 Drops: 17804790 + 142965 (76.248 mbps) Drop_Ratio: (73.840) </div><div>Thread: AFPacketeth53 Bytes: 6219376156 Mbps: 20.448 Pkt/Pps: 17499586 /7645.333 Drops: 7809993 + 41825 (22.307 mbps) Drop_Ratio: (44.630) </div>
<div>Thread: AFPacketeth54 Bytes: 45025327 Mbps: 0.344 Pkt/Pps: 21800877 /1215218.200 Drops: 21744370 + 18227246 (9721.198 mbps) Drop_Ratio: (99.741) </div><div>Thread: AFPacketeth55 Bytes: 2084616048 Mbps: 8.954 Pkt/Pps: 21084958 /9916.800 Drops: 16526356 + 113868 (60.730 mbps) Drop_Ratio: (78.380) </div>
<div>Thread: AFPacketeth56 Bytes: 3236157769 Mbps: 15.029 Pkt/Pps: 36598967 /15831.400 Drops: 30451094 + 190275 (101.480 mbps) Drop_Ratio: (83.202) </div><div>Thread: AFPacketeth57 Bytes: 5547821519 Mbps: 24.318 Pkt/Pps: 32547090 /13555.933 Drops: 23132492 + 130494 (69.597 mbps) Drop_Ratio: (71.074) </div>
<div>Thread: AFPacketeth58 Bytes: 19851376 Mbps: 0.000 Pkt/Pps: 10216575 /0.000 Drops: 10188819 + 0 (0.000 mbps) Drop_Ratio: (99.728) </div><div>Thread: AFPacketeth59 Bytes: 2927674101 Mbps: 11.225 Pkt/Pps: 22598751 /11942.267 Drops: 18451574 + 147325 (78.573 mbps) Drop_Ratio: (81.649) </div>
<div>Thread: AFPacketeth61 Bytes: 2426135593 Mbps: 9.297 Pkt/Pps: 27365049 /12910.533 Drops: 23109109 + 160822 (85.772 mbps) Drop_Ratio: (84.448) </div><div>Thread: AFPacketeth610 Bytes: 1910417797 Mbps: 7.170 Pkt/Pps: 22264452 /9111.533 Drops: 19127882 + 112051 (59.761 mbps) Drop_Ratio: (85.912) </div>
<div>Thread: AFPacketeth611 Bytes: 3951772716 Mbps: 18.887 Pkt/Pps: 18207944 /11483.933 Drops: 12540217 + 127111 (67.793 mbps) Drop_Ratio: (68.872) </div><div>Thread: AFPacketeth612 Bytes: 622812604 Mbps: 1.710 Pkt/Pps: 20629165 /22146.933 Drops: 19644158 + 328107 (174.990 mbps) Drop_Ratio: (95.225) </div>
<div>Thread: AFPacketeth613 Bytes: 2661408245 Mbps: 10.490 Pkt/Pps: 32391332 /16740.667 Drops: 28326169 + 220328 (117.508 mbps) Drop_Ratio: (87.450) </div><div>Thread: AFPacketeth614 Bytes: 2221086307 Mbps: 8.914 Pkt/Pps: 28652510 /12693.867 Drops: 25452745 + 164758 (87.871 mbps) Drop_Ratio: (88.833) </div>
<div>Thread: AFPacketeth615 Bytes: 3512347703 Mbps: 10.944 Pkt/Pps: 17582785 /13286.533 Drops: 11632869 + 151065 (80.568 mbps) Drop_Ratio: (66.161) </div><div>Thread: AFPacketeth616 Bytes: 486827305 Mbps: 1.838 Pkt/Pps: 23357938 /14689.400 Drops: 22406725 + 213170 (113.691 mbps) Drop_Ratio: (95.928) </div>
<div>Thread: AFPacketeth62 Bytes: 2053540892 Mbps: 9.747 Pkt/Pps: 35952787 /30316.067 Drops: 32944477 + 434221 (231.585 mbps) Drop_Ratio: (91.633) </div><div>Thread: AFPacketeth63 Bytes: 4433960688 Mbps: 16.308 Pkt/Pps: 35950960 /17537.133 Drops: 27496651 + 202528 (108.015 mbps) Drop_Ratio: (76.484) </div>
<div>Thread: AFPacketeth64 Bytes: 8043394639 Mbps: 26.669 Pkt/Pps: 26656005 /13408.133 Drops: 14772843 + 109805 (58.563 mbps) Drop_Ratio: (55.420) </div><div>Thread: AFPacketeth65 Bytes: 2462063424 Mbps: 10.764 Pkt/Pps: 26018483 /14212.667 Drops: 21762348 + 180354 (96.189 mbps) Drop_Ratio: (83.642) </div>
<div>Thread: AFPacketeth66 Bytes: 1971617560 Mbps: 8.703 Pkt/Pps: 22118650 /7928.400 Drops: 18729282 + 91224 (48.653 mbps) Drop_Ratio: (84.676) </div><div>Thread: AFPacketeth67 Bytes: 4831551279 Mbps: 18.394 Pkt/Pps: 19235610 /12802.600 Drops: 10284691 + 128445 (68.504 mbps) Drop_Ratio: (53.467) </div>
<div>Thread: AFPacketeth68 Bytes: 8257637668 Mbps: 38.944 Pkt/Pps: 29011595 /14320.667 Drops: 16893384 + 122469 (65.317 mbps) Drop_Ratio: (58.230) </div><div>Thread: AFPacketeth69 Bytes: 2298683778 Mbps: 8.474 Pkt/Pps: 21089907 /8833.133 Drops: 17079228 + 101718 (54.250 mbps) Drop_Ratio: (80.983)</div>
</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>