<div dir="ltr">Alright it seems to be fixed now.<div><br></div><div>There were two problems:</div><div> - incorrect mapping of hardware queues to CPU sockets. Ended up using 12 queues per port and set_irq_affinity adjusted for each card. Otherwise when exceeding 32 cores the queues would fail in a way where only 24 of 48 cores were processing packets at a time; as you mentioned I guess because of cache misses. Not really 100% sure of how exactly things failed, each CPU socket has 12 cores and the problem with processing packets didn't become clear until mapping more than 30 queues (15 per port) to CPU sockets. Maybe looking at the motherboard architecture and PCI interconnections will explain it better. </div>
<div><br></div><div> - something in one of a few rulesets causes high CPU. </div><div><br></div><div><br></div><div class="gmail_extra">So now there are much less drops with near 13,300 rules enabled and 2.0+ Gbps.</div><div class="gmail_extra">
<br></div><div class="gmail_extra">Thanks again to all for the suggestions. If anyone wants to continue discussing details plz email on/off list.</div><div class="gmail_extra"><br></div><div class="gmail_extra">--TC</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">#############################################################################################################################</div>
<div class="gmail_extra">Date: 8/22/2013 -- 01:02:51 (uptime: 0d, 00h 38m 59s) </div><div class="gmail_extra">#############################################################################################################################</div>
<div class="gmail_extra">Total MBPS: 2546.9156688 </div><div class="gmail_extra"><br></div><div class="gmail_extra">Thread: AFPacketeth41 Bytes: 16366415124 Mbps: 31.984 Pkt/Pps: 19575467 /5186.733 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth410 Bytes: 10245036394 Mbps: 18.384 Pkt/Pps: 14758433 /4369.800 Drops: 299905 + 0 (0.000 mbps) Drop_Pct: (2.032) </div><div class="gmail_extra">
Thread: AFPacketeth411 Bytes: 13131147917 Mbps: 72.449 Pkt/Pps: 16514029 /10131.800 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">Thread: AFPacketeth412 Bytes: 16607848510 Mbps: 49.942 Pkt/Pps: 22673735 /9903.200 Drops: 48624 + 0 (0.000 mbps) Drop_Pct: (0.214) </div>
<div class="gmail_extra">Thread: AFPacketeth42 Bytes: 21279424611 Mbps: 64.457 Pkt/Pps: 23420066 /8133.733 Drops: 100581 + 0 (0.000 mbps) Drop_Pct: (0.429) </div><div class="gmail_extra">
Thread: AFPacketeth43 Bytes: 20952062599 Mbps: 52.664 Pkt/Pps: 22544918 /8296.800 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">Thread: AFPacketeth44 Bytes: 21646709377 Mbps: 50.936 Pkt/Pps: 25781204 /7140.133 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth45 Bytes: 15383150783 Mbps: 78.167 Pkt/Pps: 20261690 /9784.133 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">
Thread: AFPacketeth46 Bytes: 17634089691 Mbps: 87.919 Pkt/Pps: 20062644 /13358.933 Drops: 280329 + 0 (0.000 mbps) Drop_Pct: (1.397) </div><div class="gmail_extra">Thread: AFPacketeth47 Bytes: 14852922251 Mbps: 28.568 Pkt/Pps: 18906176 /5472.067 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth48 Bytes: 19192180601 Mbps: 65.349 Pkt/Pps: 21169979 /8179.867 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">
Thread: AFPacketeth49 Bytes: 17092426607 Mbps: 42.345 Pkt/Pps: 18896665 /6360.600 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">Thread: AFPacketeth51 Bytes: 21921608968 Mbps: 226.190 Pkt/Pps: 24163908 /30492.333 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth510 Bytes: 11550726486 Mbps: 51.381 Pkt/Pps: 14427526 /6452.000 Drops: 132820 + 0 (0.000 mbps) Drop_Pct: (0.921) </div><div class="gmail_extra">
Thread: AFPacketeth511 Bytes: 54708051319 Mbps: 175.302 Pkt/Pps: 71734215 /41205.800 Drops: 19512622 + 318040 (169.621 mbps) Drop_Pct: (27.201) </div><div class="gmail_extra">Thread: AFPacketeth512 Bytes: 21054036130 Mbps: 73.930 Pkt/Pps: 26583869 /9712.333 Drops: 5083234 + 0 (0.000 mbps) Drop_Pct: (19.121) </div>
<div class="gmail_extra">Thread: AFPacketeth52 Bytes: 13038902616 Mbps: 32.743 Pkt/Pps: 17146663 /5707.867 Drops: 173228 + 0 (0.000 mbps) Drop_Pct: (1.010) </div><div class="gmail_extra">
Thread: AFPacketeth53 Bytes: 14177901992 Mbps: 36.754 Pkt/Pps: 18706243 /8427.467 Drops: 18963 + 0 (0.000 mbps) Drop_Pct: (0.101) </div><div class="gmail_extra">Thread: AFPacketeth54 Bytes: 11913396211 Mbps: 106.956 Pkt/Pps: 16096919 /13416.600 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth55 Bytes: 40751548800 Mbps: 172.795 Pkt/Pps: 37561958 /16581.133 Drops: 286951 + 0 (0.000 mbps) Drop_Pct: (0.764) </div><div class="gmail_extra">
Thread: AFPacketeth56 Bytes: 18435184268 Mbps: 55.993 Pkt/Pps: 21604180 /7256.667 Drops: 791600 + 0 (0.000 mbps) Drop_Pct: (3.664) </div><div class="gmail_extra">Thread: AFPacketeth57 Bytes: 18788982722 Mbps: 75.457 Pkt/Pps: 29683869 /11954.133 Drops: 168840 + 0 (0.000 mbps) Drop_Pct: (0.569) </div>
<div class="gmail_extra">Thread: AFPacketeth58 Bytes: 41693312006 Mbps: 168.752 Pkt/Pps: 49464085 /19896.933 Drops: 7144688 + 0 (0.000 mbps) Drop_Pct: (14.444) </div><div class="gmail_extra">
Thread: AFPacketeth59 Bytes: 11084075844 Mbps: 38.537 Pkt/Pps: 31496208 /24812.667 Drops: 144971 + 0 (0.000 mbps) Drop_Pct: (0.460) </div><div class="gmail_extra">Thread: AFPacketeth61 Bytes: 19851090605 Mbps: 86.707 Pkt/Pps: 23917426 /10486.733 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth610 Bytes: 15065317294 Mbps: 45.003 Pkt/Pps: 19284480 /7485.267 Drops: 1090397 + 0 (0.000 mbps) Drop_Pct: (5.654) </div><div class="gmail_extra">
Thread: AFPacketeth611 Bytes: 25313092307 Mbps: 122.486 Pkt/Pps: 25216675 /14637.800 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">Thread: AFPacketeth612 Bytes: 15141364019 Mbps: 53.058 Pkt/Pps: 18619986 /7684.933 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth62 Bytes: 23394644144 Mbps: 57.088 Pkt/Pps: 25380248 /10430.800 Drops: 323662 + 0 (0.000 mbps) Drop_Pct: (1.275) </div><div class="gmail_extra">
Thread: AFPacketeth63 Bytes: 19220970896 Mbps: 31.680 Pkt/Pps: 24602996 /6162.467 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">Thread: AFPacketeth64 Bytes: 21323350105 Mbps: 47.244 Pkt/Pps: 24003593 /8142.733 Drops: 121045 + 0 (0.000 mbps) Drop_Pct: (0.504) </div>
<div class="gmail_extra">Thread: AFPacketeth65 Bytes: 17659360005 Mbps: 34.660 Pkt/Pps: 20544438 /5478.200 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">
Thread: AFPacketeth66 Bytes: 17395262057 Mbps: 50.690 Pkt/Pps: 21360464 /7982.133 Drops: 66797 + 0 (0.000 mbps) Drop_Pct: (0.313) </div><div class="gmail_extra">Thread: AFPacketeth67 Bytes: 15013426883 Mbps: 39.373 Pkt/Pps: 18403927 /6545.667 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div>
<div class="gmail_extra">Thread: AFPacketeth68 Bytes: 14639157324 Mbps: 27.275 Pkt/Pps: 18057579 /5032.200 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra">
Thread: AFPacketeth69 Bytes: 13983930153 Mbps: 93.697 Pkt/Pps: 19197920 /13527.467 Drops: 0 + 0 (0.000 mbps) Drop_Pct: (0.000) </div><div class="gmail_extra"><br></div><div class="gmail_extra">
<br></div><div class="gmail_extra"><br></div></div></div>