<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Everyone,<div><br></div><div>So finally after a lot of time spent on the troubleshooting from
Peter, Michal, Victor and Travis , figured out what was the exact cause of missing TCP alerts in suricata was.</div><div>It was because of the under assignment of memory for the "flow" and "stream" sections in suricata.yaml, which were set to default values, and suricata was assigning the memory at the run-time for them.</div><div><br></div><div><u>Cause:</u></div><div>It was caught while looking at the stats.log file, which was having pretty high values for tcp.ssn_memcap_drop, tcp.segment_memcap_drop, tcp.reassembly_gap.</div><div>That indicated that the TCP sessions were not properly tracked and processed due to the stream.memcap/stream.reassemble.memcap setting being too low.</div><div><br></div><div><u>Solution:</u></div><div>So I changed these following settings to ~10x in suricata.yaml:<div><span style="color:rgb(80,0,80)"><br></span></div><div><span style="color:rgb(80,0,80)">flow.memcap</span> : 2048mb # changing the default value from 128mb<br></div><div><span style="color:rgb(80,0,80)">flow.hash size</span> : 655360 # changing the default value from 65536</div><div>stream.memcap: 1000mb # changing the default value from 64mb<br></div><div>stream.reassembly.memcap: 4000mb # changing the default value from 256mb</div></div><div><br></div><div>And the TCP alerts started getting triggered nicely!</div><div>I also had an increased capture loss because of this, which got resolved by re-compiling suricata with disabled debugging and profiling options (Yes, I had those enabled when I built suricata before).</div><div><br></div><div>BIG THANKS to the patience and co-operation of Peter, Michal, Victor and Travis in helping me out to troubleshoot the issue!</div><div><br></div><div>Hope this helps to any future suricata newbie who is unaware of those memcap default settings and how it's so crucial in configuring suricata correctly for the given environment :)</div><div><br></div><div>Thanks everyone for the help and responses!</div><div><br></div><div>Fatema. </div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 16, 2018 at 7:59 PM Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com">michalpurzynski1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="ltr"></div><div dir="ltr">The number of suricata threads should be indeed equal to what you have pfring configured to. Not sure if pfring requires configuration in two places, though.</div><div dir="ltr"><br></div><div dir="ltr">Some parts of the Septun tuning, like pinning workers to cores, correct memory settings and making sure you’re not allocating memory during runtime would still apply.</div><div dir="ltr"><br></div><div dir="ltr">Ethtool, etc - depending on what pfring says.</div><div dir="ltr"><br></div><div dir="ltr">They basically bypassed most of the Linux network stack and it’s hard to say what it is they did there. </div><div dir="ltr"><br></div><div dir="ltr">The only advantage I can find of pfring is that it should work on your kernel version.</div><div dir="ltr"><br>On Oct 16, 2018, at 11:53 AM, fatema bannatwala <<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr">Had a quick question regarding using pf_ring, I think I am doing something wrong: <div>So in the suricata.yaml it says: <br><br></div><div>"#Number of receive threads (>1 will enable experimental flow pinned runmode)"<br> #threads: 1</div><div><br></div><div>And I am using 18 as the threads value here in pf_ring section. So should I change it to 1?</div><div><br></div><div>Also, I am running kernel Linux 3.10 and for h/w NIC based RSS hashing it requires at least Linux 3.14, so I can't use it unless I upgrade the kernel.</div><div>But I think if I can set the pf_ring the correct way it should work. </div><div>Do I have to undo the ethtool settings mentioned in Septune to use pf_ring over af_packet? I believe those settings were mentioned for using af_packet and shouldn't be needed/set for pf_ring?</div><div><br></div><div>Thanks,</div><div>Fatema</div><table class="m_1807857327139856315gmail-highlight m_1807857327139856315gmail-tab-size m_1807857327139856315gmail-js-file-line-container" style="box-sizing:border-box;border-spacing:0px;border-collapse:collapse;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td id="m_1807857327139856315gmail-LC1724" class="m_1807857327139856315gmail-blob-code m_1807857327139856315gmail-blob-code-inner m_1807857327139856315gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre-wrap"></td></tr><tr style="box-sizing:border-box"><td id="m_1807857327139856315gmail-L1725" class="m_1807857327139856315gmail-blob-num m_1807857327139856315gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;color:rgba(27,31,35,0.3);text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_1807857327139856315gmail-LC1725" class="m_1807857327139856315gmail-blob-code m_1807857327139856315gmail-blob-code-inner m_1807857327139856315gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre-wrap"></td></tr><tr style="box-sizing:border-box"><td id="m_1807857327139856315gmail-L1726" class="m_1807857327139856315gmail-blob-num m_1807857327139856315gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;color:rgba(27,31,35,0.3);text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_1807857327139856315gmail-LC1726" class="m_1807857327139856315gmail-blob-code m_1807857327139856315gmail-blob-code-inner m_1807857327139856315gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre-wrap"></td></tr></tbody></table></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 16, 2018 at 1:30 PM fatema bannatwala <<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I tried with pf_ring, but couldn't get the "SearchProtect" (sid:2022813) to trigger still.<div>Next I am going to try the hardware hashing and af_packet in qm mode and see if there's any difference, will keep posted..</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 12, 2018 at 2:45 PM Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com" target="_blank">michalpurzynski1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">After talking a bit more with Fatema, we learned there might be a problem with the kernel version on that server.<div><br></div><div>Victor's patch shows 30% of packets sent to a wrong thread. Bro and Snort use pf_ring.</div><div><br></div><div>Suggestions would be to use hardware hashing and af_packet in QM mode. Note - the XDP part is not needed for this setup to work.</div><div><br></div><div><font color="#1155cc"><u><a href="https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst#setup-symmetric-hashing-on-the-nic" target="_blank">https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst#setup-symmetric-hashing-on-the-nic</a></u></font><br></div><div><font color="#1155cc"><u><br></u></font></div>just do the symmetric hashing part and then change Suricata's configuration to use cluster_qm</div><div dir="ltr"><br></div><div dir="ltr"><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:13.6px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;background-color:rgb(246,248,250);border-radius:3px;color:rgb(36,41,46)"> cluster-type: cluster_qm # symmetric hashing is a must!
</pre><br class="m_1807857327139856315m_3655313643501646730m_-2856382719543780792gmail-Apple-interchange-newline"></div><div dir="ltr"><br></div><div dir="ltr"><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 5, 2018 at 2:20 PM Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com" target="_blank">michalpurzynski1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">How about we make Suricata write us a pcap in afpacket workers mode? I’m pretty sure a rule can do that.<br>
<br>
> On Oct 5, 2018, at 8:17 PM, fatema bannatwala <<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@gmail.com</a>> wrote:<br>
> <br>
> Changing $HOME_NET to any in sid 2022813 didn't help though, still not getting that alert fired.<br>
> One difference I had in suricata.yaml when running in offline pcap reading mode was, I set runmode to "single", while when suricata runs in packet sniffing mode it's set to "workers".<br>
> <br>
> I tried to set it to "runmode:single" while on interface sniffing mode but was hit by ~60% capture loss, which makes sense as single threaded suricata can't handle the traffic flowing through the interface. <br>
> <br>
> The fact that alerts are fired when in offline single threaded mode and same alerts are not fired when online packet sniffing multi-threaded mode, makes me think it has to do with multi-threading vs single threaded mode and how "workers" are capturing packets.<br>
> <br>
> I will keep looking.<br>
> <br>
> (The good thing is that Interrupt/IRQ pinning has helped to reduce capture loss to 0%)<br>
> <br>
> Thanks,<br>
> Fatema <br>
> <br>
> <br>
> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> <br>
> Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
> Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div></blockquote></div>