<div dir="ltr">I'm going to try passing the nic directly to the vm instead of using a vSwitch. I'm using server grade hardware. dual socket quad core E5 Xeons. With the various tips around af_packet, the performance is getting better but the drops are still a little too high for my liking when the rules are on turned on. I also have 34,000 rules on so I will give a serious look at what I really want to be alerting on.<div>ESXi does have offloading enabled on the interface before it gets to the vm so I suspect I need to turn that off too. It's using the built in VMware drivers which don't allow for turning off offloading but I've been reading that it can be done it I install the drivers from the server manufacture. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 15, 2015 at 8:48 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Oct 15, 2015 at 6:50 AM, Christophe Vandeplas<br>
<<a href="mailto:christophe@vandeplas.com">christophe@vandeplas.com</a>> wrote:<br>
> Just to be sure, how many queues do your vnics have?<br>
><br>
> You can easily check this with :<br>
> $ cat /proc/interrupts | fgrep eth<br>
><br>
> This is important as the multiple threads of af_packet will need to<br>
> grab the packets from each queue. If you only have one queue in the<br>
> NIC, then only one thread can take care of reading these queues and<br>
> will jump to 100%, while the other receiving threads of af_packet will<br>
> do nothing.<br>
> Depending on the af_packet configuration this might even be worse.<br>
> If you're only having one queue, make sure af_packet is set to<br>
> "cluster-type: cluster_flow". You should see a considerable<br>
> improvement.<br>
><br>
> I had this problem with cheap commodity hardware as explained in my<br>
> post: <a href="http://christophe.vandeplas.com/2013/11/suricata-capturekerneldrops-caused-by.html" rel="noreferrer" target="_blank">http://christophe.vandeplas.com/2013/11/suricata-capturekerneldrops-caused-by.html</a><br>
><br>
> However, trying to get 10 Gbps with visualized hardware is perhaps a<br>
> little bit optimistic.<br>
><br>
<br>
</span>My view as well in terms of accuracy.<br>
I have not yet seen (in my experience anyway) a hypervisor capable<br>
traffic mirroring without doing some sort of negative interference<br>
impacting the IDS/IPS inspection in a virtual environment - aka<br>
packets reordered, offloading enabled (sometimes hardcoded and not<br>
configurable) etc.... that requires a lot of time to investigate and<br>
fix (if possible).<br>
<br>
I must agree it is much easier to fire up a virtual machine than to<br>
procure HW within corporate policy regulations and approved suppliers<br>
- both in terms of time and process.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
> On 15 October 2015 at 02:48, Brian Hennigar <<a href="mailto:bhennigar@gmail.com">bhennigar@gmail.com</a>> wrote:<br>
>> I think having 8 cores really is my issue. With no rules enabled, I'm still<br>
>> getting drops with af-packet although it is better.<br>
>><br>
>> capture.kernel_drops | AFPacketeth71 | 19611<br>
>> capture.kernel_drops | AFPacketeth72 | 23942<br>
>> capture.kernel_drops | AFPacketeth73 | 964<br>
>> capture.kernel_drops | AFPacketeth74 | 14720<br>
>> capture.kernel_drops | AFPacketeth75 | 0<br>
>> capture.kernel_drops | AFPacketeth76 | 0<br>
>> capture.kernel_drops | AFPacketeth77 | 0<br>
>> capture.kernel_drops | AFPacketeth78 | 19216<br>
>><br>
>><br>
>> Thanks again for all of the help! There's still much I need to learn about<br>
>> tuning Suricata.<br>
>><br>
>> On Wed, Oct 14, 2015 at 8:23 PM, Brian Hennigar <<a href="mailto:bhennigar@gmail.com">bhennigar@gmail.com</a>> wrote:<br>
>>><br>
>>> I've looked into pf_ring. vmxnet3 isn't supported by pf_ring and the<br>
>>> E1000 interface choice by ESXi is only 1gb which wouldn't work for 10Gb.<br>
>>> vmxnet3 supports 10gb. Passing the interface directly through to the VM<br>
>>> might be an option but not ideal.<br>
>>><br>
>>> I'm just starting on configuring it to use workers and af-packet.<br>
>>><br>
>>> Thanks,<br>
>>> Brian<br>
>>><br>
>>> On Wed, Oct 14, 2015 at 8:19 PM, Cooper F. Nelson <<a href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a>><br>
>>> wrote:<br>
>>>><br>
>>>> -----BEGIN PGP SIGNED MESSAGE-----<br>
>>>> Hash: SHA1<br>
>>>><br>
>>>> I didn't notice that either. All my deployments are bare metal, so I<br>
>>>> don't know well that will work. If the NICs support recieve-side<br>
>>>> scaling everything should work well.<br>
>>>><br>
>>>> - -Coop<br>
>>>><br>
>>>> On 10/14/2015 2:38 PM, Chris Wakelin wrote:<br>
>>>> > Also it seems you're using virtual NICs ("vmxnet3")?<br>
>>>> ><br>
>>>> > Depending on which interface type you use and whether it supports<br>
>>>> > AFPacket, you might need something like PF_RING ZC<br>
>>>> ><br>
>>>> > (<a href="http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/" rel="noreferrer" target="_blank">http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/</a>).<br>
>>>> ><br>
>>>> > Best Wishes,<br>
>>>> > Chris<br>
>>>><br>
>>>><br>
>>>> - --<br>
>>>> Cooper Nelson<br>
>>>> Network Security Analyst<br>
>>>> UCSD ACT Security Team<br>
>>>> <a href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a> x41042<br>
>>>> -----BEGIN PGP SIGNATURE-----<br>
>>>> Version: GnuPG v2.0.17 (MingW32)<br>
>>>><br>
>>>> iQEcBAEBAgAGBQJWHuLnAAoJEKIFRYQsa8FWrvsH+wRBuQfoKKRFamD2qLXzuVUX<br>
>>>> JR9IeY22XRfoCrMGjD0h7Yic0fkt6DPLng/z4rmn0brgCjkSxYukdnhvHUyZzPTi<br>
>>>> lkDdkEevXGcA1CDqw2+ZyQsqRao2GO6EfOJ7pvH1QIL4rG7Aa2Nl+PVL1La2hq8k<br>
>>>> 8OEiTZr4/nGs7cUOGyFLooKgPh5lOeEjhRdkO0QueYK46IgWClRg/haIQEBT/YUK<br>
>>>> QbedoaAViBbQti2sWYbNi0MIZtWoELNuJxG+79aKEQkWWUbztbej29guX+mafojA<br>
>>>> el9JK1BuEnHz/VdIp+e1XCc39mur5qJMS47vwlVDD9IMFFfi2o69+ZdD5SiiiuQ=<br>
>>>> =2PmI<br>
>>>> -----END PGP SIGNATURE-----<br>
>>><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>> Suricata User Conference November 4 & 5 in Barcelona: <a href="http://oisfevents.net" rel="noreferrer" target="_blank">http://oisfevents.net</a><br>
> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> Suricata User Conference November 4 & 5 in Barcelona: <a href="http://oisfevents.net" rel="noreferrer" target="_blank">http://oisfevents.net</a><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Regards,<br>
Peter Manev<br>
</font></span></blockquote></div><br></div>