[Oisf-users] Out of band 10Gb Suricata

Thu Oct 15 13:13:17 UTC 2015

I'm going to try passing the nic directly to the vm instead of using a
vSwitch. I'm using server grade hardware. dual socket quad core E5 Xeons.
With the various tips around af_packet, the performance is getting better
but the drops are still a little too high for my liking when the rules are
on turned on.  I also have 34,000 rules on so I will give a serious look at
what I really want to be alerting on.
ESXi does have offloading enabled on the interface before it gets to the vm
so I suspect I need to turn that off too. It's using the built in VMware
drivers which don't allow for turning off offloading but I've been reading
that it can be done it I install the drivers from the server manufacture.

On Thu, Oct 15, 2015 at 8:48 AM, Peter Manev <petermanev at gmail.com> wrote:

> On Thu, Oct 15, 2015 at 6:50 AM, Christophe Vandeplas
> <christophe at vandeplas.com> wrote:
> > Just to be sure, how many queues do your vnics have?
> >
> > You can easily check this with :
> > $ cat /proc/interrupts  | fgrep eth
> >
> > This is important as the multiple threads of af_packet will need to
> > grab the packets from each queue. If you only have one queue in the
> > NIC, then only one thread can take care of reading these queues and
> > will jump to 100%, while the other receiving threads of af_packet will
> > do nothing.
> > Depending on the af_packet configuration this might even be worse.
> > If you're only having one queue, make sure af_packet is set to
> > "cluster-type: cluster_flow". You should see a considerable
> > improvement.
> >
> > I had this problem with cheap commodity hardware as explained in my
> > post:
> http://christophe.vandeplas.com/2013/11/suricata-capturekerneldrops-caused-by.html
> >
> > However, trying to get 10 Gbps with visualized hardware is perhaps a
> > little bit optimistic.
> >
>
> My view as well in terms of accuracy.
> I have not yet seen (in my experience anyway) a hypervisor capable
> traffic mirroring without doing some sort of negative interference
> impacting the IDS/IPS inspection in a virtual environment - aka
> packets reordered, offloading enabled (sometimes hardcoded and not
> configurable) etc.... that requires a lot of time to investigate and
> fix (if possible).
>
> I must agree it is much easier to fire up a virtual machine than to
> procure HW within  corporate policy regulations and approved suppliers
> - both in terms of time and process.
>
> >
> >
> > On 15 October 2015 at 02:48, Brian Hennigar <bhennigar at gmail.com> wrote:
> >> I think having 8 cores really is my issue. With no rules enabled, I'm
> still
> >> getting drops with af-packet although it is better.
> >>
> >> capture.kernel_drops      | AFPacketeth71             | 19611
> >> capture.kernel_drops      | AFPacketeth72             | 23942
> >> capture.kernel_drops      | AFPacketeth73             | 964
> >> capture.kernel_drops      | AFPacketeth74             | 14720
> >> capture.kernel_drops      | AFPacketeth75             | 0
> >> capture.kernel_drops      | AFPacketeth76             | 0
> >> capture.kernel_drops      | AFPacketeth77             | 0
> >> capture.kernel_drops      | AFPacketeth78             | 19216
> >>
> >>
> >> Thanks again for all of the help!  There's still much I need to learn
> about
> >> tuning Suricata.
> >>
> >> On Wed, Oct 14, 2015 at 8:23 PM, Brian Hennigar <bhennigar at gmail.com>
> wrote:
> >>>
> >>> I've looked into pf_ring.  vmxnet3 isn't supported by pf_ring and the
> >>> E1000 interface choice by ESXi is only 1gb which wouldn't work for
> 10Gb.
> >>> vmxnet3 supports 10gb.   Passing the interface directly through to the
> VM
> >>> might be an option but not ideal.
> >>>
> >>> I'm just starting on configuring it to use workers and af-packet.
> >>>
> >>> Thanks,
> >>> Brian
> >>>
> >>> On Wed, Oct 14, 2015 at 8:19 PM, Cooper F. Nelson <cnelson at ucsd.edu>
> >>> wrote:
> >>>>
> >>>> -----BEGIN PGP SIGNED MESSAGE-----
> >>>> Hash: SHA1
> >>>>
> >>>> I didn't notice that either.  All my deployments are bare metal, so I
> >>>> don't know well that will work.  If the NICs support recieve-side
> >>>> scaling everything should work well.
> >>>>
> >>>> - -Coop
> >>>>
> >>>> On 10/14/2015 2:38 PM, Chris Wakelin wrote:
> >>>> > Also it seems you're using virtual NICs ("vmxnet3")?
> >>>> >
> >>>> > Depending on which interface type you use and whether it supports
> >>>> > AFPacket, you might need something like PF_RING ZC
> >>>> >
> >>>> > (
> http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/
> ).
> >>>> >
> >>>> > Best Wishes,
> >>>> > Chris
> >>>>
> >>>>
> >>>> - --
> >>>> Cooper Nelson
> >>>> Network Security Analyst
> >>>> UCSD ACT Security Team
> >>>> cnelson at ucsd.edu x41042
> >>>> -----BEGIN PGP SIGNATURE-----
> >>>> Version: GnuPG v2.0.17 (MingW32)
> >>>>
> >>>> iQEcBAEBAgAGBQJWHuLnAAoJEKIFRYQsa8FWrvsH+wRBuQfoKKRFamD2qLXzuVUX
> >>>> JR9IeY22XRfoCrMGjD0h7Yic0fkt6DPLng/z4rmn0brgCjkSxYukdnhvHUyZzPTi
> >>>> lkDdkEevXGcA1CDqw2+ZyQsqRao2GO6EfOJ7pvH1QIL4rG7Aa2Nl+PVL1La2hq8k
> >>>> 8OEiTZr4/nGs7cUOGyFLooKgPh5lOeEjhRdkO0QueYK46IgWClRg/haIQEBT/YUK
> >>>> QbedoaAViBbQti2sWYbNi0MIZtWoELNuJxG+79aKEQkWWUbztbej29guX+mafojA
> >>>> el9JK1BuEnHz/VdIp+e1XCc39mur5qJMS47vwlVDD9IMFFfi2o69+ZdD5SiiiuQ=
> >>>> =2PmI
> >>>> -----END PGP SIGNATURE-----
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >> Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> >> List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> Suricata User Conference November 4 & 5 in Barcelona:
> http://oisfevents.net
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > Suricata User Conference November 4 & 5 in Barcelona:
> http://oisfevents.net
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20151015/f78c82da/attachment-0002.html>