[Oisf-users] Massive kernel drops with HTTP traffic

[Peter Manev]

On Tue, Aug 21, 2018 at 8:20 AM Konstantin Klinger
<konstantin.klinger at dcso.de> wrote:
>
> Good morning all,
>
> I've made multiple tests with different settings and you can find the
> results (drops in percentage) for each run in the attached table. We
> will rewrite our filestore rules without the "filemagic" keyword and try
> them in production. Further I will open a bug report.
>

Looking at the sum up - it seems the biggest impact(responsible for
14-37% drops just by having it on even with no rules) is having the
following combination in the config with filestore v1 -

filestore (v1) = on
force-magic = on

filestore v2 seems to behave  better but for the purpose of
completeness of the tests - I am curious of how it would behave with
rules loaded and filestore v2 off?

Thanks for testing!


> Thanks,
>
> Konstantin
>
> On 20.08.2018 15:43, Peter Manev wrote:
> >
> >
> >> On 20 Aug 2018, at 07:39, Konstantin Klinger <konstantin.klinger at dcso.de> wrote:
> >>
> >>
> >>
> >>> On 20.08.2018 15:29, Peter Manev wrote:
> >>>
> >>>
> >>>> On 20 Aug 2018, at 06:53, Konstantin Klinger <konstantin.klinger at dcso.de> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On 18.08.2018 15:57, Peter Manev wrote:
> >>>>>
> >>>>>
> >>>>>> On 17 Aug 2018, at 07:35, Michael Stone <mstone at mathom.us> wrote:
> >>>>>>
> >>>>>> On Fri, Aug 17, 2018 at 03:24:31PM +0200, you wrote:
> >>>>>>>> Do you have filemagic enabled?
> >>>>>>>
> >>>>>>> Yes. We currently use filestore v1. And we use the filemagic value in
> >>>>>>> our rules for filestoring.
> >>>>>>
> >>>>>> Unless you have customized the magic file it is very likely that you won't hit your performance target this way. I'd suggest rules specific to what you're trying to save rather than relying on libmagic (which is very inefficient).
> >>>>>>
> >>>>>
> >>>>>
> >>>>> That could be easy to test and confirm if it is contributing or creating the mess- Konstantin is it possible to try it out and see ?
> >>>>>
> >>>>>
> >>>>
> >>>> We made some test runs without filestore enabled and after that only
> >>>> without libmagic/filemagic (but filestore on) and that helped to
> >>>> decrease the number of packet drops (~30% -> ~5% and ~50% -> ~10%).
> >>>> Thank you. Our workaround will be not using filemagic rules anymore.
> >>>>
> >>>
> >>> If I remember correctly (please correct me if otherwise) you had a test run where you run Suri with no rules and the drops where still bad (30%+) - what is different from that test and the tests you mentioned above ? (Just having filestore switched to enabled in yaml?)
> >>
> >> Yes, you are completely right. With filestore enabled in yaml and no
> >> rules loaded we had still 30%+ packet drops.
> >
> > If not mistaken this is filestore v1 , correct ?
> > Is this the case with filestore v2 as well ?
> > Can you please post a bug report describing all the findings including Suricata version you are using (latest git if not mistaken?)
> >
> >
> >>
> >>>
> >>>
> >>>> @Mike: Do you have further experience in a workaround to not use libmagic?
> >>>> @all: Is someone using libmagic/filemagic on high traffic sensors
> >>>> (>5Gb/sec) and has no performance issues? Is someone already using
> >>>> filestore v2 (we are still using v1) and has any experience with it's
> >>>> performance?
> >>>>
>
>
> --
> Konstantin Klinger
> Security Content Engineer
> Threat Detection & Hunting (TDH)
>
> +49 160 95476260
> konstantin.klinger at dcso.de
>
> dcso.de
> blog.dcso.de
>
> PGP: 180D C5B3 3C68 5C9A FB58 6F33 400E 5A35 3307 8D46
>
> DCSO Deutsche Cyber-Sicherheitsorganisation GmbH • EUREF-Campus
> 22 • D-10829 Berlin
> Geschäftsführer: Dr.-Ing. Gunnar Siebert, Sitz der Gesellschaft: Berlin,
> Amtsgericht Charlottenburg HRB 172382



-- 
Regards,
Peter Manev